Tests are called by statistics.
Each test starts with a hypothesis, a split and a planned length. A statistical check decides whether a result is real; the agent reports it.
A finished test becomes a learning.
The result goes into the agent’s notebook, and its next plan starts from it.
Ahead, called, remembered
One idea at three stages: a test that’s ahead but not yet called, the finished test it grew from, and the learning the agent keeps.
- 1Ahead, not called
Gift sets, day 9 of 14: roast date at 7.2% against 5.4%, 87% confident. It’s called on day 14, and only at 95%.
- 2Called
Coffee, October 7 to 20: roast dates won, with 37% more first orders per click, 95% confident. No difference for brewing gear.
- 3Remembered
The learning in the New Customers agent’s notebook, and the October 21 move that cited it.
Screen by screen
- 1
A hypothesis, then a test
“People buying a gift respond more to how fresh the coffee is than to how it’s made.” Roast date against “small batch,” half the traffic each, for 14 days, measured on first orders per click. A test starts the morning after you approve its proposal.
A hypothesis, then a test - 2
Ahead isn’t the same as won
On day 9, roast date leads 7.2% to 5.4%, at 87% confidence. That’s promising, but it isn’t a verdict. The call comes when the test ends on day 14, and only at 95% or more.
Ahead isn’t the same as won - 3
Too early to call, on purpose
A free-shipping bar on product pages is on day 3 of 21. It reads 6.0% against 4.8%, which looks good, but it’s still too early to call: every version needs at least 20 conversions before a reading counts.
Too early to call, on purpose - 4
Learnings that carry forward
“Roast dates won: 37% more first orders per click, 95% confident. For brewing gear alone there was no difference.” A finished test becomes a learning in the notebook, and later plans start from it. The next budget move cited this one.
Learnings that carry forward - 5
Holdouts, for what a split can’t test
Some effects, like an email’s, can’t be tested by splitting traffic. Instead, a share of the audience is deliberately left out, and the email is measured against the people who didn’t get it. On the running-low email, 93 people, 5%, are held out.
Holdouts, for what a split can’t test
The New Customers agent’s testsOne example, step by step
Oct 7 – Oct 20 · Finished
Roast-date messages against “small batch,” across every product line: 5.0% against 3.6%. Winner found: 37% more first orders per click, 95% confident. For brewing gear alone, no difference.
Oct 21 · New Customers agent
Moved $60 a day toward the Holiday Blend and gift sets, citing the result: roast-date messages beat the control by $8 per first order.
Oct 20 – Nov 2 · Running, day 9 of 14
Roast date against “small batch” for gift sets: 7.2% against 5.4%, 87% confident. Ahead, not called.
Oct 26 – Nov 15 · Running, day 3 of 21
A free-shipping bar on product pages: 6.0% against 4.8%. Too early to call.
Elsewhere · Reorders agent
A short follow-up email against a long one: clicks to reorder more than doubled, from 3% to 7%, over 99% confident. Its next test, a brew-guide sequence, is too early to call.
How a test is judged
The rules stated under “How experiments are judged” on every experiments page, enforced by the software.
The leading challenger faces the current versionBuilt in
The leading challenger is compared with the current version, run side by side over the same days, so a busy week can’t make one look better than the other.
Called once, at 95%Built in
A winner is called only when the planned run ends, at 95% confidence from a one-sided test of the two conversion rates. A gap that size would turn up less than one time in twenty if the versions performed the same.
At least 20 conversions per versionBuilt in
Until every version has 20 conversions, the reading is “Too early to call,” however big the gap looks.
Below 75%, no difference yetBuilt in
With enough data but under 75% confidence, the reading is “No clear difference yet.”
No winner at the end means no changeBuilt in
If nothing is clearly better by the planned end, the test is recorded as no difference and the current version stays.
A holdout on email sendsCompany setting
A share of each email audience is held out to measure the email’s effect: 5% by default.
What happens when…
What happens when a test looks great on day 3?
Nothing yet. Early leads are often noise. The test runs its planned length, and it’s called once at the end: at 95% or more, with at least 20 conversions in each version.
What happens when a test ends without a winner?
The current version stays and the test is recorded as no difference. The notebook keeps what was tried, so the agent doesn’t repeat it blindly.
What happens when the result that matters is slow, like a second order?
The agent waits for it rather than calling the test on a faster signal. The Reorders agent saw open rates of 61% against 47% and still called its test too early, because opens aren’t the goal; second orders are.
What happens when my volume is low?
Plan longer tests. The 95% bar and the 20-conversion minimum don’t move, so with less volume a test needs a longer run or a bigger difference; otherwise it ends as no difference.
Why wait for 95%, instead of acting as soon as a version is ahead?
Because checking every day and stopping when a result looks good finds winners that aren’t real. One call at the planned end, a fixed bar and a minimum number of conversions keep the results honest.
Do tests need my approval?
A test starts from a proposal, so it follows that action’s permission level. By default, launches wait for your approval, and an approved test starts the next morning.
What’s a holdout?
A group of people deliberately left out of a send, so the send’s effect can be measured against people who didn’t get it, instead of assumed.
Does the AI decide whether a test won?
No. Confidence is calculated from the conversion counts. The agent decides what to test and what to do with the answer.
In the demo Two running tests and one finished, each with its reading and confidence.
Open experiments in the demo (opens in a new tab)Let code call the tests.
Join the beta, or try the demo first.