How an A/B test becomes a verdict.
Follow one experiment from start to finish. You write down what you expect, build the variation, choose who sees it, wait for the data, and read the verdict. Seven steps, in order. Start at step one, or jump straight to the answer.
Where you end up: a verdict, the range around the lift, and a trust grade. Steps 01 to 07 get you there.
Say what you expect. Then the plan seals.
Every experiment starts with a claim: change this, and that number moves. AvsB makes you write the claim down before any visitor sees anything. That way nobody can adjust the question once the data starts arriving.
A guided path takes you from idea to launch in 5 steps, in this order: Targeting, Variations, Metrics, Analysis, Review. Nothing goes live until you have been through all 5.
How an experiment is planned and read, end to endThe question cannot move once data starts arriving
This is where you settle the engine that will judge the result, the metric it is judged on, and the metrics you only want to watch beside it. At launch the plan seals. After that, any change is recorded as an amendment against the sealed values rather than quietly replacing them, and your result is graded on how closely you stuck to the plan.
Describe the change, or make it by hand.
Describe the change and let the copilot build it, or do it yourself: point at the page and edit it, or write the variation in code. Every route ends in the same place: one list of changes you can read, adjust or undo before anything goes live.
The copilot works on your real page
It uses the colours, type and spacing already on your site, so what it proposes looks like it belongs there. You do not upload a style guide. There are hard limits too: it cannot delete content a person built, the JavaScript it writes is checked on our side before it is stored, and it never launches or stops a test. Those decisions stay with you.
Or make the change yourself
8 kinds of change by handOnly the copilot can do the other 2: remove an element outright, and attach custom behaviour (a small script tied to an element).
Inside the editor and the copilotAsk in plain English. Review the changes as a list.
Every edit becomes one row in a list you can read, adjust or undo before anything goes live. There are 8 kinds of change to make by hand: text, style, visibility, image, reorder, insert, move, section. Each one can be scoped to the widths it applies at: all widths, mobile, tablet, desktop, a custom range you set.
This is what you end up with
One page and one change, side by side. An experiment can carry up to 4 versions of a page at once: the original and up to 3 challengers. The cap is deliberate, so the statistics can be sized for it.
Choose who sees it.
Show the test to the visitors the question is actually about. There are 13 built-in conditions, combined with and and or, nested as deep as the question needs. Save an audience once and reuse it across experiments and flags.
Then split the traffic between the versions. Before anything ships you can check the audience, the split and the launch checklist side by side.
13 built-in targeting conditionsEvery condition, and where rules runThen wait. Know how long before you start.
No tool can shorten the wait honestly. What a tool can do is tell you how long the wait will be before you start, so you do not talk yourself into a decision the data cannot support yet.
How many visitors, and how many days
Give the calculator three things: the rate you convert at today, the smallest improvement worth having, and the confidence you want. It tells you how many visitors each variation needs, and how long that takes at your traffic. If the answer is three months, it is better to know now.
Not every metric is a conversion rate. The calculator also plans for a per-event rate, a percentile, or a weighted blend of several metrics.
While you wait, the engine strips out noise you already knew about, a technique called CUPED, so tests finish sooner and nobody touches the data. It is on by default, and applied only when it helps.
Metrics, guardrails and how the sums are doneRead the answer, three ways.
You get the raw arithmetic first: one row per side of the split, each with its own visitors, conversions and conversion rate, the lift between them, and the range drawn around that lift. Anyone who wants to check the sums can. Then the engines tell you what the numbers mean.
Most tools give you one formula and a settings panel. AvsB runs 3 genuinely different engines. Each one answers a different question, each is built on published and cited methods, and you can read the same experiment through all of them side by side.
| Question | Bayesian | Frequentist | Sequential |
|---|---|---|---|
| In plain English | Answers "how likely is the challenger actually better", as a plain probability, with no fixed sample size to wait for. | The textbook significance test most stats teams already trust, planned around a sample size you set in advance. | Checking the result every day cannot manufacture a false winner. |
| What you read |
|
|
|
| Where it earns its keep | A direct, exact read on which variation is likely better, useful at low to mid traffic where waiting for a fixed sample size is expensive. | High-traffic teams who want the same framework their stats team already reports on. | Teams that check results often and want to stop the moment a real result lands, without inflating the false-alarm rate. |
When the engines agree, you are done. When they disagree, the result is closer than it looked, which is worth knowing before you act on it. And switching engines cannot rescue a losing test: the verdict stays keyed to the engine you sealed in the plan, and reading it under any other engine is labelled exploratory.
And a grade on how the test was run
Every result with traffic is scored A to F, out of 100. The score is about how the experiment was run, not about how much you wanted it to work. The formula is fixed: the sealed plan, the sample against plan, how often you peeked, traffic integrity, and runtime health. There is no AI in it, and nobody can override the letter.
The grade also works against you when it should: a broken traffic split caps it at D, and a lift too large to believe caps it at B.
- Sample ratio
- Statistical confidence
- Traffic health
Three health checks sit beside every result, so a number that came through a broken split cannot pass as a finding.
How a result is read, end to endShip the winner. Without a deploy.
When a variation wins, make it the default with a feature flag. Turn it up, down, or off from the dashboard, with no redeploy and nothing for your visitors to wait on. Pause a rule and it is off as soon as the change reaches your site.
A flag can carry a simple on or off, a piece of text, a number, or a set of structured settings. A rollout is measured with the same three engines that judged the experiment.
- 4 kinds of value
- A key per environment
- Reviewed after 14 quiet days
Every change has a name on it
Going live is its own permission, separate from building, so the person who writes a variation is not automatically the person who ships it. More than 90 kinds of action are recorded with who did it, the address they worked from, and the values before and after. When something changes on the live site, you can say who changed it and what it was before.
Then do it again.
None of this is complicated. It is just easy to do badly, and you only find out at the end, when you have to decide whether to trust the answer. The loop is four steps, and you have just walked through all of them.
- 01
Say what you expect.
Write the claim down and seal the plan.
- 02
Build the variation.
Describe it, point at it, or code it. It all ends in one list of changes.
- 03
Then wait.
The calculator told you how long. Every peek is counted.
- 04
Read the answer.
Three engines, one verdict, a grade on how it was run.
One test answers one question. The habit is what moves the number.
Now run one yourself.
Free for your first 10,000 visitors a month, with experiments, feature flags and all 3 statistics engines. No card. Not a trial.