Infinite Variants, Same Traffic

The cost that collapsed was the wrong cost

A model can now write 10 headline variants, five layouts, and a fresh call-to-action in the time it takes to open the ticket. The marginal cost of producing an experiment arm has gone to roughly zero, and every experimentation vendor is selling that as the breakthrough. Optimizely, Statsig, and Eppo have all shipped AI-variant tooling in the last year, and growth teams report shipping more tests than they did a year ago. That volume is the wrong thing to chase, because the constraint that actually gates learning never moved.

More tests is not the goal. Learning is the goal, and cheap variants do nothing for it. The binding constraint on an experimentation program was never the number of ideas anyone could generate. It was the traffic available to evaluate them, and that number did not move.

The cost collapse is real and it changed the wrong line item. Variant production was already cheap relative to the thing that gates a program. Making it cheaper buys more arms to split a fixed pool of users across, which is the opposite of what a program under a power constraint needs.

The constraint that didn’t move is arithmetic

Traffic is fixed on any given surface for any given window. A pricing page gets the visitors it gets. A checkout flow sees the sessions it sees. You can run one test on that traffic or you can run a test with more arms, but you cannot conjure more sessions by generating more variants.

Split fixed traffic across more arms and two things happen, both bad. First, each arm gets fewer users, so each comparison is underpowered. An underpowered test does not fail loudly. It returns a result that looks like a result and is mostly noise. Second, every additional arm is another comparison against control, and every comparison carries its own false-positive risk. This is the multiple-comparisons problem, and the standard treatment of it in experimentation goes back to Ronald Fisher: run enough comparisons at a 5 percent threshold and you will find “significant” effects that are pure chance at a rate you can predict in advance.

The math is unforgiving. Test one variant against control at the usual threshold and you accept a 5 percent chance of a false win. Test 10 variants against the same control and, without correction, the chance that at least one crosses the line by luck alone climbs past 40 percent. Correct for it properly and you need more traffic per arm to clear the higher bar, which you do not have, because you spent it on arms.

This is not an opinion about experimentation culture. It is arithmetic that holds regardless of how good the variants are. AI can make every arm a plausible idea. It cannot make the traffic pool bigger, and it cannot repeal the penalty for asking a fixed sample too many questions at once.

The failure mode is a program the org stops trusting

Here is how this plays out over two quarters. The team runs more tests, because tests are cheap now, and reports more wins, because underpowered multi-arm tests manufacture wins. Leadership sees a rising count of shipped improvements. Everyone feels productive.

Then the wins do not show up in the aggregate metric. Revenue per visitor was supposed to move; it drifts. Someone re-runs a shipped winner as a clean two-arm test and it comes back flat. Someone else notices that the sum of the quarter’s claimed lifts, if real, would imply a conversion rate above 100 percent. The replication rate quietly craters, and a replication rate that craters is how an experimentation program loses the confidence of the organization it serves.

That loss is the expensive part. A program’s real asset is not its test count. It is that when the team says a change worked, product and finance believe them enough to act. Trust in the number is the thing being spent here, and it is far more expensive to rebuild than to protect. Cheap variants let a team spend that trust faster than they ever could before, and most of them will, because the tooling makes the spending feel like progress.

Spend the freed capacity on the question, not the arm count

The right response to cheaper variants is to hold arm count roughly where it was and redirect the freed capacity upstream, into the part of the pipeline that AI did not touch. Here are concrete rules for a program that wants to learn more, not test more.

Cap the arms. Two arms is the default. Three when you have a genuine reason to compare two distinct hypotheses, and the traffic to power all three at the corrected threshold. Beyond that, you are buying noise.

Raise the bar on the hypothesis. A test should encode a belief about why users behave a certain way, and the variant should be the sharpest possible probe of that belief. Ten AI-generated headlines that all say roughly the same thing are one hypothesis wearing 10 costumes. One headline that tests a specific claim about what the user is worried about at that step is worth more than all 10.

Sequence deliberately. The freed capacity buys you the ability to run a considered second test after the first resolves, informed by what the first taught you. A program that learns is a chain of tests where each one is chosen because of the last. A program that runs 10 arms at once has thrown that chain away for a lottery.

Power the test before you launch it, not after. If the fixed traffic on a surface cannot power a clean comparison in a reasonable window, the answer is a bigger swing worth waiting for, not more small swings run in parallel to fill the calendar.

Freed capacity should raise the quality of the question, not inflate the number of guesses evaluated against a sample that cannot afford them.

The bandit objection is real and it doesn’t rescue the arm count

The sharpest objection: multi-armed bandits and sequential testing genuinely relax the power constraint. A bandit shifts traffic toward arms that are performing well as data arrives, so you waste less of the fixed pool on obvious losers. Sequential methods let you stop early when the signal is clear. Both are real, both beat naive fixed-horizon tests for many problems, and a team using them well can evaluate more variants per unit of traffic than one running a static split.

But they change the allocation of information, not the total amount of it. A bandit still has to gather enough evidence on each arm to know it is a loser before it can stop feeding it traffic, and with many arms and a small pool it gathers that evidence slowly and noisily. The teams reaching for bandits to justify more AI-generated arms are mostly the teams without the traffic to make a bandit converge before the quarter ends. On low-traffic surfaces a bandit with 10 arms spends most of its budget exploring and produces the same underpowered conclusions as the fixed split it replaced, dressed in better math.

So the objection is right that the frontier moved. It is wrong that it moved far enough to make arm count free. The constraint is softer than it was, and it is still the constraint. The team that treats cheap variants as permission to ask more questions of the same traffic will learn less, ship more that does not replicate, and spend down the one thing an experimentation program exists to accumulate.