Before running an A/B test, work out one number — for most Vietnamese sites it exceeds a year of traffic

A familiar situation: the marketing team proposes testing two versions of a landing page — a different button colour, a reworded headline — running them side by side for a few weeks to see which converts better. It sounds entirely reasonable: rather than argue, let the numbers decide.

Before building the test, there is one calculation worth doing. It takes about a minute. For most Vietnamese business websites, its answer is to stop.

The calculation: how many visitors are needed

An A/B test splits visitors into two groups, shows each a different version, and compares conversion rates. The difficulty is that conversion rates fluctuate randomly — two identical versions still produce two different numbers. To claim an observed difference is real rather than luck, you need enough visitors.

The count needed for each arm follows a compact formula:

• n ≈ 16 × p × (1 − p) / d², where p is your current conversion rate and d is the absolute difference you want to detect.

The constant 16 comes from two standard choices: accepting a 5% chance of declaring a false winner, and wanting an 80% chance of detecting a difference if one genuinely exists.

Take an online shop converting at 2% — a fairly typical figure — with 500 visitors a day:

• To detect a 5% relative improvement (2% to 2.1%): about 630,000 visitors, or 1,259 days.
• A 10% improvement (2% to 2.2%): about 161,000 visitors, or 322 days.
• A 20% improvement (2% to 2.4%): about 42,000 visitors, or 84 days.
• A 50% improvement (2% to 3%): about 7,600 visitors, or 15 days.

What matters most is the shape of that series rather than any single figure. Sample size is inversely proportional to the square of the difference: to detect an improvement half the size you need four times the traffic. That is why the first and last rows are more than eighty times apart.

The practical consequence: small changes cannot be measured at small scale

Changing a button colour, rewording a headline, moving an input field — when these work at all, they usually move things by a few percent. The table says that at 500 visitors a day, such a test needs one to three years to conclude.

This does not mean those changes are pointless. It means an A/B test is not the tool for deciding them. Running a three-week test and declaring version B the winner is reading noise rather than signal — and worse, believing it.

There is a point here that often gets missed: an underpowered test does not return "unclear", it returns a systematically biased result. When traffic is too small, the only differences large enough to cross the significance threshold are large random swings. So if you only publish the tests that "produced a result", almost everything in that collection is selected noise.

The second trap: checking results every day

The calculation above also assumes something almost nobody does: reading the result once, at the end.

In practice people open the dashboard every morning. And when version B is ahead by enough to cross the threshold, the natural reflex is to stop, declare a winner, and ship it. Which sounds sensible — why wait when you already have a result?

Because every look is another chance for randomness to fool you. The 5% threshold holds for one test. Look twenty times and the chance that at least one look crosses the line by luck is far higher.

To put a number on it rather than speak in generalities, we ran a simulation. The method, so you can reproduce it: set up two identical arms, both with a true conversion rate of 2%, 250 visitors per arm per day for 28 days, repeated 4,000 times. Because the arms are identical, every "there is a difference" conclusion is a false positive.

• Reading the result once at the end: 4.2% false positives.
• Reading daily from day seven, stopping the moment the threshold is crossed: 20.1% false positives.

The 4.2% is the sign the simulation is working — it should sit near the 5% nominal level we chose. The 20.1% means one in five "wins" obtained this way is entirely imaginary, on data that contains no difference at all.

Worth noting: both rows use the same volume of data over the same 28 days. The only difference is the habit of looking. No extra traffic needed, no better tooling — just not looking.

So what to do when traffic is short

Four directions, ordered by how easy they are.

One: test bigger changes. The table shows a 50% improvement is measurable in fifteen days at the same traffic. Rather than changing a button colour, rewrite the landing page, change the pricing structure, remove a step from the flow entirely. Large changes both stand a chance of producing a difference big enough to measure, and are worth the effort.

Two: move the measurement upstream. If completed orders are too rare, measure a step closer to the change: button clicks, detail-page views, add-to-cart events. These usually carry tens of times more volume and reach the required sample far sooner. The price is an assumption — that an improvement there flows through to revenue — which is not always true, so write it down rather than leave it implicit.

Three: fix the stopping date first, then stop looking. This is the cheapest of the four: it costs nothing but discipline. Calculate the sample, convert it to days, put that date in the calendar, and open the results only on that day. If operations force you to monitor, look only at how many visitors have accumulated, not at which arm is ahead.

Four: for small changes, decide by reasoning and call it what it is. Choose on design rationale, consistency with the rest of the product, and direct user feedback. Then record that this was a decision, not a measurement. Confusing the two is exactly where confidently worded reports get built on numbers that cannot survive a check.

When an A/B test genuinely is worth running

In fairness, A/B testing is a good tool when conditions allow. Three signs you are in those conditions:

• Traffic large enough that the calculation returns an acceptable duration — usually a few thousand conversions a month, not a few thousand visits.
• A change big enough to expect a double-digit percentage difference, or a willingness to measure at an upstream step.
• Somebody accountable for the discipline of reading results, since as the simulation shows, that is where most tests fail rather than in the tooling.

If all three hold, run it. If not, the approaches in the previous section give truer information — and cost far less than three weeks spent on a test whose result says nothing.

This is also why we wrote separately about a reporting screen only staying alive when a decision is waiting on it: both cases are the same mistake — building the measuring instrument before knowing which decision it serves.

Conclusion

The calculation at the top takes a minute and answers the most important question before any resources are spent: can this test conclude anything at all.

For most Vietnamese business websites, the answer for small changes is no, and that is arithmetic rather than opinion. Knowing it in advance is not pessimism — it frees three weeks to make a change big enough to be worth measuring, or to decide quickly by reasoning and move on.

The second trap is cheaper still to avoid: fix the stopping date before you start, then do not look. That alone brings the false-positive rate from one in five back to the level you chose, without a single extra visitor.

If you would like someone to work out the sample size for a specific test, or to set up the measurement before you need it, get in touch or see KonexForge's Optimization Loop and Data Analytics capabilities.