A/B testing in the warehouse: judging a change fairly

A new method has been introduced: a second employee now checks received goods against the document by barcode. A month later everyone is pleased because discrepancies have dropped. But that month may simply have brought less stock, and the two least experienced employees may have left. A "before and after" comparison does not compare a change; it compares two different months. A reliable answer comes from an A/B test: at the same time, one group uses the new method and another keeps the old one, and the difference is judged by a number.

How to set up the test

1. One question and one metric. For example: in what percentage of received batches was a shortage, surplus or defect found within two weeks?
2. One change. If you alter two things at once, you will not know which one worked.
3. Two similar groups. For example, receiving gate 1 and gate 2, or even and odd days. Decide by a coin toss which group gets the new method; do not pick the convenient gate yourself.
4. Work out the volume in advance. To tell 4 percent from 7 percent with confidence you need about 900 receipts per group (the calculation is below). Do not stop the test the moment the result happens to look good.
5. Swap the groups halfway. If the new method runs at gate 1 for the first two weeks and at gate 2 for the next two, the difference between the gates cannot distort the outcome.

How to judge the result

Say each group handled 900 receipts. With the new method a discrepancy was later found in 36 of them (4.0 percent); with the old one in 63 (7.0 percent). The difference is 3 points. Could it be chance?

Pooled share p = (36 + 63) ÷ 1,800 = 0.055. Standard error = √(0.055 × 0.945 × (1 ÷ 900 + 1 ÷ 900)) ≈ 0.0107. z = 0.03 ÷ 0.0107 ≈ 2.79. When z exceeds 1.96, the chance of such a difference arising randomly is below 5 percent; here it is about 0.5 percent. The difference is convincing.

Now the same percentages with 300 receipts each: 12 (4 percent) and 21 (7 percent). p = 33 ÷ 600 = 0.055, standard error ≈ 0.0186, z ≈ 1.61. That is below 1.96: a gap like this appears by chance in roughly one test out of ten. So keep testing. The percentages are identical, but confidence depends on volume.

The formula for the volume, worked out in advance: n = 7.84 × (p₁ × (1 − p₁) + p₂ × (1 − p₂)) ÷ (p₁ − p₂)². For 4 and 7 percent: 7.84 × 0.1035 ÷ 0.0009 ≈ 902, that is about 900 per group. The figure 7.84 corresponds to 95 percent confidence and an 80 percent chance of spotting a real difference. As the difference shrinks the volume grows fast: for 4 and 5.5 percent you need about 3,150.

The probability of chance in Excel is 2 × (1 − NORM.S.DIST(z, TRUE)). The separator depends on your Excel settings.

A convincing difference is not yet a benefit

Suppose there are 12,000 receipts a year. A 3-point drop means 12,000 × 0.03 = 360 fewer discrepancies. Say resolving each one (calls, rechecks, correcting the document) costs 180,000 so'm: 360 × 180,000 = 64.8 million so'm. If the second employee spends 4 minutes on every receipt, that is 12,000 × 4 = 48,000 minutes, or 800 hours; at 18,000 so'm an hour it comes to 14.4 million so'm. The net effect is about 50.4 million so'm. Only then does it make sense to roll the method out to every gate. If the cost exceeds the gain, even a convincing difference is useless.

Typical mistakes

• Giving the new method the most convenient area: the difference then comes from the area, not the method.
• Checking the result every day and stopping as soon as it looks good. A chance spike then passes for a "win".
• In the first weeks staff know they are being watched and work more carefully. Assess the first week separately.
• Looking at a single metric and missing side effects: how much longer does receiving take, and are trucks queuing?

Conclusion

An A/B test rests on a simple idea: two similar groups, one change, and a volume and a measure fixed in advance. Only then can you say "it helped". To see discrepancies reliably you need an accurate count: CountLab splits stock-count results by group and presents them with discrepancy analysis and a management report.