Running 1,500 experiments a year is not about tooling. It is about governance, instrumentation and a culture that treats every release as a hypothesis.
Running 1,500 experiments a year sounds like a tooling achievement. It is not. The tooling is the easy part. What makes experimentation work at enterprise scale is governance, instrumentation and a culture that treats every release as a hypothesis rather than a decision already made.
At HSBC, the experimentation programme was less about the platform and more about the operating discipline around it: a shared definition of what counted as a valid test, instrumentation trustworthy enough that nobody argued about the numbers, and a cadence that turned results into decisions quickly enough to matter.
Scale introduces problems small teams never face. Tests interact. Traffic is finite and has to be allocated like a budget. Statistical literacy has to be distributed, not concentrated in one analyst. And leadership has to genuinely accept that most experiments will fail — because a programme where everything wins is a programme that is measuring the wrong things.
The payoff, once the discipline is in place, is compounding. A 400% optimisation uplift is not one brilliant idea; it is hundreds of small, validated improvements stacked on top of each other, by an organisation that finally trusts its own evidence.
Gerard Short
Enterprise Transformation Executive