QUICK SUMMARY

Rapid experimentation, also known as A/B testing, means subjecting program changes or operational improvements to a randomized control trial, rather than relying on assumptions or weak evidence about what works. A/B tests can often be done quickly and at low cost.

STRATEGY DETAILS

Q1. What is rapid experimentation?

Rapid experimentation means subjecting program changes or operational improvements -- from small tweaks on a website to broader adjustments to programs -- to a rigorous test, usually a randomized control trial.  The approach is used a lot in the business word by companies that embrace what one might call “disciplined innovation:” For any new product or business program, they always run an A/B test.

Q2. Why is rapid experimentation important?

Rapid experimentation is important because of a fact of human decision making: Our guesses about what works and about causation are often wrong. Rather than conduct a rigorous test of an operational change, leaders often rely solely on their experience and intuition.

Q3. What's an example from the business word, where A/B testing is best known?

Kohl's Department Store was considering adding a new product category—furniture—and many of the department store's executives were enthusiastic about the revenue potential. Rather than simply going with their gut, however, they ran a test at 70 stores over six months. The results showed a comparative drop in revenue, as other products had less space in the test stores. It was disappointment, but the initiative was dropped.

Q4. Why's an example from government? 

At the federal level, officials at the Treasury Department redesigned the letters it sends to encourage payment by people who owe debts to the government. They made the letters clearer and more personable, and they added a shorter URL for easier online payment. But instead of assuming that the revised letters were more effective, they tested them using a randomized experiment, sending several thousand people the new letter and several thousand others the old one. Within a few weeks, they had the answer: There was no difference in payment rates between the two groups. What happened? Most likely, these indebted Americans had given up opening letters from the federal government. In other words, the department may have needed a better envelope, not just a better letter. These two examples -- Kohl's and Treasury -- are cases when assumptions about what works turned out to be wrong. Of course, there are lots of examples of assumptions turn out to be right, but it's useful to find out with a rigorous test.

Q5. How does one run a rapid experiment?

An A/B test or rapid experiment is simply a randomized controlled trial (RCT) applied to an operational improvement. RCTs that test operational changes can often be done at very low cost.

Q6. What does it take to build an experimental culture that embraces rapid experimentation?

Public leaders need to be willing to do three things:

  • Subject their hypotheses and those of their organizations to rigorous testing, rather than relying on guesses, weak evidence or the loudest voices in the room, whether constituent groups or lobbyists. Legislatures can help by requiring and funding rigorous program evaluations.
  • Test those hypotheses using credible, reliable methods, such as randomized trials or well-designed quasi-experiments.
  • Act on the evidence that is produced.

ADDITIONAL RESOURCES

CUSTOMIZED ASSISTANCE

Please contact us if your organization needs help in planning or conducting A/B tests, or in building capacity within your agency or organization to conduct them.

PDF VERSION OF THIS PAGE (click here)

Credit: This page was adapted from Andrew Feldman and Jim Manzi, “What Government Really Can Learn from Business," in Government Executive.