The Why Axis: Hidden Motives and the Undiscovered Economics of Everyday Life (32 page)

Netflix got so many complaints that it had to hire extra customer service employees. The company’s stock plunged 51 percent. Then, in September of 2011, CEO Reed Hastings apologized to customers and announced that Netflix was going to try to correct the situation. How? By splitting the company into two operations: one, called Qwikster, would be the mail-delivery business, run by a new CEO; the other would be the online streaming service, called Netflix.

This announcement made customers even angrier. Now, subscribers to both streaming video and DVDs would see two separate charges on their credit card statements and have to log on to two different websites. The stock dropped another 7.4 percent.

Realizing they’d made things even worse, “The Netflix Team” sent out the following email to customers in October 2011:

         
Dear [Customer’s Name Here],

         
It is clear that for many of our members two websites would make
things more difficult, so we are going to keep Netflix as one place to go for streaming and DVDs. This means no change: one website, one account, one password . . . in other words, no Qwikster.

Netflix thought that some customers would leave; but they were shocked that close to a million customers would drop Netflix. By this time, Netflix was being universally slammed as a badly managed company. Even
Saturday Night Live
wound up making fun of it.
²

To see just how costly it was not to experiment, take a look at Netflix’s stock price, pre– and post–mess-up in 2011:

We tell you this story because the company could have avoided the loss of billions of dollars and the damage to its brand if it had run some simple field experiments. Rather than coming up with a national scheme to thrust upon customers, and instead of relying on rough-hewn ideas (based on the intuition of some very smart people on the board, maybe a few focus groups, or some expensive consulting firms), all Netflix had to do was run a pilot of their grand plan in a small portion of the country—say, San Diego—and then study its customers’ reactions. The small-scale experiment could have saved the company lots of money without cutting its value.
Netflix might have lost a few customers in San Diego, but it would have had a chance to improve the plan (or maybe cancel it altogether) and remain the market leader. Even if this experiment had stirred up some negative attention, Netflix executives could have explained it was a local snag. The damage would have been much smaller and the experiment worth its weight in gold. Netflix has since recovered, and we expect that, given its product base and strong customer profile, the company will continue to do well, especially if it improves its performance by conducting field experiments.

When we discuss experimentation with business leaders, they usually reply by saying, “Tests are expensive to run.” After we point out that they are not, we turn the tables on them by showing how expensive it is
not
to experiment, as the Netflix example shows. We politely explain that every day that they set suboptimal prices, place ads that do not work, or use ineffective incentive schemes for their workforce, they effectively leave millions of dollars on the table.

Of course, many businesses do experiment, and often. Businesses always tinker with the machine and try new things. For instance, Apple’s Steve Jobs was constantly experimenting with design and with new ways to sell products. The problem is that businesses rarely conduct experiments that allow a comparison between a treatment group and a control group. Jobs’s launch of the iPod and of the iTunes music store revolutionized an industry. But for years, Jobs insisted that recording artists and record companies charge exactly 99 cents per song on iTunes. Defending any justification Apple can offer for this policy is difficult, however. The company never compared the impact of iTunes prices on its sales of songs and iPods. And in the absence of solid evidence, Apple executives turned to their intuitions. They did well with this strategy, but could they have moved “from good to great,” as author Jim Collins puts it, through experimentation?

Put differently, say that you have a serious illness. You go to your doctor, and she prescribes a new treatment regimen for you. When you ask what evidence she has for trusting this treatment, she says, “It’s my intuition.” In such a case, you’d probably leave and never come back, because you prefer to entrust your life to someone whose medical decisions are based on scientific evidence.

How does making the right business decisions differ from choosing the right medical treatment? You might say lives aren’t at stake, but executives who are paid millions of dollars a year to sign off on decisions can cost people their jobs and the economy billions. Business experiments are research investigations that give companies the opportunity to get fast and accurate data regarding important decisions. By manipulating various factors in the environment, companies can better understand the causal relationship between a change in strategy and a response in consumers’, competitors’, employees’, or other stakeholders’ behavior.

Field experiments in business are also different from other research efforts—say, focus groups—because participants make real-life decisions often without even knowing that they are part of a study. When designed properly, business field experiments can provide invaluable insights and reveal surprising results, which the company can then implement on a larger scale. In this chapter, we tell the story of two great executives who have guided their companies’ futures with field experiments. Along the way, we mix in experiments that we have conducted with these and other firms.

Innovation at Intuit

Intuit, the Silicon Valley–based firm famous for its QuickBooks and TurboTax software, has spent years building experimentation into the core of its being. “We used to make decisions through managerial analysis and opinion, and from the top-down,” says founder and
chairman Scott Cook. “Now we let our small, rapid-fire experiments make the decisions for us.”

In the old days, Intuit was run like most large organizations. Product-development folks would come up with ideas. Managers of the business units would pull together data from focus groups and other research, run some analysis, stuff their findings into PowerPoints, disseminate the information to the rest of the company, and their higher-ups would decide whether to fund the project or not. But Cook began to understand that this way of doing work was like walking in concrete shoes. “I started to be convinced that experimentation was the solution to two problems,” Cook says. “The first was how to get a large, successful company to be agile and innovative, because the larger and more successful a company it is, the less innovative and entrepreneurial it can become. The second problem was that the decisions made in the old-fashioned way were often wrong.”

Intuit trained people in “design thinking,” a methodology for investigating problems (especially fuzzy, vague ones), gathering information, and coming up with creative solutions. Design thinkers use a holistic approach, bring creativity to their work, and then innovate new approaches to problems. A small group of design thinkers and executives trained 100 leaders in the organization to run experiments that tested assumptions and hypotheses; they gathered data and came up with solutions. And these leaders taught people who worked for them to do the same. In addition, 150 “innovation catalysts” throughout the organization work in all the firm’s departments to drive this culture of experimentation. Today, everyone is encouraged to toy with new ideas using the same, Galileo-like, scientific experimental methods that we use in our work.

In the old days, people in the
Turbotax.com
division ran seven experiments a year. Today they are running 141 rapid, low-cost
experiments during tax season on a weekly cycle, beginning on Thursdays. They test the idea, run the experiment, read the data, tweak the experiment, and the following Thursday they test again. The rapid experimentation cycle “uncorks innovation and entrepreneurship,” Cook says.

As a company, Intuit frees its employees to spend 10 percent of their time working on projects of their own invention. Today, Intuit experiments whenever possible on a small, cheap basis as the core of the discovery process. Employees who come up with innovative ideas must prove that the concepts work by tracking results from real customers; the most promising ideas rise like cream to the top. In this way, Intuit developed SnapTax (which prepares taxes on a camera or a mobile phone); SnapPayroll (which enables employers to pay their employees via mobile phone); and an Intuit Health Debit Card, which offers health coverage to small businesses that cannot afford health insurance for their employees; and more.

Very often, such experiments result in new product features. For example, the development team used specific experimental questions about their tax situations; based on the answers, the software could recommend either the standard deduction or the itemized deduction. Testing showed that the feature could reduce the time it took to finish tax forms by 75 percent, so all subsequent versions of the product incorporated the new feature, called “Fast Path,” into its free “Federal edition” of the TurboTax software.

Intuit’s development teams also created an “Audit Support Center” that helped guide all customers through the audit experience, just as if they received an audit letter from the IRS. Testing confirmed that more customers started and completed their TurboTax filings when the feature was presented on the website. “Our customer conversion rate—the number of people who purchase the product after shopping around on the Internet—is up 50 percent in six years,” says Cook.

Employees are also encouraged to come up with solutions to serious social problems. In one instance, a team in India developed a service for Indian farmers called “FASAL” (“harvest” in Hindi). The team members had observed that farm families—comprising half of Indian society—were so poor that they didn’t have access to some of the most basic necessities. How, the engineers wondered, could they make these farmers’ lives better?

The team from Intuit conducted its own study, observing the poor farmers both in the fields and when they went to market. Most farmers had access to just one or two markets at a distance from each other, and they had to work through one middleman at each market to get a price for their produce. The middleman sat under a cloth and signaled the price through hand gestures. There was no transparent pricing, and the system worked against the farmers. But the farmers had one big thing going for them: they had cell phones.

So the engineers conceived of a cell phone texting application that let farmers know what the middlemen from a variety of markets were offering. In just weeks, the engineers tested the concept with a quick-and-dirty experiment, hand-typing text messages to 120 farmers that told them which markets could secure them better prices for their crops. The test worked, and farmers began adopting the application. Today, the FASAL service is helping 1.2 million farmers out of poverty.

“FASAL is not a charity. We run it as a business, so we can attack head-on one of the most pernicious problems of the developing world, rural poverty,” says Cook. “We go out and look for the biggest problems we can solve, and a number of them are social problems. We attack these by running low-fidelity, rapid experiments.”

Working with Intuit, we now have dozens of field experiments under way that promise to shed light on what works and why. We suspect many will help move Intuit’s bottom line. Intuit is a great company because the field experiment gene is built into their DNA.

Interventions at Humana

Another company that likes running field experiments is Humana, the giant health benefits firm that started out as a chain of nursing homes and hospitals. “I like to know what makes things hum,” says Mike McCallister, Humana’s affable, mustachioed chairman and CEO. Indeed, McCallister is one of those guys who is constantly thinking about better ways of doing things. In fact, he thinks a lot more like an entrepreneur—or even a field economist—than a CEO. Whereas others may trust their intuitions, he trusts his counter intuitions. “I try to find what is doable,” he says. “People assume that things are not doable, but who is to say they aren’t? Let’s find out!”

For example, in the old days before Humana was a health benefits provider, it owned hospitals and medical buildings, and McCallister was then in charge of the medical offices. The medical offices were money losers; but hospital pharmacies were money-makers. McCallister’s bright idea: attach some pharmacies to medical offices and see how they performed financially when compared to medical offices that didn’t have attached pharmacies. Lo and behold, the medical offices with the pharmacies proved more profitable. Evidence in hand, Humana expanded the pairing across its medical offices and made money. Nobody had ever tried this kind of thing before. It just wasn’t “done” at Humana, or elsewhere in the healthcare industry for that matter. Breaking the mold takes guts, we argue, and evidence from a field experiment that gives you confidence your idea is actually correct.