Fail Early and Often

By Seamus McMahon

To extract the benefits of innovation and limit its risks, companies need to elevate testing to a science.

Innovation has gained a fairly dangerous connotation of late. Bold ventures with Internet-only banks have fizzled, as have cross-sell campaigns designed to lift revenues. Broadly in the economy, more than 75 Internet startups have failed this year, and investors are demanding that companies give a far more rigorous accounting of their new projects. Instead of bold pioneers opening new markets, the imagery has shifted to depleted gamblers fleeing the casinos.

This reversal underscores an important fact about corporate innovation, which is that it requires far more than guesswork and emotion. From a managerial standpoint, feverish pursuits and retreats to the mental bunker are equally deficient, because in both cases there's a lack of specificity about the circumstances under which projects are failing and succeeding. A tool is needed to illuminate project dynamics so that decision-makers can make the best possible use of corporate resources.

That tool is rigorous testing. Although many companies are either unfamiliar or struggling with the concept, it is time that testing officially joins the corporate lexicon as a key element of project design. Historical data often sheds little light on what customers will do next. Surveys show what customers want but not what works best. Testing, by contrast, places new business propositions in a laboratory setting where processes can be carefully observed, modified and re-tested.

Such preparation sharply boosts the odds for a successful full-scale launch. Indeed, this methodology is applicable to a variety of customer-related activities, including prospecting, cross-selling, retention, service and billing/collection. By no means is this a stretch for large-scale companies, whose every move can affect throngs of customers.

From a larger perspective, testing also can provide vital information for strategic decisions. Assumptions, even highly educated ones, about what the customer wants can prove an unstable foundation for pivotal decisions. Executives who proceed unilaterally risk expending enormous energies to acquire product-manufacturing capabilities and enter new markets, for example, only to find themselves out of step with customer demand and out of favor with shareholders.

Realistically, though, managers must come to grips with some serious issues if they are to get a payoff from rigorous testing. Objectivity is crucial, and achieving that often will entail involving people from outside a given business unit, which some managers will find threatening. A certain blend of resources and expertise is needed, and capabilities must either be built internally or outsourced. And developers will have to adjust to unfamiliar quantitative techniques.

But the need for rigorous testing isn't going away. Strategists know that cost-cutting alone won't carry the day; they must innovate to assure organizational viability. They also know that frenzied expansion, of the sort attempted by many now-extinct dotcoms, can be an invitation to disaster. The middle path is disciplined innovation, which entails careful testing before rollout. Instead of "build once and hope for the best," the initial approach is to "fail early and often" — and be sure it's done in a controlled environment.

Inside the Laboratory

Pharmaceutical companies for decades have employed decision sciences and rigorous methodologies at each stage of drug development, evaluation and rollout. In the lab, researchers seek out specific therapeutic effects. They combine thousands of compounds to test for benefits in small-scale processes, with much rapid iteration. Promising combinations are tested for efficacy and safety. And dedicated facilities are utilized to test the ability to expand production, considering issues ranging from contamination to temperature control.

This level of care seems daunting enough to replicate in other industries, but the developmental challenge is even greater in financial services, where volatile competitive dynamics also must be taken into account. It's no accident that leaders in the credit card industry, such as Capital One Financial Corp. and MBNA Corp., are renowned for their expertise in conducting disciplined innovation in a rapid-fire setting.

Indeed, the proliferation of card-like competition in all sectors of financial services is the driving force for testing as a corporate competency. Customers are gaining infinitely more choices as they shake off the constraints of geographic boundaries, physical branches and paper-based transactions. Providers, in turn, must learn to think on their feet. Testing can't be viewed as a one-time activity, but rather as part of a continuous process that helps the institution align itself with the market and meet the competition.

One critical factor in incorporating this approach in financial services is establishing a complete setting. It's not only the solidity of the abstract concept, but also how well it works in practice. Accordingly, a truly thorough test "operationalizes" a concept by replicating all of the processes and interactions that bring it to life. This includes customers, company representatives, variations of offers, various delivery settings, databases, and even the scripts that reps will use for certain presentations and responses.

A broad perspective enables developers to assess ramifications in places where those effects might not be instinctively apparent, both inside and outside the organization. Internally, second-order effects such as cannibalization of profits from other product streams need to be taken into account. External considerations might include ensuring the efficacy of suppliers, or determining whether call center support is acceptable.

Another critical factor is rapid pilot testing accompanied by statistically sound evaluation. The idea is to begin with a few promising approaches (hypotheses) and then test a variety of permutations. With every change in ingredients such as price, features and sales script, measurements are taken on customer responses, logistical implications, profitability implications, and so on. These measurements are then used to refine the concept, which then is subjected to further testing. Successive cycles of feedback and revision ultimately lead to the creation of prototypes fit for larger-scale rollout.

This is not to suggest that the judgment of skilled professionals no longer has a place. To the contrary, the starting point for hypothesis testing rests on lessons the organization has learned from its own experiences. Science is something that supplements — not obliterates — the best practice of intuition and experience.

In any case, rigorous testing is an improvement over conventional developmental methods. In one common approach, strategic thinkers develop and implement propositions almost entirely on the basis of their planning, often corroborated by exhaustive market research. That's like composing the score of a symphony and then attempting to play it, without rehearsal or revision, in a crowded music hall. Experience shows that "theoretical testing" is rarely sufficient to assure the success of innovative propositions.

The same applies to firms that place all their faith in the past behavior of their customers as revealed by the miles of information generated from their data warehouses. The wonder of this information is that it generates a million observations about customer behavior patterns. Unfortunately, the data cannot say which variables cause which events to happen. Nor can it tell the observer whether the same thing would happen again — especially given the rapid pace at which business circumstances are changing.

The market research model and the behavioral model are thus inherently limited, and it is a tribute to the intuition of business leaders that innovation succeeds as often as it does.

Terms of Engagement

That said, there are many challenges in setting up a proper testing facility. A lot of pieces have to come together in a certain way if the exercise is to be worthwhile. For example, it can be difficult for creators to objectively assess their own work, raising the question of whether testing facilities should be housed outside the respective business units. A certain blend of people, processes and technology is required. And managers have to adapt to the new procedures.

On the surface, it seems to make sense to operate a testing facility in the same environment where services are developed and produced. But that approach can prove counterproductive. Production environments are usually optimized for consistency and efficiency, as are the mindsets of the people who work there. Even with the best will in the world, it can prove impossible for a group of people to design what they perceive to be the best crafted products and services and then perform the truly objective tests needed to verify the circumstances — if any — under which their creations actually work.

Instead, testing is best done in a separate learning environment. A completely different atmosphere is needed, one that can rigorously replicate the key real-world circumstances in which the services would be offered. That entails assembling the right combination of resources necessary to ensure the most scientific levels of testing rigor. In particular, such an environment would always need the following resources to ensure the optimum success:

The right people. A testing environment draws on four kinds of staff expertise. The team begins with strategic and marketing experts, who are responsible for developing the lines of services. Then there are the technical experts, who envision, construct and validate the analytical tools necessary to measure cause-and-effect relationships between variables. Technology experts build the databases and communications tools. Finally, customer interaction specialists evaluate live research and feedback.

The right process. The people involved in this environment need to develop the most efficient and disciplined process for quickly testing and retesting initiatives through changes in variables. Their mission is to test a prototype to extremes, adjust variables, retest to extremes, and so on until the optimal balance of variables is found. Instead of "build once and hope for the best," the new approach is "fail early, fail often."

The right technology. To test multi-channel propositions as faithfully as possible, you have to replicate them in the testing environment, from the phone and Web customer interfaces, to the campaign management and analytic tools, through to the databases and scripts used in live situations.

Even after all of these capabilities are in place, many companies still will experience initial difficulties in taking advantage of them. For managers weaned on "brave intuition," rigorous testing of pet hypotheses can feel very uncomfortable. Others will be reluctant to share ideas and knowledge. It can take a long time for key managers to accept these techniques and consistently apply them in making important decisions.

To be sure, there are quick payoffs from this form of testing, flowing to the bottom line within a matter of months as margins and growth rates improve and losses on failed campaigns are minimized. The really serious returns are slower in coming, however, taking up to two years. That's the amount of time it might take for this form of scientific testing to be absorbed into a company's business processes.

There is a political dimension as well. This is one more area where a new way of doing things threatens established hierarchies and the way people are valued within the company. Traditional star developers may resist out of a fear of losing status. The heads of business units may resist out of a fear of losing control. Rather than ignore or fight these tendencies, senior managers should prepare themselves by emphasizing the individual and organizational rewards that can be gained by adopting the new approach.

Also, senior managers will have to make some crucial decisions about how to avail themselves of testing capabilities. Companies will have to build their own laboratories if they want to lift development activities out of narrowly focused business units while still keeping the process within the confines of the organization. But it doesn't always make economic sense for individual companies to build and maintain such capabilities.

Alternatively, companies can outsource testing activities. Parallels in the manufacturing world include BMW Group, which uses the labs of Porsche Group to test the use of high-tech material in advanced applications, and General Motors Corp., which uses subcontractors to test elements of ergonomic design. Engineers at BMW and GM retain the overall architecture/solution. But they outsource testing to ensure objectivity and speed, and to simplify the decision process.

The Business Case

It's important to keep sight of the varied potential advantages of testing when weighing the options for acquiring and configuring such capabilities. Testing enhances and accelerates the developmental process, can be used in diverse areas, and can yield powerful results. But it poses its own challenges.

The decision to relocate testing/ developmental activities out of the business units and into an independent environment can be a momentous one, but the potential payoffs are many. A dedicated unit can help ensure that customer initiatives are founded on objective, rigorous testing. It can build a rich database of empirical results. It allows live interaction with customers and prospects and accelerates the process of design, test and scale-up. And it lifts a burden from end users, allowing them to focus on desired results.

As companies grow more comfortable and adept with scientific developmental processes, they will find them to be applicable in a variety of spheres. After all, companies are constantly devising new ways to acquire customers, sell more products to each client, preserve valuable relationships, refine service and improve billing and collection. There are testing activities that will more than pay for themselves in each of these five areas.

Testing also can play an important role in strategy formulation. For example, one strategy that's in conspicuous use today is vertical integration, where an institution buys an entire company in order to obtain new product and service capabilities. A prominent practitioner is Citigroup Inc. But there's a strong countervailing force, aggregation, which allows providers to establish "best of breed" product sets through partnerships and alliances. Obviously, a lot is riding on what the customer really wants, and testing certainly seems preferable to conjecture in making momentous "buy-or-network" decisions.

In fact, serious setbacks can befall companies who eschew rigorous testing. Nowhere has this been more amply illustrated than with some of the much-hyped Internet-only banks in the United States, whether launched by newcomers to the banking industry or long-established players. In both cases, marketing budgets running into tens of millions of dollars were spent to little effect, mainly because the protagonists never verified general demand in the first place, much less tested specific market segments or value propositions.

Cross-selling is another area requiring far more testing/developmental activity. Innumerable institutions have pinned their hopes for growth on selling more to each customer. As these efforts have fallen short, executives have had a tendency to blame execution, citing factors such as poor systems integration or inadequate training. But in many cases, the true culprit is misalignment with what the customer wants. Managers didn't spend enough time test-ing various combinations of products, delivery alternatives and pricing arrangements before launching cross-sell campaigns.

There's a better way. After a period in which rigorous testing has been in place, a library of test data can be built up and acted upon, and a business can truly distinguish itself. Services can be devised, tested and implemented with confidence that they have been constructed in response to rigorous testing, rather than the fashionable whims of investment trends or as a "me too" response to rival maneuvers. That's the sort of disciplined innovation that will be required of market leaders.

Mr. McMahon is a managing director of Novantas, a consultancy based in New York.

Copyright © 2003 by Banking Strategies, published by BAI.

back to top