“Le mieux est l'ennemi du bien.” – Voltaire
Inevitably, while writing this blog, I have a coffee in front of me. Over the last few years, in the course of researching and writing (then rewriting) two books and a host of accompanying blogs, I have consumed a lot of coffee. It just so happens that this particular coffee was provided courtesy of British Airways, as I’m writing this while making good use of some travel time in Europe, but most my writing is done at home. I therefore have good reason to be thankful to Nestlé and their introduction of Nespresso capsules – either they have saved me a lot of time over the last few years, or they have saved me from a lot of instant coffee. I’ll return to Nestlé later in this post as an example of how bravery is important during testing.
First, though, I’ll point you to a 2019 paper I co-authored about controlled field testing of a new directional drilling technology and how it was deliberately split into manageable blocks. A suitable test facility was secured and booked, and testing carried out to a defined cadence. Tests were conducted approximately once per month, with one week of testing followed by an approximately three-week period in which issues seen during the tests were diagnosed, and solutions identified and implemented. This strict cadence was used to keep the project on a rapid schedule and proved very effective in terms of identifying, rapidly deploying and testing improvements. It changed the nature of the conversation with the engineering team when we returned to base. The direction was not “Lick your wounds, find out what went wrong, create solutions, and let’s repeat the test when we are ready”. It was more like “Right, we are going back in three weeks – fix as much as you can but whatever happens you must be ready on the agreed date”. This introduced more urgency, more tension, more risk – and forced prioritisation. Anything that was “perfect” according to the Voltaire quote that heads this blog was rejected, and the team focused on “good enough”. It won’t surprise you to read that not every problem was solved in each of the three-week periods, but enough improvement was made to ensure that the next test was meaningful and drew out additional problems and failure modes. This strict cadence was used to force the pace and to keep the project to a rapid schedule and proved effective in terms of identifying, rapidly deploying, and testing improvements. When combined with a willingness and a desire to make things fail during early testing, such a cadence, or drum beat, with imposed “ready” dates for testing can lead to rapid learning and therefore rapid development and early commercialisation with much lower risk.
To stay on track, and to accelerate learning as much as possible, you need to think of your testing schedule as being fixed in time. If technical progress is delayed, do not succumb to the temptation to reschedule your testing. Instead, try to figure out what you can deliver on time to your test location, or to your testers, that will still give you useful results, even if not everything is tested at the first pass. I guarantee that this will provide you with the fastest route to market.
The most important thing while testing is to reveal as many problems as possible as quickly as possible and, as perverse as it sounds, you should aim for rapid failure and not success as you plan your testing – allowing you to learn and rapidly recover. Identify somebody who is accountable for delivering as much learning as possible from a given test and imbue them with the authority to make appropriate decisions as testing progresses without having to refer to bureaucratic approval processes. During testing, make the buck stop with your test coordinator or test manager. Make it clear to them that, as described by Capodagli and Jackson (2010) “it is easier to ask for forgiveness than permission”.
It’s important, though, that any failure must be productive. Gary Pisano (2019) pointed out that successful organisations understand the difference between productive failure and incompetence. Failure that results from genuine uncertainty about how new technologies or business models will work is acceptable and should be celebrated because it teaches you the way forward, and it can do so much more quickly than had you not failed. Pisano suggested that it’s really the learning we should be celebrating, not the failure – and he makes a good point. He went on to remind us that failure resulting from poor management or poor analysis (remember the Ford Edsel?) is nothing more than incompetence and should be dealt with ruthlessly. I agree with him.
And, of course, testing is not just about proving technical capability. Returning to my coffee, in The Business Model Navigator (2014), Oliver Gassmann, Karolin Frankenberger and Michaela Csik wrote about how Nespresso spent its first few years looking like a product failure for Nestlé. Then, because of testing a new route to market, the company hit on the right business model: selling capsules via the Internet or in its own dedicated Nespresso stores and appealing to domestic consumers instead of just aiming the product at businesses. The authors used this example to illustrate their recommendation to de-emphasise traditional, formalised business models, instead taking a rapid prototyping “design-prototype-test cycle” to try new business model ideas, test them quickly, and thereby accelerate learning. You’ll see that this is very similar to the approach I’ve suggested for testing new technologies.
Once your offering becomes commercial, you need to think a little differently about ongoing testing but there are still opportunities to learn rapidly and to fine-tune and improve what you are providing. Stefan Thomke, professor at Harvard Business School, wrote in 2020 about Booking.com and its approach to testing: anybody in the company can test anything by modifying the company’s customer-facing website without asking for permission. Think about this. Can you try out a new idea on one of your customers without permission? It sounds like a crazy idea, but it can be powerful. Of course, there are some controls. In the case of Booking.com, the modified user experience is only seen by a relatively small number of randomised customers in an A/B test – a randomised controlled experiment to compare two versions of the interface. The experiment is watched carefully and is quickly withdrawn if metrics are adversely affected. What Booking.com achieves by allowing this, and a driver for its success, is a rich and diverse seam of ideas that it might not otherwise be able to mine. To be clear, it benefits from these ideas because of the trust it places in its own employees to do the right thing when they are experimenting live with real customers. Perhaps it’s easy to do this with a website, where you can quickly introduce and withdraw experiments and where you have a large pool of customers to experiment on?
In my experience, I have found that you can also do this with hardware projects, and I’ve done so myself when developing and testing several different products. I have found that if you trust people, including people outside the direct control of the engineering function, to introduce changes that could improve the performance of a product, service or business model – and you give them the authority to do so – you will be more successful than if you don’t offer that trust and if you try to maintain some form of centralised control over what gets tried out.
Trust people, and allow them to fail, and your offerings will get through testing and to market much more quickly and much more effectively.
Want to Know More?
The second volume of my book: Strategy and Innovation for a Changing World – Part 2: Sustainability Through Velocity is now available to pre-order in paperback or as an e-book. The topics introduced in this blog are expanded on and covered in much more detail in Chapter 6: “Getting it Done: Doing Things Right (and Quickly)”. I hope this short introduction has whetted your appetite!
References and Further Reading
Capodagli, B., & Jackson, L. (2010). Innovate the Pixar Way: Business Lessons from the World’s Most Creative Corporate Playground. New York: McGraw Hill.
Clegg, J., Mejia, C., & Farley, S. (2019). A Paradigm in Rotary Steerable Drilling – Market Demands Drive a New Solution. The Hague: SPE/IADC International Drilling Conference and Exhibition.
Gallo, A. (2017, June 28). A Refresher on A/B Testing. https://hbr.org/2017/06/a-refresher-on-ab-testing
Gassmann, O., Karolin Frankenberger, K., & Csik, M. (2014). The Business Model Navigator. Harlow: Pearson.
Pisano, G. (2019). The Hard Truth About Innovative Cultures. Harvard Business Review January–February.
Thomke, S. (2020). Building a Culture of Experimentation. Harvard Business Review, March–April, 40–48.
© 2022 J M Clegg Ltd
Image © lena_rx7 – stock.adobe.com
6 thoughts on “The Appeal of a Pod”
Thanks John, a nice thought piece.
Given the Bookings.com angle, what do you make of Mr Musk’s live testing and rapid removal of new features on Twitter?
Quick iterative design changes in a live environment can damage reputation quickly, and prove hard to recover from? Does that matter in the long run? Seemingly not to Mr Musk. What’s your take on this ‘live’ situ?
Thanks J-P, that’s a great question and I think Amy Edmondson nailed the answer a few days ago, see her tweet.
The thing Bookings does right is to limit the modified user experience to a small and (as I understand it) randomly selected subset of visitors to the site, so that experiments are done largely out of the gaze of the public. Experiments that don’t work are quickly and quietly withdrawn. That’s a lot more effective, and more intelligent, than continually changing the experience for everyone with the attendant confusion and bad user experience. I personally think there is a clear and present danger to Twitter’s reputation. How many of us are questioning our use of it right now?
This philosophy is not just for testing either. At Wytch Farm BP brought in some guys (I’ll think of their name later) who promoted “Breakthrough thinking/philosophy”, the gist of it being that you needed to deliberately have, and solve, as many problems as you could early on in a project to accelerate the overall learning curve. The success of this was evident even as we prepared the rig in 1992/93 which really pleased my German boss. We delivered the rig early!
Thanks John. Couldn’t agree more – aim for as many problems as possible as early as possible. Glad it paid off for you!
I believe that the name of these “breakthrough thinkers” was JMW, an American organisation, whom BP used throughout their worldwide operations. While doing this research I found this article from 1997 but which goes back to the time when we started the Wytch Farm ERD project in the early ’90’s:
There’s a lot in here which I found very familiar in the way that BP operated back then.
There’s a great quote in that article you linked” “Rather than take a cataclysmic approach, let’s try it, see how it works, understand what we’ve learned, and only then apply it more widely.” Very topical right now!