During WWII, a group of data scientists was given an important task: figure out why so many Allied planes were shot down over enemy territory.
The task force set out to assess damage done to aircraft that had returned from bombing missions. They would analyze trends in the data and then compile recommendations for improving the aircrafts with additional armor.
The data was represented by a diagram of an aircraft, highlighting areas that had received the highest concentration of damage. The diagram showed that the most amount of damage was collected in the wings, around the tail gunner, and down the center of the body of the planes. As per their recommendations, the Navy began increasing the amount of armor in these three regions.
Unfortunately, this did nothing to improve the survival rate of bomber crews. In many cases, the additional armor slowed down planes, making them more susceptible targets.
Andrew Wald, a statistician and member of Columbia University’s Statistical Research Group (SRG), was brought on the case and almost immediately realized the error—the recommendations were made based on an incomplete data set.
The planes that were returning home safely weren’t the problem. They should have been studying the planes that did not return from battle!
Of course, they didn’t have access to this data, but they could make some insights based on what they had in front of them. After taking a closer look at the diagram, it became clear that there was one area of the plane that accumulated very few, if any, little red dots—the engine.
Wald’s recommendations would save the lives of thousands of Allied soldiers. His work would lay the foundation for what is now known as Survivorship Bias: a logical error in statistics that occurs as a result of concentrating on data that passed some selection process and overlooks other data that did not.
The Allied statisticians were focusing on the planes that survived when they should have been focusing on the planes that didn’t. Marketers fall victim to survivorship bias every day, but the survivors aren’t heroes returning from war—they are your customers.
Many of us are guilty of letting existing customers skew our perception of our target market, when we should be focusing on those individuals who could have become customers if they hadn’t been shot down over competitive territory.
Say, you have an ecommerce store specializing in tools and other hardware. You view yourself as a B2B company with a target customer base of auto mechanics and midsize construction companies. Sure, there is nothing that would stop my father from going to your site and buying a new hammer, but he’s not really your target customer, and your marketing efforts should never be built on the premise that your products are for everybody. Instead, there is a specific niche that you are in business to serve, and you should tailor your marketing efforts accordingly.
We have a client who was facing this exact issue, and their previous PPC manager was failing to profitably scale revenue because they fell victim to survivorship bias.
When sifting through the search term report, we found that nearly 50 percent of historical sales came from broad searches, such as tools. It didn’t even matter that this search term had converted at a profitable ROAS; these were not the kinds of searches that were being performed by auto mechanics or managers of midsize construction companies.
This became apparent after digging through their attribution reports. Their Time Lag and Path Length reports showed that 85 percent of their customers had completed checkout within the first day of visiting the site. Few customers had visited the site multiple times before converting.
You might think that this is great news. A proud marketing manager would likely brag to his CEO that their website was set up to convert customers on their very first visit, and they shouldn’t waste money on remarketing!
But does that really make sense? If you’ve worked on as many accounts as we have over the years, you would understand that this doesn’t pass the smell test. A B2B ecommerce store selling high ticket items should have a significant Time Lag. These products are not bought impulsively and often require careful consideration before a customer reaches a final purchase decision. The best customers are likely to be considering multiple suppliers, and they should require a bit of courtship, through remarketing, to convince them to choose us.
It’s more likely that the lack of remarketing efforts, among other failures in the previous strategy, has been prohibiting their chances of appealing to users who would have converted on subsequent visits. It became clear that the account was set up in a way that neglected their core audience. It was sheer luck that they managed to convert other types of customers, as small as that population may have been.
It’s crucial to keep in mind that the sample of data that makes up these attribution reports is exclusive to customers who have converted. These are the planes that returned from battle, despite having taken on some fire, in the way of an imperfect shopping experience. It is extremely likely that, by ignoring the realities of your core target audience’s buyer journey, you are allowing potential customers to fall into the hands of your competition.
