Marketing Researchers are often asked by clients to defend a particular market research methodology, sampling scheme, or result by answering the question “If I do as you propose, will I make the same business decision?”. Fortunately, statistics gives us the tools to answer this question, and the clear answer is “I don’t know”. That is because the answer is unknowable – because while we may know the specific criteria a client used to make the business decision, we don’t know if it was the right decision. And, if the decision was the wrong decision we would hope that we would indeed make a different decision the next time around.
Let’s look at this in a little more detail. Suppose we are deciding whether to develop a product concept to introduce into the market. Our decision criteria is that at least 50% of the population in a survey say that they would like to buy this product, based on its description. I do a survey, and get an answer – 54%. For some reason, one week later we do the same survey and we get an answer of 46%. I am shocked and appalled! I would make a completely different business decision based on these two studies. This is exactly the scenario painted by Kim Dedeker (then of Procter & Gamble) a few years ago in her indictment of the online research industry. How could the same study be done a week apart and get significantly different results AND get results that led to completely different business decisions?