Answering the Big Research Questions
In this post, we’ll dive into two elements that are important to a successful product test. First, the test needs to deliver a clear answer to the underlying research question(s), e.g. “is my product better (or not) than the competition?” Secondly, the research needs to answer the question “why?” – e.g. “why is my product better (or not) than the competition?”
Understanding Action Standards and KPIs
While there are a lot of factors that come into play in successfully answering a research question (all those factors that make for “good research”), here we focus on one key element: action standards. Action standards are predetermined criteria that define the level of performance required for a product to meet specific business objectives. These standards serve as benchmarks against which the results of the research are evaluated. Action standards can vary depending on factors such as the stage of product development, target market, and business strategy. They help decision-makers determine whether a product meets the necessary criteria to progress to the next stage of development, launch into the market, or undergo further refinement. They are critical for decision-making and for building an effective research study.
In well-designed product tests, the action standards match up against key performance indicators (KPIs) to make informed business decisions after the research is completed. Common KPIs in early-stage product testing include overall liking, preference, purchase interest, and alignment with consumer expectations. Only at later stages do financial metrics, such as sales volume projections, become relevant. Based on our extensive database of early-stage product tests and our understanding of the most sensitive metrics, we can suggest numeric benchmarks that can serve as reference points for what constitutes a “good” score. However, we always recommend that clients refer to their own history of product tests when it comes to identifying action standards, or engage with us in mining their historical data to develop the best, product category-specific KPIs and benchmarks.
That said, it is not enough to determine whether the product does or does not meet action standards. The question is, why? What about the product contributed to its appeal? What attributes explain why it was less preferred? Insights into a product’s shortcomings guide product modifications, and insights into its strengths can suggest communication strategies when going to market. Here are three factors to consider that can enable answers to “why.”
Study design:
One of the most powerful ways for answering “why” questions is using an experimental design. In an experimental design, product features or attributes are systematically varied over several prototype formulations. For example, in the personal care category, this could mean samples of a mouthwash are varied in both flavor intensity and color. In a beverage, the product developer may have varied the level of a sweetener. Experimental designs can vary from the simple (only one factor is varied) to the complex (multiple factors), testing many variations of a product. P&K Research has conducted experimental design-based studies with as many as 20 product variants and more. In either case, an analysis of how a key performance indicator varies in response to the known differences in the products provides the most direct evidence for what is causing that variation.
In the case of experimental designs, it is the external information about the products (e.g. the level of fragrance in an air freshener) that helps drive insights. Another type of external information is that derived from trained sensory panels via descriptive analysis. Descriptive analysis quantifies the sensory product experience in an objective and reliable manner. The quantifications of sensory attributes (strength of fragrance, sweetness intensity, etc.) can be used to understand the basis for variations in the key performance measures.
Consumer questionnaire design:
The vast majority of product tests depend on answering the why question based on consumers’ answers to questions other than the key performance measures. These questions, often referred to as “diagnostic attributes,” are included to help elucidate the reasons consumers may be scoring a product high or low on a KPI. For example, in a personal care study on hand soap, the questionnaire may ask about the level of fragrance – was it too high, too low or just right? Additional questions may ask about the performance of the hand soap – how effective was it in removing grease? Obviously, the insights gained from this approach to answering “why” are only available if the right questions have been included in the consumer survey. This is where experience with testing in a product category plays an important role. One of the advantages P&K has to offer its clients is that in the thousands of products tests we have conducted, we have touched almost every category in food, beverage and personal care, with additional experience in many categories related to home care and consumer health.
Analytics:
Arriving at answers to the why question requires analytics that are tailor-made to uncover the relationships between a KPI and product features or attributes. Collectively, these analytics are referred to as “driver analyses” – since their purpose is to uncover what is driving variations in a KPI. There are many different kinds of drivers analyses, and they include some highly statistical approaches suited for the analyses of experimental designs or for correlating sensory descriptive analyses with KPI responses. At P&K, we distinguish between “micro” and “macro” level analyses. A “micro” analysis comprises analytics that are focused on a single product and aim to uncover what attributes account for high versus low scores on a KPI. “Macro” analyses compare the performance on a KPI across two or more products. P&K’s proprietary Variable Influence Analysis (VIA) is a macro level analytic approach that identifies those attributes that best explain the difference between two (or more) products, for example isolating those characteristics needed to close the gap between a client’s product and a competitor’s. Many of our clients have found VIA extremely useful and have selected VIA as a standard deliverable in our reports.
To wrap up, product tests need to be designed to inform business decisions. Clear and sensitive action standards serve an important purpose and are key to moving an organization forward following their research investment. Win or lose by virtue of those action standards, clients importantly also require an understanding as to why the product performed the way it did. Only with those insights into a consumer’s product experience can a client brand move forward with confidence in their product development and marketing efforts.