With legislative pressures mounting for food and beverage companies to remove artificial colors and flavors from their products, product formulators are struggling to comply. Natural ingredients pose challenges from supply chain limitations to higher costs. Most important are the concerns about consumer acceptance. The industry is rife with examples of product changes that companies had hoped would go unnoticed or be embraced but instead inspired a revolt in consumers, with complaints that the product they knew and loved had changed. In some cases, companies reverted to the original formulation in the face of such strong consumer resistance. History has shown that ingredient substitution is indeed a risky business, and you must approach any product reformulation with care.

Key Takeaways: Product Reformulation Research Best Practices

Seeking consumer feedback before introducing product changes can help companies mitigate adverse impacts. Since consumer research comes at a cost, companies often start with a basic risk assessment. Products in the top tier of sales volume deserve a much greater investment in research than those with low sales. For the latter, a management decision without research might be justifiable.

A sensory evaluation using a discrimination test can be a good starting point for determining the consumer impact of an ingredient substitution. Discrimination testing can reveal whether a change is detectable. Unfortunately, replacing artificial ingredients with natural ones often leads to noticeable differences. For starters, natural colors may not be as bright as artificial ones. Even when the color matches, natural colors can add an unexpected or unwanted flavor.

When these differences are noticeable, learning whether consumers think the reformulated product is as good as (or better than!) the current product requires additional research. To answer that question with confidence, research should follow a few critical practices.

1. Recruit current product users

Companies need to avoid disappointing current users—which makes current users the best research target. Whether the reformulated product can attract new users is a separate research question.

One key consideration is whether the research target should consist exclusively of heavy users. After all, disappointing heavy users is likely to do the greatest damage to the franchise. On the other hand, with a natural product that shows a lot of variability under normal production, heavy users might be more used to product differences. As a result, they could be more forgiving of a change that is not too far outside the range of normal variability. In that case, lighter users may be more critical of a product change, and including them in the research could offer a more complete risk assessment.

2. Choose a sample size that reflects the decision risk 

When testing a small sample of consumers, the research is more likely to conclude that the original and reformulated products are equally liked. However, these small samples don’t provide the statistical power to confidently decide that the reformulation is not inferior to the current product. Especially for products with high sales, a larger sample is required.

3. Define clear criteria for judging the product reformulation

Is preference in a central location test sufficient? Should a home use test be run? Is a limited test market needed before launching the reformulation nationwide? Keeping potential business impact in mind, you need to decide what proof is enough to say that the reformulation is an acceptable substitute for your current product.

4. Align the research with marketing plans

Does your company intend to call out the product change? Perhaps the packaging will include a burst that says “No artificial colors!” or product advertising will highlight this feature. If so, this communication should accompany the product presented to consumers during research. Study participants can view a picture of the new packaging or hear a verbal statement that reflects the anticipated message. The research will always be more valid when it mirrors the reformulation marketing.

Go Deeper: How an Alienation Analysis Can Help

When conducting a preference test, digging deep into the results with an alienation analysis can provide further insight into the risk of an ingredient substitution. In a preference test, product users (often only heavy users) are presented with the original and reformulated product. While other measures can be included in an alienation test, the key outcome measures are purchase interest (on a 1-5 scale) and preference.

To measure alienation, researchers look at what percent of people who prefer the current product would reject the reformulation. After all, some consumers might prefer the current version but still be willing to buy the reformulated product.

Alienation is the percentage of participants who prefer the original and would not buy the reformulated one. An acceptable degree of alienation is one that falls below a threshold set by the client or does not statistically exceed a measure of respondent error (see graphic). If alienation exceeds that threshold or fails that statistical criterion, the reformulated product poses an unacceptable risk of hurting the product franchise.

Final Takeaways:

In P&K’s experience, the alienation analysis is an appealing way to quantify the risk of an ingredient substitution or other product reformulation. However, brand stakeholders can struggle with what level of alienation is safe to accept. For them, statistics such as equality in liking or preference may carry more weight. Therefore, alienation may not serve as the sole decision criterion, but as an ancillary one.

While no one research design fits all circumstances, the above considerations and performance metrics can provide strategic guidance on whether to proceed with your current reformulation or to continue formulating to better meet consumer expectations.

If you’d like to discuss your upcoming product changes and research needs, contact P&K Research today.

The Background:

Qualitative and quantitative research each have their place in product development. Early on, qualitative methods, like focus groups and one-on-one interviews, help brands explore consumer reactions to initial product concepts or prototypes. Follow-up quantitative research then provides the numbers needed to proceed confidently through the remaining stages of the process to the final launch decision. When done properly, this research produces results directly related to key performance indicators and enables progress in line with a company’s action standards.

Still, even during quantitative surveys, brands benefit from hearing the voice of the consumer. Collecting reactions in a consumer’s own language delivers insights beyond the numerical counts and averages of the responses to closed-ended questions. To get this blend of qualitative and quantitative information, surveys usually have several open-ended questions, such as “What did you like about this product?”. To take it a step further, they may select a small subgroup of participants (such as a focus group) for exit interviews after the quantitative phase of the research.

Breakthroughs in conversational AI unlock a third opportunity. P&K can now incorporate smart probing to open-ended survey questions, having AI guide a qualitative exploration after the quantitative phases. In this case study, P&K demonstrates the power and advantages of including conversational AI in quantitative studies.

The Challenge:

A confectionary company sought to benchmark its breath mint product to that of a leading competitor. The company asked P&K to design research that would provide insight into how to enhance both the product and its packaging.

The P&K Solution:

P&K Research saw an opportunity to leverage AI in this research. In standard self-administered surveys, open-ended questions by themselves offer limited learning. People in a food or beverage study asked “What did you like about this product?” might simply say “The taste.” However, an AI-driven chatbot can follow up with a probe, such as “What did you like about the taste?” Depending on the reply (“It was not too sweet”), the AI can dig deeper (“Why is the sweetness important to you?”) or ask if any other attributes were appealing. With AI, we can seamlessly weave open-ended probes throughout a quantitative survey, providing a more in-depth understanding of consumer opinions and feelings about the product experience.

For the breath mint study, P&K recruited 200 category users – split between client and competitive brand users – for a central location test. Respondents evaluated samples of each unbranded breath mint in sequential order, with timed intervals for in-mouth experience and a between-sample wait to minimize sensory carryover.

The self-administered survey asked about overall liking and product perceptions on key attributes. After trying both samples, users answered a preference question. The survey then used AI-driven prompts to explore the reasons behind their preference in a conversational manner. Each prompt builds on past answers, asking questions such as:

• Why did you prefer this product?
• Other than [insert prior response], what other reasons do you have for preferring it?
• What would you change to improve your preferred product?
• What else would you want changed?

If someone had no preference, prompts also examined that person’s reasoning.

In the final phase of the survey, respondents received packages of the client and competitor products. Participants rated each package on dimensions such as overall appeal and functionality. Likes and dislikes were probed in detail with conversational AI.

The Outcome:

Qualitative AI was able to uncover reasons behind consumers’ preferences and successfully replaced exit interviews with guided one-on-one conversations based on a series of prompts and probes. And while researchers traditionally conduct exit interviews with a subset of respondents, AI made it possible for us to collect qualitative data at scale – with the entire sample of participants!

The findings laid out a clear picture of the strengths and opportunities for product and package improvement. The voice of the consumer elicited by the conversational AI yielded nuanced insights that expanded on the quantitative ratings and guided the client on the steps for improving their product and package.

Key Takeaways:

Around the world, cultural differences and data collection methods influence how adults and children use scales.

In Africa, Middle East and South Asian countries, our research shows that people avoid the low ends of survey scales. With a 9-point liking hedonic scale, these consumers rate products an average of 7.4 or higher, a pattern we interpret as acquiescence bias. In the data from Saudi Arabia, not a single person used any scale points between 1 and 6.

Other scales have similar results in these regions. In Saudi Arabia and Egypt, almost everyone said all attributes were “Just Right” using the Just About Right (JAR) diagnostic scale. In India, nearly half of consumers in the study rated 15 of 28 attributes as “Extremely/Very Important” on the 5-point importance scale. Such results can fail to provide the information brands need for strategic product decisions.

On top of a cultural tendency to avoid low ratings, interview formats also can affect ratings. Across most cultures, central location tests using in-person interviews tend to result in higher ratings than those using self-administered surveys. In countries like Saudi Arabia and South Africa, where in-person interviews are the preferred survey format, the acquiescence bias is  therefore further amplified, inflating ratings even more.

Taken together, these results suggest that averaging product ratings across countries is not advisable and can distort the patterns of findings. With global data sets, brands must analyze each country’s ratings separately to get the best insights.

The good news? While rating scale behaviors vary across regions, consumer preferences remain largely consistent. Of course, tests can uncover regional preferences, providing valuable direction for brands (another reason it’s worthwhile to look at data per country). Still, in most cases, the rank order of product appeal remains the same across very distinct cultures. Even when countries use different parts of the scale, consumers agree on which products are the best—and the worst.

Go Deeper:

While global product scale ratings should not be averaged, P&K’s findings suggest that some global data harmonization is possible. Alternative scale-free methods or analyses focused on things like preference or ranking can offer a sound basis for collecting more discriminating results and summarizing global data.

When assessing product attributes, companies can explore replacing the JAR scale with a JAR Check-All-That-Apply (CATA) method. Follow-up questions about attributes not marked “just right” can help deliver more discriminating results.

Likewise, the KANO model is an alternative to the 5-point importance scale. With KANO, businesses can categorize and prioritize product features from least important to expected, most important, and delighter.

Final Insights:

While consumer rating scales can be useful tools, global brands must never overlook the significance of cultural differences and survey methods. By examining country results individually and leveraging alternative methods as needed, companies can acquire the data to create and refine products aligned with real consumer preferences.

To discuss your product research needs and best practices, contact P&K Research today.

Skipping in-use testing of packaging changes can have disastrous consequences for CPG companies. Unfortunately, many brands today rely solely on online evaluations of packaging design and graphics. While P&K does those packaging tests and believes they offer helpful direction, we recommend not stopping there. The value of getting a package into the hands of the consumer for them to use cannot be overstated. Whether through a central location or home use test, the most important thing is measuring how the package performs when consumers handle and use the product in a realistic way.

The Challenge:

A salad dressing brand was exploring changing its bottle design and package material. The company wanted a bottle that would differentiate its current product from competitors. After creating several prototypes varying in shape and material, the team knew that an in-use evaluation would give them critical insights for making a final design decision.

The P&K Solution:

A limited supply of prototype bottles made an in-home test infeasible. Instead, P&K designed a central location test to meet the brand objectives. The test evaluated and compared the prototype bottles to the current bottle and a competitor’s bottle based on shape, feel, and pourability.

P&K conducted the research with 500 consumers, split among people who used the client’s brand and the competitive brand. The test evaluated five different bottles with a vinaigrette and a creamy dressing. Each person received an unbranded bottle for pouring dressing onto a salad. They assessed the bottle’s comfort in hand, the dressing’s appearance in the bottle, and the ease of pouring. Participants also tasted the salad with dressing to gauge any effect of the package on the product experience.

At the end of the test, consumers reviewed branded and priced concepts to determine how the packaging options aligned with the client brand.

The Outcome:

P&K found that one of the new bottle designs was superior to the current design and on par with the competition. Additionally, our results showed that a design change would not alienate current product users.

Our package testing also enabled the company to reject one of the premium design options they were considering. This design scored the highest on visual appeal and performed well when used with the vinaigrette dressing. However, consumers were not satisfied with the pourability of the creamy dressing in this bottle. Instead, the premium bottle shape hurt the total experience by negatively impacting the taste experience of creamy dressings—a clear halo effect. If the client had chosen a package on appearance alone (for example, using an online test), they would likely have seen fewer repeat purchases due to the packaging’s inferior handling properties and perceived inferior taste.

As a result of P&K’s packaging test, the company successfully identified a package design that minimizes the risk of disappointing current users and simultaneously offers an opportunity to differentiate its product and drive increased business.

Key Takeaways:

In consumer sensory research, there’s a common saying: context matters. Whether it’s tasting snacks at home or sampling products in a controlled central location test (CLT), the setting can significantly shape consumer perceptions. Ratings tend to rise, product differences may become clearer, and consumers are more engaged and satisfied when the environment feels right.

But here’s the catch—while context can polish the presentation, it doesn’t change how consumers rank one product against its competitors. At P&K, our research has consistently shown that no matter the testing environment, the overall ranking of products almost always stays the same. In other words, you can’t put lipstick on a pig. A product that falls short will still rank at the bottom, no matter how perfect the setting.

Go Deeper:

To better understand this phenomenon, let’s look at a few case studies. First, however, it’s worth noting that individual consumers don’t necessarily provide the same rank order of liking by condition or context. Yet, when analyzed in aggregate, the overall rank order of performance remains stable across all contexts.

Branded vs Unbranded CLT: In this central location test, sharing the brand name had a positive impact on the absolute rating of all products. While the brand name helped some products become slightly more competitive, the rank order of consumer liking remained nearly identical.

Traditional vs. In-Context CLT: In this study, each time more context was provided, the overall liking scores increased for both food products tested. Subjects rated the food highest when served with toppings and sides during an in-context CLT. The lowest scores were for food with no toppings in a traditional CLT. Still, in each of the four contexts tested, the competitor’s product performed significantly higher every time.

Testing in Three Environments: We tested four wines in a traditional CLT, an immersive virtual reality environment, and an actual restaurant. Again, the overall liking rating increased in the immersive and actual environments, but the product performance rank order did not change.

Final Insights:

As these examples show, context simply can’t disguise a poor performer. A stronger product will always win out—which is why it’s essential to invest in optimizing products and sensory experiences to delight your customers and fulfill their needs.

Interested in learning more about context? Read more about what we’ve learned from providing context in central location tests.

Lab Server – all 4 locations

Reports to: Location Specific Lab Manager (Chicago, White Plains, Santa Ana, Plano)

Description: P&K Research is looking for food-handlers/servers in our consumer research lab. This role supports the team leaders in facilitating and conducting taste tests and product usage surveys.

Responsibilities Include: A typical day of work may include the following activities: checking respondents in to individual computers and pulling up their survey, passing out samples and/or assisting in the kitchen prep of samples, cleaning and replenishing work stations with water/palate cleansers, cleaning kitchen equipment and sample prep space, & reading a short script to the room of testers before their survey begins

Requirements:

• Must have basic knowledge and ability to use Windows OS and Chrome web browser
• Must be able to stand for up to 5 hours at a time without sitting
• Basic customer services skills/ past customer service role a plus
• High school diploma or equivalent
• Basic math, reading and computer skills
• Ability to follow directions with minimal supervision
• Food handling experience a plus
• Must have flexible availability, including possibly 1 Saturday a month.

P & K is an equal opportunity employer

www.PK-Research.com

Fragrance is highly emotive. The perfume industry is built around this connection, with perfumers developing fragrances that seek to convey a vast array of emotions and feelings; from desire to power, vitality to relaxation. Personal care and consumer health companies also leverage the power of fragrance.

The Challenge:

A manufacturer of a topical analgesic wished to incorporate a fragrance into its spray-on product that signaled comfort and relief, emotions that could be expected to amplify the pain reduction benefits of its active ingredients. Three candidate fragrances were developed. P&K was asked to assess their overall appeal and emotional profile compared to each other and to a competitor’s fragrance.

The P&K Solution:

P&K conducted a central location test with consumers who reported repeated episodes of muscle pain within the past three months. Fragrances were presented on blotters. Consumers were asked to sniff and evaluate each in succession, with ample breaks in between to minimize sensory fatigue. P&K employed traditional measures for assessing fragrance liking, fragrance intensity, and other sensory attributes. However, for assessing the emotional impact, P&K employed the Implicit Association Test, which can uncover subconscious associations and is well suited for identifying the emotions conveyed by a fragrance.

The Outcome:

Two of the candidate fragrances performed at parity to each other (and to the competitor) in terms of overall liking and their sensory characteristics. However, while similar on these measures, the two fragrances performed differently in the Implicit Association Test, with one eliciting stronger feelings of comfort, relief, and relaxation. This fragrance was also superior to the competitor in these respects. The results provided P&K’s client clear direction for which fragrance to incorporate into the finished product.

The Challenge:

A retail chain sought to choose fragrances for a new line of bath soaps marketed under their store brand. Vendors submitted sample fragrances for each of five pre-established flavor families, and the retailer asked P&K to provide consumer learning as to which submissions performed the best. A total of 18 fragrances needed to be evaluated.

The P&K Solution:

P&K recruited shoppers of the retail chain to participate in two-day central location tests. Fragrances were presented to consumers in unbranded, 2-oz jars for sniffing. For this research, P&K took advantage of its Discovery Lab, consisting of eight rooms, each equipped with a rapid air exchange system that prevents odor build-up and contamination across successive fragrance exposures. With this precaution in place, P&K was able to conduct the research efficiently and cost-effectively, with each consumer evaluating multiple fragrances in succession without the risk of aroma carryover between samples.

The Outcome:

For each for the five flavor families, P&K was able to recommend a winning prototype that met or exceeded the action standard, based on measures of overall liking, fit with the name of the flavor family, and performance on key sensory characteristics.

The Challenge:

A leading personal care company wanted to benchmark its in-market liquid gel hand sanitizer against the competition and determine if a new prototype performs better than their in-market product.  Thirteen competitive products were selected to represent the market.

In addition to benchmarking the current product and the product prototype, the client wished to identify:

• Key drivers of consumer overall liking.
• The ideal sensory profile.
• Gaps and points of differentiation among tested products.

The P&K Solution:

About two hundred liquid gel hand sanitizer users were pre-recruited to participate in a Home Use Test. The test involved each respondent using and evaluating ten of the 15 products over the span of approximately three weeks. In parallel, a sensory descriptive panel consisting of 10 panelists, trained in sensory evaluation, provided a detailed quantitative profile of each product, including aspects of scent, texture and skin feel experienced during dispensing and application. Based on the consumer and descriptive panel data, P&K developed a predictive model that identified key drivers of liking and the sensory profile of the “ideal” product.

The Outcome:

• The client’s products performed well, but were not the top scoring products among consumers.
• The in-market product and prototypes were liked equally by consumers.
• Sensory analysis provided a detailed market landscape, showing points of sensory similarity and difference among products.

By identifying the key drivers of liking and the ideal sensory profile, P&K was able to provide clear product innovation and reformulation recommendations, which the client incorporated into their product development strategy.

Answering the Big Research Questions

In this post, we’ll dive into two elements that are important to a successful product test. First, the test needs to deliver a clear answer to the underlying research question(s), e.g. “is my product better (or not) than the competition?” Secondly, the research needs to answer the question “why?” – e.g. “why is my product better (or not) than the competition?”

Understanding Action Standards and KPIs

While there are a lot of factors that come into play in successfully answering a research question (all those factors that make for “good research”), here we focus on one key element: action standards. Action standards are predetermined criteria that define the level of performance required for a product to meet specific business objectives. These standards serve as benchmarks against which the results of the research are evaluated. Action standards can vary depending on factors such as the stage of product development, target market, and business strategy. They help decision-makers determine whether a product meets the necessary criteria to progress to the next stage of development, launch into the market, or undergo further refinement. They are critical for decision-making and for building an effective research study.

In well-designed product tests, the action standards match up against key performance indicators (KPIs) to make informed business decisions after the research is completed. Common KPIs in early-stage product testing include overall liking, preference, purchase interest, and alignment with consumer expectations. Only at later stages do financial metrics, such as sales volume projections, become relevant. Based on our extensive database of early-stage product tests and our understanding of the most sensitive metrics, we can suggest numeric benchmarks that can serve as reference points for what constitutes a “good” score. However, we always recommend that clients refer to their own history of product tests when it comes to identifying action standards, or engage with us in mining their historical data to develop the best, product category-specific KPIs and benchmarks.

That said, it is not enough to determine whether the product does or does not meet action standards. The question is, why? What about the product contributed to its appeal? What attributes explain why it was less preferred? Insights into a product’s shortcomings guide product modifications, and insights into its strengths can suggest communication strategies when going to market. Here are three factors to consider that can enable answers to “why.”

Study design:

One of the most powerful ways for answering “why” questions is using an experimental design. In an experimental design, product features or attributes are systematically varied over several prototype formulations. For example, in the personal care category, this could mean samples of a mouthwash are varied in both flavor intensity and color. In a beverage, the product developer may have varied the level of a sweetener. Experimental designs can vary from the simple (only one factor is varied) to the complex (multiple factors), testing many variations of a product. P&K Research has conducted experimental design-based studies with as many as 20 product variants and more. In either case, an analysis of how a key performance indicator varies in response to the known differences in the products provides the most direct evidence for what is causing that variation.

In the case of experimental designs, it is the external information about the products (e.g. the level of fragrance in an air freshener) that helps drive insights. Another type of external information is that derived from trained sensory panels via descriptive analysis. Descriptive analysis quantifies the sensory product experience in an objective and reliable manner. The quantifications of sensory attributes (strength of fragrance, sweetness intensity, etc.) can be used to understand the basis for variations in the key performance measures.

Consumer questionnaire design:

The vast majority of product tests depend on answering the why question based on consumers’ answers to questions other than the key performance measures. These questions, often referred to as “diagnostic attributes,” are included to help elucidate the reasons consumers may be scoring a product high or low on a KPI. For example, in a personal care study on hand soap, the questionnaire may ask about the level of fragrance – was it too high, too low or just right? Additional questions may ask about the performance of the hand soap – how effective was it in removing grease? Obviously, the insights gained from this approach to answering “why” are only available if the right questions have been included in the consumer survey. This is where experience with testing in a product category plays an important role. One of the advantages P&K has to offer its clients is that in the thousands of products tests we have conducted, we have touched almost every category in food, beverage and personal care, with additional experience in many categories related to home care and consumer health.

Analytics:

Arriving at answers to the why question requires analytics that are tailor-made to uncover the relationships between a KPI and product features or attributes. Collectively, these analytics are referred to as “driver analyses” – since their purpose is to uncover what is driving variations in a KPI. There are many different kinds of drivers analyses, and they include some highly statistical approaches suited for the analyses of experimental designs or for correlating sensory descriptive analyses with KPI responses. At P&K, we distinguish between “micro” and “macro” level analyses. A “micro” analysis comprises analytics that are focused on a single product and aim to uncover what attributes account for high versus low scores on a KPI. “Macro” analyses compare the performance on a KPI across two or more products. P&K’s proprietary Variable Influence Analysis (VIA) is a macro level analytic approach that identifies those attributes that best explain the difference between two (or more) products, for example isolating those characteristics needed to close the gap between a client’s product and a competitor’s. Many of our clients have found VIA extremely useful and have selected VIA as a standard deliverable in our reports.

To wrap up, product tests need to be designed to inform business decisions. Clear and sensitive action standards serve an important purpose and are key to moving an organization forward following their research investment. Win or lose by virtue of those action standards, clients importantly also require an understanding as to why the product performed the way it did. Only with those insights into a consumer’s product experience can a client brand move forward with confidence in their product development and marketing efforts.