Hello! Welcome back to our course.
In our last lesson, we explored how to apply incrementality insights to make strategic budget allocation decisions. We focused on the power of marginal ROI and the importance of a dynamic, portfolio-based approach to managing your marketing investments. The key takeaway was that to maximize overall business growth, you must be willing to shift budget from channels with low causal impact to those with high causal impact, even if it contradicts platform-reported metrics.
However, making these bold, data-driven decisions hinges on one critical assumption: that the incrementality insights you receive are accurate and reliable. What happens when the data is ambiguous or the methodology is questionable?
This brings us to today's lesson, which is designed to make you a more discerning and effective consumer of data. Our learning outcome is to formulate key questions to ask an analytics team when presented with incrementality study results. Your role as a leader isn't to run the analysis yourself, but to pressure-test it, understand its limitations, and build the confidence needed to act on it. This lesson will provide you with a structured framework for that critical dialogue.
1. The Skeptical Leader: Why We Must Question the Results
In theory, an incrementality test is a clean, simple comparison between a group that sees your marketing (the test group) and one that doesn't (the control group). In reality, the business world is a noisy and complex place.
A poorly designed experiment can lead to misleading conclusions and, consequently, costly business mistakes. Your job is to be the first line of defense against this. To get started, let's review some common traps that even experienced teams can fall into.
Building an Incrementality Practice: A Practical Guide
The article 'Building an Incrementality Practice: A Practical Guide' by Haus provides a sharp, strategic overview of common mistakes to avoid.
Please read the section 'Key mistakes to avoid in incrementality modeling'. As you read, think of these as red flags to look for when your team presents their findings.
As the article points out, the biggest dangers are:
- Misaligned KPIs: Measuring lift on a metric that doesn't actually drive business value (e.g., clicks instead of revenue).
- False Precision: Presenting results with an unrealistic level of certainty, ignoring the inherent noise in business data.
- Blind Trust in Platforms: Accepting platform-reported conversions without verifying if they are truly incremental.
To navigate these pitfalls, you need a systematic way to probe the results you're given.
2. A Leader's Framework for Interrogating Data
The best way to structure your inquiry is to follow the lifecycle of the experiment itself. This creates a logical flow for your conversation with an analyst or vendor. We will structure our key questions around four phases:
- The Business Question & Hypothesis
- The Experimental Design
- The Results & Interpretation
- The Strategic Context
Let's dive into the specific questions you should be asking at each stage.
Phase 1: Questioning the Business Question & Hypothesis
Before you even look at a number, you need to understand the purpose of the test. An experiment without a clear business objective is an academic exercise, not a strategic tool.
-
Key Question 1: "What was the specific business decision this experiment was designed to inform?"
This first question ensures the test was tied to a meaningful action. Was it to validate scaling a new channel? To justify cutting a legacy tactic? To set a new CPA target? If the analyst can't answer this clearly, the test may lack strategic focus. -
Key Question 2: "What was our pre-test hypothesis, and what evidence was it based on?"
A good experiment is designed to prove or disprove a specific belief. For example: "We believe our YouTube prospecting campaigns are driving an incremental ROAS of at least 2.0, based on initial attribution data and industry benchmarks." This question helps you check for "data fishing"—running tests without a clear idea of what you're looking for and then trying to find a narrative in the noise.
Phase 2: Questioning the Experimental Design (The Methodology)
This is the most technical part of the conversation, but your questions don't need to be technical. They are about understanding the choices the analyst made and their implications.
For pub on TwG _ [External Playbook] Modern Measurement
Google's 'Modern Measurement' playbook provides an excellent checklist for experiment design. Understanding these components is key to formulating your questions.
Please read the section 'Design incrementality experiments following best practices'. Focus on the 'Checklist for experiment design' and the descriptions of the different methodologies (Conversion Lift based on users vs. Geo Experiments).
Based on this, here are the crucial questions about methodology:
-
Key Question 3 (Methodology): "What type of incrementality test was this (e.g., Geo-lift, User-based Conversion Lift)? Why was this method chosen over others?"
As you read, user-based tests are subject to tracking gaps, while geo-based tests can be more robust but are also more resource-intensive. Understanding this choice reveals the potential biases and limitations of the study. For instance, a user-based lift study might under-report the impact of a channel that drives offline sales. -
Key Question 4 (Groups): "How were the test and control groups created? What steps were taken to ensure they were comparable and there was no 'contamination' between them?"
The validity of the entire experiment rests on the control group being a true "what if" scenario. You want to hear about how they matched geographies on key variables or how they ensured users in the control group weren't accidentally exposed to the test campaign. -
Key Question 5 (KPIs): "What was the primary KPI for this test? How does the measurement of this KPI in the test account for things like offline sales, returns, or different sales channels?"
This is about ensuring KPI parity. If you're testing the impact on total company revenue, but the test can only measure website sales, the results will be incomplete. You need to know what's not being measured. -
Key Question 6 (Power & Duration): "Was this test 'powered' to detect a meaningful effect? What was the minimum lift we could have reliably detected?"
"Statistical power" is a technical concept, but you can frame it in business terms. This question translates to: "Was the test big enough to even find the lift we were looking for?" An underpowered test that finds "no significant lift" is meaningless—it doesn't mean there was no effect, only that the test was too small to find it.
Phase 3: Questioning the Results & Interpretation
Once you're comfortable with the methodology, you can dig into the results themselves. Don't just accept the headline number.
)
-
Key Question 7 (Significance & Confidence): "What was the measured lift, and what is the confidence interval? Is the result statistically significant?"
This is the most important question about the result. A reported lift of "10%" is less meaningful than "a 10% lift with a 95% confidence interval of [2%, 18%]". The confidence interval gives you the "range of plausible outcomes." A wide interval that includes zero (e.g., [-5%, 25%]) means the result is not statistically significant, and you can't be confident there was any real effect at all. -
Key Question 8 (The Sanity Check): "How does this result compare to our expectations or industry benchmarks? If it's wildly different, what could explain that?"
An unexpected result isn't necessarily wrong, but it demands more scrutiny. The FAQ in the Google Playbook (LINK, section 11) notes that incremental lift is often between 0% and 25%. A result of 50% or -30% should trigger a deep dive into potential causes or flaws in the experiment. -
Key Question 9 (The Calibration): "How does this incremental ROAS (iROAS) compare to the platform-reported ROAS for the same period? What is the resulting 'calibration multiplier'?"
This directly connects to our previous lesson. If platform ROAS was 4.0x and the iROAS from the test is 2.0x, the calibration multiplier is 0.5. This is a powerful, simple metric that helps you discount platform-reported numbers across your portfolio. You want to ensure this calculation is being done and discussed.
Phase 4: Questioning the Strategic Context
Finally, a single test result doesn't exist in a vacuum. You need to place it within the broader context of your measurement strategy and business goals.
-
Key Question 10 (Holistic View): "How does this result fit with what our other measurement tools, like MMM or attribution, are telling us? Are there discrepancies, and how do we reconcile them?"
As the Google Playbook emphasizes, different tools answer different questions. If an incrementality test shows a channel has low impact, but your MMM shows it's a key long-term driver, you need to have a discussion. It could be that the experiment measured short-term sales while MMM is capturing long-term brand equity. -
Key Question 11 (Durability): "How long do we believe this result will be valid? When should we plan to re-test to keep our insights fresh?"
The market changes, your creative gets stale, and competitors react. An incrementality result from six months ago may no longer be true. Best practice, as noted in the Google Playbook (LINK, section 4), is to re-test every 3-6 months to keep your "calibration multipliers" fresh. -
Key Question 12 (Actionability): "Given this result and its limitations, what is the specific, recommended action? What is our confidence level in that recommendation?"
This is the final, bottom-line question. It forces the analyst to move from reporting data to recommending a business decision, while also being transparent about the level of uncertainty.
Test your understanding!
An analyst on your team presents a single slide that says: "Q3 YouTube Campaign Geo-Test: The campaign drove a 15% lift in sales with an iROAS of $2.50."
Based on what you've learned, what are the top 3-4 questions you would immediately ask to understand the validity and context of this claim?
Show answer
While many questions are valid, here are four excellent starting points:
- "What was the confidence interval around that 15% lift? Was the result statistically significant?" (This is the most critical first question to verify if the "lift" is real or just noise.)
- "How does that $2.50 iROAS compare to the ROAS reported in Google Ads for the same campaign? What's the calibration factor?" (This connects the finding to your day-to-day metrics and helps you understand the gap between correlation and causation.)
- "How were 'sales' defined and measured in this test? Did it include all revenue streams, or was it limited to, for example, website conversions?" (This probes for KPI parity and uncovers what might have been missed.)
- "How were the geo-regions for the test and control groups selected? How did we ensure they were truly comparable?" (This questions the core of the experimental design to ensure the foundation of the test was solid.)
Conclusion
As a marketing leader, your greatest asset is your ability to ask the right questions. When it comes to incrementality, being a critical, informed consumer of data is what separates successful, data-driven strategies from expensive missteps. You don't need to be a statistician, but you do need to be a structured and persistent interrogator of the data presented to you.
Key Takeaways:
- Be a Healthy Skeptic: Real-world experiments are messy. Your job is to uncover the potential biases and limitations before making a decision.
- Use a Structured Framework: Question the results by following the lifecycle of the experiment: Hypothesis -> Design -> Results -> Context.
- Focus on Business Implications: Frame your questions in business terms. Instead of asking about "p-values," ask "What is our confidence that this lift is real and not just noise?"
- Demand Transparency: Don't accept black-box answers. Use the questions in this lesson to foster a culture of transparent, rigorous, and strategically-aligned analytics within your team.
Preview of the Next Lesson:
Now that you have critically evaluated the results and are confident in the findings, what's next? You need to share these insights with the rest of the business, particularly with executives and finance partners who control the budget. In our next lesson, we will focus on how to develop a communication strategy for explaining incrementality findings to executives, translating complex analytical results into a simple, persuasive narrative that drives action.