Evaluating MMM: Methodology and Limitations

Hello! Welcome back to our module on Marketing Mix Modeling.

In our previous lessons, we've explored the core outputs of an MMM, like contribution charts and response curves, and unpacked the key transformations of adstock and saturation that power them. You now have a solid understanding of what an MMM produces and the conceptual mechanics inside it.

However, a model is only as good as its underlying methodology. A flawed MMM can produce outputs that look plausible but lead to dangerously misleading strategic decisions. As a leader, your role isn't to build the model, but to be its most insightful and critical consumer.

This lesson is designed to equip you with that critical lens. Our learning outcome is to critically assess the methodology and potential limitations of a proposed or completed MMM. We will build a practical framework for you to evaluate any MMM you encounter, so you can ask the right questions, identify red flags, and have confidence in the multi-million dollar budget decisions that rely on it.

1. The Foundation: Assessing the Inputs

The axiom "garbage in, garbage out" is especially true for MMM. Before you even look at the model itself, you must scrutinize the data it was built on. A sophisticated model cannot compensate for incomplete or poor-quality data.

To start, let's get a high-level view of the data collection and preparation process in a typical MMM project.

Based on the video and industry best practices, here are the first set of questions you should ask when assessing an MMM proposal or report:

• Data History and Granularity:

  • Question: "What is the time period and granularity of the data used?"
  • What to look for: Ideally, you want at least two to three years of weekly data. Less than two years makes it very difficult for the model to accurately distinguish seasonality from marketing impact. Daily data can sometimes be used, but weekly is the standard. A model with only one year of data is a potential red flag.

• Completeness of Variables:

  • Question: "Beyond media spend and sales, what other variables were included in the model?"
  • What to look for: A credible model must account for factors that influence sales outside of your marketing. The list should include:
    • Seasonality: How is the natural rhythm of your business captured?
    • Promotions & Pricing: Were your sales events and price changes included?
    • Macroeconomics: Factors like consumer confidence, inflation, or even COVID-19 lockdowns.
    • Competitor Activity: If available, data on major competitor campaigns can be crucial.
  • A model that only includes your media spend and your sales is almost certainly going to be inaccurate, as it will wrongly attribute all sales fluctuations to your marketing.

The process of gathering and validating this data should be a collaborative one. If a vendor or internal team presents a model without having first had a deep conversation with you about these business-specific drivers, it's a sign they may be using a generic, one-size-fits-all approach.

2. The Engine: A Checklist for Model Methodology

Once you're confident in the data inputs, it's time to inspect the model's engine. Modern MMMs have evolved significantly, and there are key features that separate a robust, modern model from an outdated or overly simplistic one.

This checklist is your guide to asking pointed questions about the methodology.

Let's break down the most critical items from that checklist.

✅ Changing ROI Over Time (Time-Varying Effects)

This is arguably the most important feature of a modern MMM. The effectiveness of a marketing channel is not static. It changes due to platform algorithm updates, creative refreshes, competitor actions, and audience saturation.

• The Red Flag: A model that calculates a single, fixed ROI for a channel (e.g., "Facebook ROI was 3.5") across a multi-year period is making a dangerous assumption of stability. This is often referred to as using "static betas" or "constant coefficients".
• The Right Question: "Does your model assume channel effectiveness is constant, or does it allow for ROIs to change over time?"
• Why it Matters: A channel that was a top performer two years ago might be inefficient today. A model with time-varying effects can capture this dynamic, providing you with a current, actionable understanding of performance.

✅ Handling of Adstock and Saturation

From our last lesson, you know these concepts are crucial. When assessing a model, you need to check if the outputs for these transformations are plausible.

• The Red Flag: The model produces outputs that defy business logic. For example, it might claim that Branded Search has a 10-week carryover effect (adstock), or that a nascent channel like TikTok is already completely saturated.
• The Right Question: "Can you walk me through the adstock decay rates and saturation curves for my key channels? Do they align with our intuitive understanding of how these channels work?"
• Why it Matters: Your business expertise is a vital sanity check. If the model's fundamental assumptions about how marketing works are implausible, its budget recommendations will be untrustworthy.

✅ Handling of Seasonality and Promotions

How a model treats predictable events like holidays or sales is a sign of its sophistication.

• The Red Flag: The model simply uses a "dummy variable" to control for an event, treating it as completely independent of marketing.
• The Right Question: "How does the model treat the interaction between our marketing spend and key seasonal or promotional periods?"
• Why it Matters: You wisely spend more during your peak season. A naive model might see high sales and high spend occurring at the same time and wrongly conclude your marketing is ineffective then, attributing all the lift to "seasonality." A good model understands that marketing and seasonality work together multiplicatively.

3. The Reality Check: Validating the Outputs

A model can have sound data and a sophisticated methodology but still produce incorrect results. The final and most important step is to validate the model's outputs against reality.

✅ Out-of-Sample Predictive Fit

This is a critical technical check. It answers the question: "How well does the model predict the future?"

• How it works: The model is trained on a portion of the data (e.g., Jan 2020 - Nov 2022) and then asked to "predict" the results for a period it has never seen (e.g., Dec 2022). The predictions are then compared to what actually happened.
• The Red Flag: The vendor or team heavily promotes a high "R-squared" value. R-squared measures in-sample fit (how well the model explains the data it was trained on). It's easy to get a high R-squared with an overfitted model that is useless for prediction.
• The Right Question: "What was the model's out-of-sample predictive accuracy? Can you show me the back-test comparing its forecast to actual sales for a holdout period?"

✅ Causal Validation via Incrementality Testing

This is the gold standard for building trust in an MMM. MMMs are fundamentally correlational models. They find statistical relationships but can't, on their own, prove causation. Incrementality tests (like geo-lift or conversion lift studies) are designed to measure causality. A modern, robust measurement strategy uses incrementality tests to "anchor" the MMM to reality.

• The Red Flag: The MMM operates in a silo, with no connection to any causal experiments you've run. The vendor dismisses lift test results that contradict the model.
• The Right Question: "How do you incorporate findings from our incrementality tests into the model? How do you ensure the model's recommendations are calibrated to causal impact?"
• Why it Matters: This integration provides the best of both worlds: the holistic, cross-channel view of MMM and the rigorous, causal proof of incrementality. It builds confidence that you're not just chasing correlations.

Conclusion: Your Role as the Critical Leader

This lesson has armed you with a comprehensive framework for scrutinizing a Marketing Mix Model. You don't need to understand the complex statistics or code, but you now know the critical concepts and the right questions to ask.

Key Takeaways:

• Assess the Foundation: Start with the data. Ensure it has sufficient history (2-3 years, weekly) and includes all key business drivers, not just media spend.
• Use the Methodology Checklist: A modern MMM should account for time-varying ROI, handle adstock and saturation plausibly, and intelligently model seasonality and promotions.
• Demand a Reality Check: Validate the model through out-of-sample predictive testing and, most importantly, by integrating it with causal incrementality tests. Be skeptical of vanity metrics like in-sample R-squared.
• Trust Your Business Intuition: If the model's outputs seem wildly implausible, challenge them. Your domain expertise is an essential part of the validation process.

Preview of the Next Lesson:
Now that you know how to assess a model and build confidence in its outputs (or understand its weaknesses), we can move on to using it for strategic planning. In our next lesson, you will learn how to use MMM-derived response curves to inform optimal budget allocation across channels. We will translate the model's insights into actionable budget scenarios and forecasts.