Quarterly Experimentation Roadmapping

Hello! Welcome to the final lesson in our module on "Designing and Interpreting Experiments."

In our last session, we focused on bringing order to a long list of potential experiments by applying prioritization frameworks like ICE and RICE. You learned how to move from subjective "gut feel" decisions to a more objective, ranked backlog based on factors like Impact, Confidence, Ease, and Reach. This gives you a powerful tool to identify the most promising ideas.

However, a ranked list is not a strategy. As a leader, your next challenge is to transform that prioritized backlog into a coherent, actionable plan that your team can execute and that your stakeholders can understand.

Today, we will focus on exactly that. Our learning outcome is to structure a quarterly experimentation roadmap that aligns with strategic business goals. We will cover how to connect your experiments to high-level business objectives, plan your team's capacity, and create a visual roadmap that serves as a powerful communication tool.

1. From Prioritized Backlog to Strategic Roadmap

A prioritized backlog tells you the relative value of individual ideas. A roadmap, on the other hand, tells a story. It outlines what you're going to test, when you're going to test it, and most importantly, why. It's a strategic document that connects your team's daily activities to the company's quarterly and annual goals.

The most effective way to establish this connection is by using a goal-setting framework. A popular and powerful one used by companies like Google, Intel, and many fast-growing startups is Objectives and Key Results (OKRs).

Objective (O): A qualitative, ambitious, and inspiring goal. It answers the question, "What do we want to achieve?"
Key Results (KRs): Two to five quantitative and measurable outcomes that prove you've achieved your objective. They answer, "How will we know we've achieved it?"

Your experimentation roadmap should be designed to directly influence these Key Results.

To understand how to frame your marketing goals in this way, let's explore an excellent article on Marketing OKRs.

Marketing OKRs: How Agile Teams Set Goals That Drive ...

The article 'Marketing OKRs: How Agile Teams Set Goals That Drive Growth' by AgileSherpas provides a superb playbook for connecting strategic goals to marketing execution.

Please read the introduction and the first few sections, up to and including 'How To Use OKRs in Agile Marketing'. Focus on: The difference between outcomes (e.g., increased organic pipeline) and outputs (e.g., published 20 posts). How to write a concise objective and define measurable key results. The step-by-step process for setting OKRs, especially Step 1 ('Clarify the business outcome') and Step 4 ('Map key results to experiments').

By starting with clear OKRs, you ensure your experimentation program isn't just about finding small wins; it's a strategic engine designed to move the most important business metrics.

2. The Anatomy of a Quarterly Experimentation Roadmap

With your strategic objectives defined, you can now begin to structure your roadmap. A good roadmap isn't just a calendar of the highest-scoring RICE experiments. It's a thoughtful plan that balances ambition, resources, and strategic focus.

Here are the key components:

a) Themes or Strategic Goals:
Instead of a flat list of tests, group your experiments under the OKR they are meant to impact. For a quarter, you might have 1-3 major themes. For example:

Theme 1: Improve New User Activation. (Objective: "Create a world-class onboarding experience for new users.")
Theme 2: Increase Average Order Value. (Objective: "Drive higher customer spending per transaction.")
Theme 3: Optimize Mobile Conversion Rate. (Objective: "Deliver a seamless mobile purchasing journey.")

b) A Mix of Experiments:
A healthy roadmap includes a portfolio of different types of experiments. You can think of this in terms of the Value vs. Effort matrix we discussed previously:

Easy Wins (Low Effort, High Value): These deliver quick results and build momentum. Schedule a few of these early in the quarter.
Big Bets (High Effort, High Value): These are your major projects (e.g., a checkout redesign). They require significant resources and should be planned carefully, often with work starting early in the quarter even if the test launches later.
Incremental Improvements (Low Effort, Low Value): These can fill gaps in the schedule but shouldn't dominate your roadmap.

c) Timeline and Sequencing:
A quarterly roadmap is typically broken down by month or by two-week sprints. This allows you to plan resource allocation and account for dependencies. For example, you might need to run a diagnostic experiment (e.g., "survey users about checkout friction") in Month 1 to inform the design of a "Big Bet" experiment that launches in Month 3.

d) Resource Allocation:
Be realistic about your team's capacity. How many experiments can your developers, designers, and marketers realistically support at one time? Your roadmap must reflect this. It's a common mistake to create an overly ambitious plan that the team can't possibly execute.

This entire process of ideation, prioritization, and execution is part of a larger growth experimentation framework.

A Complete Guide to Growth Experimentation in 2024

Let's turn to the 'A Complete Guide to Growth Experimentation in 2024' from Ikaros. This guide provides a fantastic overview of the operational process that a roadmap lives within.

Please read the following sections: 'Growth process overview': This introduces the core building blocks like a growth team and sprints. 'Part 2 - Documenting Ideas as Experiments': Pay attention to how it suggests categorizing experiments (Diagnostic, Incremental, Radical) and documenting them. This is the raw material for your roadmap. 'Part 3 - Leading a Growth Team': The practical tips here on running growth meetings are directly related to managing and adjusting the roadmap.

3. Visualizing the Roadmap and Connecting it to Execution

A roadmap's power lies in its ability to communicate a plan visually. A simple spreadsheet or a project management tool can be used to create a clear overview.

Here's an example of what a high-level strategic roadmap might look like:

Quarterly Roadmap Template — This roadmap template shows how strategic goals are broken down into milestones and allocated across a timeline (in this case, quarters). Notice how different teams (R&D, Marketing, Sales) have their workstreams visualized, making dependencies and resource allocation clear.

While the roadmap provides the high-level strategic view, you need a way to manage the details of each experiment. This is where an Experiment Tracker or board comes in. It's the tactical layer that sits beneath your strategic roadmap. Each item on the roadmap might correspond to one or more cards on your experiment board.

Experiment Tracker Template — This Experiment Tracker shows the detailed view for each test. It captures the hypothesis, method, success metrics, and learnings. This is where the day-to-day work of the experimentation team happens, while the roadmap above provides the 'why' for the C-suite and other stakeholders.

As a leader, you operate at both levels:

You use the Roadmap to align with executives, secure budget, and communicate your team's strategic direction.
You use the Experiment Tracker to manage your team's execution, review results, and ensure the learning cycle is running smoothly.

Test your understanding!

Let's say your primary business objective for Q3 is to "Increase customer lifetime value (LTV) by improving retention." Your team has a prioritized backlog from the previous lesson. The top three experiment ideas are:

Test a new "win-back" email campaign for users who haven't purchased in 90 days. (RICE Score: 1500 - Med Effort, Med Impact, High Reach)
Launch a loyalty program with points for purchases. (RICE Score: 1200 - High Effort, High Impact, Med Reach)
A/B test adding personalized product recommendations to the post-purchase confirmation page. (RICE Score: 1800 - Low Effort, Med Impact, High Reach)

How would you structure a simple 3-month (Q3) roadmap for these initiatives? Consider sequencing, resource allocation, and dependencies.

Show answer

There's no single "right" answer, but a strong strategic approach would look something like this:

Q3 Roadmap: Improve Customer Retention

Month 1: Build Momentum and Start Foundational Work
- Launch: A/B test personalized post-purchase recommendations.
  - Rationale: This has the highest RICE score and is low effort. It's a quick win to start the quarter, generate learnings, and show progress.
- Begin: Design and development for the loyalty program.
  - Rationale: This is the "Big Bet" (High Effort, High Impact). It needs a long lead time. The foundational work can happen in parallel while the team runs smaller tests.
Month 2: Iterate and Continue Major Project
- Launch: A/B test the "win-back" email campaign.
  - Rationale: This is a medium-effort project that can be tackled once the first experiment is running or complete.
- Continue: Development and internal testing of the loyalty program.
Month 3: Launch the Big Bet
- Launch: Phased rollout or A/B test of the new loyalty program.
  - Rationale: With the foundational work complete, the final month is dedicated to launching and monitoring the impact of the quarter's largest initiative.
- Analyze & Report: Review results from all three experiments and report on progress towards the LTV/retention objective.

This roadmap is strategic because it:

Aligns all activities to a single, clear business goal.
Balances a quick win with a medium-sized project and a major strategic bet.
Realistically plans for the long lead time required for the high-effort loyalty program by starting it early.
Creates a continuous flow of activity and learning throughout the quarter.

Conclusion

You have now completed the "Designing and Interpreting Experiments" module. You've progressed from the fundamentals of A/B testing validity and statistical power to the high-level strategic work of planning an entire experimentation program.

Key Takeaways:

An experimentation roadmap is a strategic document that connects your team's work to business goals, typically framed as OKRs.
It goes beyond a simple prioritized list by theming experiments, planning a balanced portfolio of tests, and visualizing the plan on a timeline.
A great roadmap accounts for team capacity, dependencies, and resource allocation, making it a realistic and actionable plan.
As a leader, you use the high-level roadmap for strategic communication and the detailed experiment tracker for managing execution.

Preview of the Next Module:
In our next module, "Incrementality and True Marketing Impact," we will elevate our measurement thinking even further. While A/B testing is excellent for optimizing user experience, incrementality testing helps us answer one of the most critical questions in marketing: "What would have happened if I hadn't spent this money?" You'll learn how to design and interpret tests that measure the true, causal impact of your marketing channels, a skill essential for making multi-million dollar budget allocation decisions.