Assessing Startup Datasets

Hello! Welcome to the fourth lesson in our special topic module, "Diligence for AI Startups."

In our last lesson, we focused on how to analyze a startup's strategy for acquiring and annotating data. We established a five-point framework to assess the defensibility of that strategy, distinguishing between weak approaches like scraping public data and strong ones like deep integration into proprietary workflows.

Today, we shift from the strategy to the asset itself. This lesson directly addresses the learning outcome: Evaluate the quality, relevance, and defensibility of a startup's dataset. A brilliant strategy is meaningless if it produces a low-quality or irrelevant dataset. As an investor, you must be able to look past the pitch and critically assess the data that truly powers the AI.

We'll cover how the definition of a "good" dataset has evolved, provide you with a practical framework to assess data defensibility, and introduce the concept of a "Golden Dataset" as the ultimate tool for measuring quality and relevance.

1. The New Reality of Data Defensibility

The AI landscape has shifted dramatically. Before the rise of powerful foundation models (like GPT-4), simply having a massive, exclusive dataset was a powerful moat. Today, that's no longer enough. The availability of synthetic data and the reasoning capabilities of large models have devalued raw data volume. What matters now is much more nuanced.

To understand this shift, let's explore a modern framework for data defensibility.

As the article argues, advantages based purely on having "millions of data points" are fragile. The most durable advantages now come from how data is generated, integrated, and used to create a feedback loop that continuously improves the product.

2. An Investor's Framework for Data Defensibility

Given this new reality, you need an updated mental model for evaluation. When a founder claims they have a "data moat," your task is to determine if it's a genuine, defensible barrier or an expensive illusion.

The same article provides an excellent framework for making this distinction.

Let's distill this into a practical due diligence checklist. When you evaluate a startup's dataset, you're looking for "Green Light" signals:

• Proprietary Generation: Is the data created as a unique byproduct of the product's use in a way that can't exist elsewhere? (e.g., Tesla's driving data from its specific sensor suite).
• Feedback Loop: Does product usage automatically and continuously improve the AI model? Is there a system for learning from user interactions, corrections, and outcomes?
• Workflow Integration: Is the product so deeply embedded in a customer's critical operations that the data captured is highly contextual and creates high switching costs? (e.g., Veeva in pharma).
• Domain Expertise & Regulation: Does the data involve a deep, vertical-specific understanding or regulatory barrier (like in finance or healthcare) that a generalist model can't replicate?

A claim based on a "Red Light" signal, such as possessing a large but static or public dataset, should be a major warning that the claimed data moat is not defensible.

3. Measuring Quality and Relevance: The "Golden Dataset"

Defensibility is crucial, but it's only part of the story. A proprietary dataset is worthless if it's low-quality or irrelevant to the problem. So, how do you measure quality?

The most professional AI teams do this by creating and maintaining a "Golden Dataset." Think of this as the master answer key for their AI system. It's a curated set of inputs and their corresponding ideal outputs, validated by domain experts. This dataset becomes the benchmark against which all model changes are measured.

When you perform due diligence, asking founders about their Golden Dataset is a powerful way to assess their maturity and commitment to quality. Here are the key questions to probe:

• "Do you have a Golden Dataset?" The answer itself is telling. A "yes" indicates a level of rigor.
• "How was it constructed?" Look for involvement from domain experts, not just developers.
• "What does it cover?" A great answer will go beyond the "happy path" and include difficult edge cases, adversarial examples (inputs designed to trick the system), and negative examples (queries the system should correctly identify as unanswerable).
• "How do you maintain it?" The best practice is to have a process where every production failure or user correction is reviewed and potentially added to the Golden Dataset. This creates a direct feedback loop from the real world to your quality benchmark.

A startup that can't give you clear, confident answers to these questions likely has a weak handle on their AI's quality and performance.

4. Data Quality Creates Data Network Effects

The concepts of a feedback loop and a living Golden Dataset are the engines of a true data network effect—one of the most powerful moats in the AI era. This isn't just about more data; it's about a self-reinforcing cycle where usage generates unique data and feedback, which is used to improve the model, which in turn delivers a better product that attracts more users.

Let's watch a short clip that explains this flywheel.

This flywheel is the ultimate expression of a defensible data asset. It combines proprietary data generation (Pillar 1) with a feedback loop architecture (Pillar 2) and deep workflow integration (Pillar 3).

The following image provides a useful visualization of how data sharing, both among peers and across a value chain, can unlock these powerful network effects.

Test your understanding!

You are evaluating an AI startup that helps sales teams write better follow-up emails. The founder makes the following pitch about their data moat:

"Our advantage is our dataset. We have a partnership with a data broker that gave us a static dataset of 10 million sales emails from 2018-2021. No one else has a dataset this large. We had it labeled for 'positive' or 'negative' sentiment by a crowdsourcing platform. Our model is trained on this, and it's highly accurate."

Using the "Green/Yellow/Red Light" framework and the concept of a Golden Dataset, how would you evaluate this claim? Identify at least two red flags and suggest a question you would ask the founder to probe deeper.

Show answer

This claim has several significant red flags, placing it firmly in the Red Light category.

Red Flags:

1. Static Historical Dataset: The data is from 2018-2021 and is not being updated. It's a depreciating asset, as sales language and techniques evolve. This is a classic "Red Light Data Moat" (#3 in the Ferguson article).
2. Purchasable / Non-Exclusive Data: The data comes from a "data broker," which implies it's not truly exclusive and could likely be purchased by a competitor. Even if the deal was exclusive, it's not generated by a proprietary process. This is another red flag.
3. Commodity Labeling: The data was labeled for simple sentiment by a generic crowdsourcing platform. This adds very little proprietary value. A competitor could easily replicate this. The labels lack domain-specific nuance (e.g., what makes an email effective beyond just "positive"?).
4. No Feedback Loop: The description gives no indication of a living system. There is no mention of learning from how users interact with the generated emails, which is where a true moat would be built.

Follow-up Question:

A great question to ask the founder would be: "Can you tell me about your Golden Dataset? Specifically, how do you measure what a 'good' follow-up email is beyond simple sentiment, and how do you incorporate feedback when a salesperson edits or dislikes an email your AI suggests?"

This question cuts to the core of the issue. It forces the founder to address the lack of a sophisticated quality benchmark and the absence of a feedback loop, likely revealing the fragility of their claimed "moat."

Conclusion

Evaluating the dataset itself is a separate and crucial step from evaluating the data strategy. A defensible, high-quality data asset is the engine of a durable AI business. By moving beyond claims of "big data," you can apply a rigorous framework to find companies building true, compounding value.

Key Takeaways:

• Data defensibility has shifted from raw volume to workflow integration, proprietary generation, and feedback loops.
• Use the Green/Yellow/Red Light framework as a quick but powerful checklist to assess the defensibility of a startup's data.
• The existence and rigor of a Golden Dataset is your best proxy for a startup's commitment to data quality and relevance.
• Look for a living system, not a static file. The most valuable datasets are those that improve automatically with every user interaction, creating a powerful data network effect.

Preview of the Next Lesson

In this lesson, we dissected the characteristics of a high-quality, defensible dataset. In our final lesson for this module, we will synthesize what we've learned about AI business models, data acquisition, and data quality. The next lesson, "Identify how proprietary data and data network effects create competitive moats for AI companies," will tie all these concepts together to focus on how these technical assets translate into long-term, sustainable business value and competitive advantage.