Capstone Requirements Brief: Data Contracts, Volumes, Consumers, Analytics, Latency, Retention, and Reliability Targets

Hello, and welcome to the first lesson in your data-engineering transition roadmap. This course is built around one production-style connectivity intelligence platform: it ingests connectivity events and reference data, produces trustworthy analytics in BigQuery, and eventually serves selected insights through a FastAPI service.

Before building pipelines or choosing GCP services, establish what the platform must actually promise. A strong requirements brief turns an appealing but vague request—“turn connectivity data into real-time intelligence”—into testable commitments about inputs, consumers, scale, timeliness, retention, and failure handling. It will become the reference point for every later decision about modelling, BigQuery design, dbt, Airflow, validation, APIs, and infrastructure.

Suggested pace: 40–45 minutes.

---

From a business request to an engineering contract

A requirements brief is not an architecture diagram and not a list of technologies. It answers:

1. Why does the data product exist?
2. Who will use it and what decisions will it support?
3. What data arrives, in what form, and under whose ownership?
4. What counts as correct, complete, secure, and on time?
5. What happens when an upstream system or pipeline fails?

This distinction matters in interviews. A common weak response to a data-platform design prompt is to begin with “I would use Kafka, Spark, and a warehouse.” A stronger response begins by clarifying the decision, data latency, volume, and correctness requirements—then derives technical choices from those constraints.

The following interview excerpt demonstrates that habit: the candidate first clarifies the problem and metrics, then estimates scale rather than assuming it.

For this capstone, we will make assumptions explicit rather than pretending they are facts. In a real job, you would validate them with product, operations, analytics, security, and source-system owners.

A focused business outcome

Use this as the capstone’s working problem statement:

Build a trusted connectivity-intelligence data product that enables product and operations leaders to identify service degradation, understand customer impact, and prioritize reliability and retention actions from recent and historical connectivity events.

This statement has a useful boundary: the platform is for analytics and decision support, not a direct network-control system. It can reveal that a location has a rising connection-failure rate, but it will not itself reboot devices or dispatch technicians. That boundary prevents us from accidentally imposing sub-second operational-control requirements on an analytics platform.

To keep “actionable intelligence” from becoming an empty phrase, convert it into analytical questions. Each question should identify a decision, a population, a time window, and a measurable outcome.

Consumer	Decision supported	Analytical question	Initial success metric
Operations lead	Prioritize investigation of degradation	Which locations, device models, or network types have the highest connection-failure rate in the last 24 hours?	Failure rate and affected-customer count
Product lead	Decide where to improve onboarding or reliability	Which service plans or device cohorts experience poor connectivity during their first 14 days?	Percentage of customers with repeated failures
Customer-success team	Proactively contact at-risk customers	Which active customers had at least three failed connection attempts in the past seven days?	At-risk customer count by plan and region
Data analyst	Explain trend changes	Did latency, failed connections, or transferred bytes change week over week for a market?	Daily and weekly trend measures
Internal service, later exposed through FastAPI	Retrieve a bounded customer or location summary	What is the recent 24-hour connectivity summary for a specified customer or location?	Fresh, correctly scoped response

Notice that these are questions, not tables. In the next module, we will derive facts and dimensions from their required grain. Here, they serve only to tell us what the data product must preserve.

---

Source contracts: defining what upstream data promises

A data contract is an agreement between the producer of data and the people or systems consuming it. Unlike a static wiki page, a mature contract can be machine-readable and continuously checked in the pipeline.

For a backend engineer, a data contract is close to an API contract, but with important additions. Both define fields, data types, semantics, versions, ownership, and compatibility expectations. A data contract must also address delivery timing, historical replay, quality thresholds, and how a consumer detects missing data.

A source contract should answer the following questions.

Contract element	What it specifies	Connectivity-event example
Asset identity and ownership	Name, version, producer, support contact	connectivity-events, version 1, owned by Connectivity Platform
Delivery interface	Pull or push method, authentication, pagination, retry behavior	Cursor-paginated HTTPS API with OAuth service credentials
Record identity	Stable key used to detect duplicate delivery	Globally unique, immutable event_id
Schema	Field names, types, requiredness, allowed values	event_type, status, occurred_at, device_id, and metrics
Semantics	What a field actually means	occurred_at is when the device generated the event, in UTC
Mutation behavior	Append-only, updates, deletes, or snapshots	Events are immutable; customer reference records may change
Quality rules	Validity and completeness conditions	latency_ms cannot be negative; required IDs cannot be null
Delivery and change policy	Expected arrival and how breaking changes are communicated	Additive fields permitted; breaking changes require notice and a version change
Security and classification	Sensitivity, permitted usage, handling rules	Pseudonymous customer IDs are confidential; direct contact details excluded
SLOs and incident path	Timeliness and reliability commitments	Producer supports replay of the previous seven days and reports delivery incidents

The most dangerous words in data requirements are “usually,” “near real time,” “clean,” and “complete.” They sound reasonable but cannot be tested. Replace them with a rule, measurement window, threshold, and owner.

For example:

• Vague: “Events should arrive near real time.”
• Testable: “For 99% of completed 15-minute ingestion windows in a calendar month, curated event data is available within 30 minutes of the source’s received_at timestamp.”

The source timestamp is crucial. You should distinguish:

• occurred_at: when the device or client says the event happened.
• received_at: when the source platform accepted the event.
• ingested_at: when our pipeline wrote the raw record.
• processed_at: when the record became available in a curated analytical dataset.

A late or offline device may report an event hours after it occurred. If freshness is measured only from occurred_at, the pipeline could seem to miss its target even though it processed the event promptly after receiving it.

Google’s discussion of contracts emphasizes that a contract includes schema, semantics, quality measures, and objectives such as freshness and completeness. It also shows how a version-controlled contract can drive automated checks rather than remain passive documentation.

---

Sizing the capstone realistically

A portfolio project should be large enough to require production-minded choices, but not so large that its costs or setup obscure the learning. The following is an explicit, credible planning assumption for a mid-sized international connectivity provider.

Expected sources and volumes

Source	Delivery contract	Estimated scale	Important behavior
Connectivity Events API	Cursor-paginated JSON API, queried in 15-minute windows	3 million events/day; 250 events/second peak; roughly 3 GB/day raw JSON	Append-only events; retries can redeliver records; late arrivals possible for seven days
Customer Account export	Daily JSON or CSV snapshot	150,000 active customers; up to 10,000 changes/day	Mutable reference data; selected attributes can change
Device Inventory API	Daily incremental JSON extract	300,000 devices; up to 5,000 changes/day	Mutable device status, firmware, and ownership attributes
Location reference file	Daily or weekly CSV file	5,000 locations; low change volume	Geographic metadata; stable identifiers required

The event stream is the dominant cost and reliability concern. At 3 million records a day, a 180-day initial backfill contains roughly 540 million records. That is enough scale to make partitioning, incremental processing, and query discipline meaningful later, without requiring a genuinely massive streaming environment.

Do not claim false precision. A useful brief writes “planning assumption: 3 million events/day, validated during source profiling” rather than presenting an invented estimate as a production fact.

Minimal event source contract

The contract below is intentionally source-facing. It says what the producer delivers; it does not yet prescribe warehouse table design.

asset: connectivity-events
version: v1
owner: Connectivity Platform
delivery:
  method: HTTPS cursor-paginated API
  format: newline-delimited JSON
  replay_window: 7 days
identity:
  primary_key: event_id
schema:
  required:
    - event_id
    - occurred_at
    - received_at
    - customer_id
    - device_id
    - location_id
    - event_type
    - status
  optional:
    - latency_ms
    - bytes_uploaded
    - bytes_downloaded
    - network_type
quality:
  rules:
    - event_id is unique at the producer
    - occurred_at and received_at are UTC timestamps
    - latency_ms is null or greater than or equal to zero
    - status is one of success, failure, timeout
change_management:
  additive_fields: permitted with notification
  breaking_changes: require version increment and 14 days notice

Two design choices are particularly important:

1. Stable identity makes retries safe. A pipeline can safely fetch an API page again only if it can deduplicate by an immutable event_id.
2. A replay window makes recovery possible. If a scheduled extraction fails at 02:00, the platform must be able to request the missed period again. Without replay, “reliable” processing is not possible.

---

Reliability, latency, and retention as measurable targets

Requirements must separate several kinds of reliability. A pipeline can be technically available while producing stale data; it can be fresh while silently dropping invalid records. Treat these as distinct targets.

Capstone service-level objectives

Area	Target	How it is measured
Curated-data freshness	99% of 15-minute windows available within 30 minutes of received_at, measured monthly	Compare the newest successfully curated received_at with wall-clock time
Completeness	At least 99.9% of records reported by a closed source extraction window appear in raw storage	Compare source page or manifest counts with persisted raw-record counts
Validity	At least 99.5% of delivered event records pass schema and business validation	Valid records divided by all delivered records; rejected rows remain auditable
Duplicate protection	No duplicate event_id in the canonical event dataset	Scheduled uniqueness test after loading
Recovery	Recover a failed 15-minute window and restore downstream data within 2 hours	Timed failure-recovery drill using source replay and rerun
Raw-data durability	No acknowledged raw extract is lost after successful object write	Immutable raw files, deterministic paths, and auditable extraction metadata
Consumer API freshness	API-visible summaries use curated data no more than 30 minutes old	API metadata exposes most recent successful data window

These are SLOs, internal engineering targets. An SLA is a contractual commitment with external consequences; do not casually call every internal target an SLA.

The targets also clarify a central scope decision: the initial capstone uses 15-minute micro-batch ingestion, not a continuously streaming system. That is appropriate because the consumers need recent decision support within 30 minutes. If a later use case requires operational alerting within seconds, that is a new requirement and may justify Beam/Dataflow or another streaming design. It is not something to add merely because the tools exist.

Retention and privacy requirements

Connectivity data can reveal behavior patterns, even if it does not contain names or phone numbers. The capstone therefore treats customer identifiers as pseudonymous, confidential data.

Data class	Retention target	Access and deletion requirement
Immutable raw connectivity extracts	13 months	Restricted engineering access; retain to support reprocessing and audits
Curated event-level analytics	13 months	Analytics and approved internal-service access; delete or anonymize data tied to a validated deletion request
Aggregated location and plan metrics	36 months	Retain longer for trend analysis; no direct identifiers in aggregate outputs
Customer reference data	While active, then 30 days after approved deletion	Restrict access; minimize fields and avoid direct PII in the analytics project
Quarantined invalid records	90 days	Restricted access; retain rejection reason for diagnosis, then delete
Pipeline run and quality audit metadata	13 months	Engineering and operations access for compliance and recovery evidence

The exact periods would need legal and product approval in a real organization. The key engineering requirement is that retention is intentional and enforceable, not an accidental consequence of storage defaults.

---

Your capstone requirements brief

Create docs/requirements.md in the repository that will become your portfolio project. Use the following concise version as the starting artifact for this course.

Connectivity Intelligence Platform — Requirements Brief, v0.1

Business objective
Enable operations and product leaders to identify connectivity degradation, quantify customer impact, and prioritize reliability and retention improvements using current and historical event data.

In scope

• Ingest connectivity events from a paginated source API.
• Ingest customer, device, and location reference data.
• Preserve immutable raw extracts.
• Produce validated, queryable analytical data in BigQuery.
• Serve selected bounded summaries to internal consumers through a later FastAPI service.
• Provide operational evidence of freshness, quality, and failed-run recovery.

Out of scope

• Real-time network control or automated remediation.
• Direct customer notifications.
• Ingesting direct personal contact data.
• Sub-second fraud detection or alerting.

Consumers and decisions

• Operations leads identify degraded locations, devices, and network types.
• Product leads compare reliability across plans and customer cohorts.
• Customer-success teams identify customers with repeated recent failures.
• Analysts investigate historical trends and customer-impact patterns.
• Internal services retrieve recent customer or location summaries.

Analytical questions

1. Which locations and device models have the highest failure rate over the previous 24 hours?
2. How has connection latency changed daily and weekly by region, plan, and network type?
3. Which active customers have repeated failed attempts in the previous seven days?
4. Which service plans or device cohorts show elevated connectivity problems during the first 14 days after activation?
5. How many customers were affected by a degradation event, and for how long?

Source contracts

• connectivity-events: append-only, cursor-paginated JSON API; immutable event_id; producer supports seven-day replay; source emits UTC event and receipt timestamps.
• customer-accounts: daily mutable snapshot or incremental export; only required pseudonymous identifier, status, plan reference, and activation attributes.
• device-inventory: daily incremental extract containing stable device ID, model, firmware, status, and customer association.
• location-reference: low-volume reference data containing stable location ID and approved geographic attributes.
• Breaking source changes require a new contract version and 14 days notice. Additive fields may arrive without pipeline failure but must be reviewed before use downstream.

Volume assumptions

• 3 million connectivity events per day.
• Peak rate of 250 events per second.
• Approximately 3 GB raw JSON per day.
• 150,000 active customers and 300,000 devices.
• Initial history load: 180 days of events.

Latency and reliability targets

• Curated event data is available within 30 minutes of source received_at for 99% of 15-minute windows each month.
• At least 99.9% completeness for closed source delivery windows.
• At least 99.5% record validity; invalid records are quarantined with machine-readable reasons.
• Canonical event data contains no duplicate event_id.
• A failed 15-minute window can be recovered within two hours without duplicate or omitted canonical records.
• Data-quality and freshness failures create an observable failed status and an alert.

Retention, security, and compliance

• Retain raw and curated event-level data for 13 months.
• Retain aggregated, de-identified metrics for 36 months.
• Retain quarantine data for 90 days and pipeline-quality audit metadata for 13 months.
• Treat customer IDs and customer-level connectivity history as confidential.
• Exclude direct personal contact information from the analytics project.
• Support deletion or anonymization for validated customer deletion requests.

Assumptions and risks

• The event API provides stable IDs, deterministic pagination, and a seven-day replay window.
• Source-side counts or equivalent checkpoints are available to measure completeness.
• The 30-minute freshness target is acceptable for decision support; it is not suitable for real-time operational control.
• Event semantics, especially status, event_type, and device timestamps, must be validated with the source owner before implementation.

---

A requirements brief earns its value when it removes ambiguity before code exists. For the capstone, you now have a defined business boundary, consumers and questions, four source contracts, credible scale assumptions, measurable SLOs, and explicit retention and security constraints. These decisions will prevent later modelling and pipeline work from becoming technology-first experimentation.

In the next lesson, you will turn this brief into a justified GCP architecture, deciding where Cloud Storage, BigQuery, dbt, Airflow, FastAPI, Redis, and optional Dataflow fit—and, just as importantly, where batch processing ends and streaming would begin.