Welcome back. In the last lesson, you designed the feature and data contracts that make training data trustworthy: point-in-time correct feature retrieval, immutable dataset manifests, validation evidence, and lineage. A training pipeline should consume those contracts rather than rebuild data informally from “latest” tables.
This lesson turns that foundation into a repeatable path from data to an approved model artifact. By the end, you should be able to describe a pipeline that records every experiment, pins the inputs needed for reproduction, evaluates candidates against explicit gates, and registers the approved result without confusing registration with deployment. This is the level of rigor expected when discussing how an ML team moves beyond notebooks.
Reproducibility is an operational capability
A training run is reproducible when the organization can answer, and ideally rerun, the question:
“Exactly how did we produce this model, and why was it approved?”
That requires more than saving a serialized model file. At minimum, each run needs four identities:
| Identity | What to capture |
|---|---|
| Code | Git commit SHA, pipeline/component version, and transformation revision |
| Data | Immutable dataset manifest, label definition, feature versions, source snapshots, and split definition |
| Environment | Container image digest, package lockfile, runtime configuration, and relevant hardware details |
| Procedure | Hyperparameters, random seeds, training command, evaluation policy version, and execution timestamps |
The prior lesson established the data identity. The training system must preserve the other three, then connect all four to the resulting model and evaluation evidence.
There are two useful levels of reproduction:
- Audit reproduction: retrieve the exact inputs, artifacts, configuration, and decisions that explain a historical model.
- Computational reproduction: rerun the pipeline from those inputs and obtain materially equivalent results.
Exact bit-for-bit reruns are not always realistic, particularly with distributed or GPU training. Nondeterministic operations, changing drivers, and floating-point behavior can produce small differences. That is not an excuse for weak metadata: pin the environment, set seeds where supported, record the hardware/runtime, and define acceptable metric tolerances.
MLflow’s tracking vocabulary is a practical way to organize this information: an experiment groups work on a task, a run records one execution, and a logged model is a produced model artifact with its own metadata.
ML Experiment Tracking | MLflow AI Platform
Read MLflow’s tracking overview to establish a concrete mental model for experiments, runs, metadata, artifacts, and shared tracking infrastructure. Treat MLflow as one possible implementation of the architecture, not as the architecture itself.
In the “Concepts” section, read run records to distinguish parameters and metrics from output artifacts. Then, in “Tracking Runs,” scan explicit logging and note why auto-logging is useful but not sufficient for organization-specific metadata. Next, go to “Set up the MLflow Tracking Environment,” especially “Components.” Read the explanations of the backend store and the artifact store. Finally, in the FAQ section “How to integrate MLflow Tracking with Model Registry?”, read the registry requirement. Focus on the separation between metadata storage, large artifact storage, and a registry record.
A useful management principle follows:
A run ID is evidence of an execution. It is not, by itself, evidence that the model is safe, approved, or deployed.
The pipeline: from governed data to a candidate model
An orchestrator turns a collection of scripts into a controlled workflow. It handles scheduling or triggers, dependencies, retries, credentials, status, and the durable record of each pipeline execution. The orchestrator could be a managed ML platform, a workflow scheduler, or an internal service; in a system-design interview, the control boundaries matter more than its brand name.
For a fraud-ranking example, a well-scoped pipeline has the following stages.
| Stage | Main output | Essential control |
|---|---|---|
| 1. Resolve inputs | Training dataset manifest | Pin feature versions, label query revision, time interval, and source snapshot identifiers |
| 2. Validate inputs | Validation report | Block training on schema breaks, severe missingness, invalid labels, or failed point-in-time retrieval |
| 3. Prepare data | Immutable prepared dataset and split manifest | Record preprocessing version, split logic, random seed, and feature schema |
| 4. Train candidates | Logged model artifacts | Record code, environment, hyperparameters, seed, training metrics, and resource use |
| 5. Evaluate | Versioned evaluation report | Measure predictive performance, guardrails, segments, and serving compatibility |
| 6. Apply approval gate | Pass/fail decision with reasons | Compare with a declared baseline and thresholds, not a human memory of past results |
| 7. Register approved model | Registry version and release metadata | Link the approved model to its run, dataset, evaluation report, and input contract |
The first two stages are deliberately before training. If the latest label job changed a target’s meaning, or an upstream feature is mostly null, spending GPU budget on training merely creates a well-tracked bad model. The pipeline should fail visibly, preserve its validation report, and alert the appropriate data or feature owner.
Make stages restartable and idempotent
A production pipeline will fail sometimes: a worker can be preempted, an artifact upload can time out, or a downstream warehouse can be temporarily unavailable. Design stages so retries do not silently produce conflicting outputs.
That normally means:
- Assign a unique pipeline execution ID and run ID.
- Write outputs to execution-specific or content-addressed locations rather than overwriting a generic
latest/model.pkl. - Persist each completed stage’s output references and validation status.
- Allow a retry to reuse verified upstream outputs when their pinned inputs have not changed.
- Treat a failed validation as a terminal business result, not merely an infrastructure error to retry repeatedly.
For example, the preparation stage might emit a dataset manifest containing a URI and checksum for the prepared data, feature-schema hash, split assignment, and transformation image digest. Training consumes that manifest as an explicit input; it does not query the warehouse again using a vague date filter.
This separation makes it possible to compare model candidates fairly. If two algorithms are being compared, they should normally use the same sealed training and evaluation data. If they use different feature versions or label logic, that difference must be visible in the run record.
Track experiments as structured evidence, not a notebook diary
The experiment tracker is the index that lets an engineer or manager compare runs later. Organize one experiment around a coherent product objective, such as fraud-risk-ranking or search-ranking-v3, rather than mixing unrelated models in a catch-all experiment.
Within each run, log information in categories that remain searchable:
| Category | Example fields |
|---|---|
| Parameters | algorithm family, learning rate, tree depth, regularization, seed |
| Metrics | PR-AUC, recall at a review threshold, calibration error, training duration |
| Tags and context | Git SHA, dataset manifest ID, feature-contract version, environment, owner, ticket |
| Artifacts | model package, preprocessing objects, evaluation report, confusion matrices, calibration plots, schema/signature, dependency manifest |
| Lineage references | previous production model, parent run, source data snapshot, pipeline execution ID |
Use auto-logging to capture common framework-level parameters and model details quickly. Still add explicit domain metadata. A framework can infer the number of trees in a model; it cannot infer that the model used merchant_risk_features_v2, that labels were delayed by 30 days, or that this run implements a policy change requested by Risk.
A training run should also log failed outcomes. A candidate that fails data validation or misses a fairness guardrail is useful organizational knowledge. Otherwise, teams repeat failed experiments, and later reviewers see only a curated history of successes.
The tracker and artifact store are separate for a reason. Metrics, parameters, and run status are small searchable metadata, typically stored in a database-backed backend. Model files, reports, plots, and possibly data samples are larger artifacts, usually kept in object storage. For shared work, place both behind a centrally managed tracking service with access controls rather than relying on every engineer’s laptop directory.
For an interview answer, a concise statement is:
“For individual prototyping, local tracking is sufficient. For a team, I would use a remote tracking service, a database-backed metadata store, and access-controlled object storage for artifacts. Runs record immutable references to code, data, environment, and evaluation evidence.”
Evaluation is a gate, not a leaderboard
A candidate with the highest single metric is not automatically the best release choice. Offline evaluation should be a policy-controlled decision process.
First, distinguish the datasets’ jobs:
- Training set: fits the model.
- Validation set: guides hyperparameter selection and candidate ranking.
- Sealed test set: provides a final, less-biased estimate after selection.
- Production reference data: may be used to compare the current approved model and the candidate under the same evaluation definition.
Repeatedly selecting based on one test set gradually overfits the organization to that test set. In a mature workflow, the policy states which metric is used for selection, which evidence is required for final approval, and when a test set must be refreshed.
A candidate gate often contains five kinds of checks.
- Primary business metric. For fraud, this might be recall at a fixed manual-review capacity, rather than raw accuracy.
- Baseline comparison. The candidate must meet an absolute floor and improve upon the current approved model by a meaningful margin.
- Guardrails and segments. Check false-positive rates, calibration, performance across regions or customer cohorts, and business-cost constraints. An overall improvement can hide a severe regression in a high-impact segment.
- Data and contract checks. Confirm the expected feature schema, missing-value behavior, and model input signature.
- Operational checks. Verify that the model package loads in the serving-compatible environment and stays within size, latency, or memory limits.
The gate should produce an immutable report, not just a green pipeline icon. A useful approval record says which policy version was applied, the baseline model version, input dataset IDs, metric values and uncertainty where relevant, segment results, and the explicit pass or fail reason.
MLOps: Continuous delivery and automation pipelines in ...
Read Google Cloud’s architecture guidance for the distinction between automated data validation, offline model validation, and execution metadata. It is useful for turning “evaluate the model” into concrete release criteria.
In the “Data and model validation” subsection, read the validation discussion. Focus on the difference between a schema problem that should stop a pipeline, a distribution shift that may motivate retraining, and model evaluation against both a baseline and meaningful segments. Then read the “Metadata management” subsection, beginning with execution metadata. Notice that pointers to intermediate artifacts make failed executions diagnosable and allow safe resumption without pretending completed work never happened.
The gate outcome should be understandable without inspecting Python logs:
- Passed: “Candidate exceeds the approved model’s recall-at-capacity by 2.1 percentage points, stays within the false-positive and latency budgets, and passed all schema checks.”
- Failed: “Candidate improved overall recall but exceeded the maximum false-positive rate in the new-user cohort.”
- Inconclusive: “The observed gain is smaller than the required margin given evaluation uncertainty; retain the baseline and collect more data.”
This process prevents a common management failure: promoting a model because a dashboard shows a slightly larger headline metric, while hiding costs or regressions that the product team actually cares about.
Register the approved artifact
A model registry is the governed catalog of model versions that have a known identity and lifecycle metadata. Its central job is to answer:
- Which exact artifact is model version 17?
- Which training run and dataset produced it?
- What input schema and feature contract does it require?
- What evaluation evidence and approvals support it?
- Which version is the currently approved reference for comparison?
Registration should store a durable link to the already logged model artifact. It should not copy a manually downloaded file from an engineer’s machine into production.
The registry metadata should include:
| Registry field | Why it matters |
|---|---|
| Model name and immutable version | Gives consumers a stable, unambiguous identifier |
| Source run and artifact URI | Preserves traceability to experiments and files |
| Dataset and feature-contract references | Explains what the model expects and what trained it |
| Evaluation report and policy version | Shows why the version passed approval |
| Model signature and dependency environment | Supports serving compatibility checks |
| Owner, approver, purpose, and risk classification | Establishes accountability |
| Release labels or aliases | Gives controlled names such as approved or champion without losing immutable version history |
A practical policy is to let every experimental run log a model artifact, but only let models that pass automated gates and required review receive the approved alias or an equivalent approval status. The product can then identify a stable approved reference without relying on ambiguous folder names such as final_v2_really_final.
Registration is not deployment. It says, “this artifact is an approved, traceable release candidate.” The next lesson will cover the separate serving decision: shadow evaluation, canary rollout, controlled promotion, and rollback.
MLFlow Tutorial | ML Ops Tutorial
Watch “MLFlow Tutorial | ML Ops Tutorial” by codebasics for a concrete walk-through of registering a logged model, adding descriptions and tags, and using aliases such as Champion and Challenger.
Watch model registration. Focus on the sequence from a tracked run and logged artifact to a registry entry with meaningful metadata and aliases. The segment also touches on moving a model toward production; for this lesson, concentrate on traceability and approval records rather than deployment mechanics.
The managerial design answer
The technical design works only when ownership is explicit. A reasonable division is:
| Boundary | Accountable role |
|---|---|
| Feature and label definitions | Data or feature owners, with domain review |
| Pipeline platform, credentials, and runtime standards | ML platform team |
| Model code, experiments, and candidate analysis | Applied ML or data science team |
| Evaluation policy and business guardrails | ML team together with product, risk, or domain stakeholders |
| Approval evidence and registry metadata | Model owner, with designated approver for higher-risk use cases |
The key is not to require a manager to manually inspect every model. It is to establish clear policy for what can pass automatically, what needs review, who is notified on failure, and how exceptions are documented.
In a time-boxed system-design interview, you can communicate the full idea in roughly this form:
“The training pipeline begins by resolving an immutable training-data manifest built from point-in-time correct, versioned features and labels. It validates schema, quality, and label completeness before any training starts. Each candidate run logs code revision, container image, parameters, seed, dataset and feature versions, metrics, and artifacts to centralized tracking and object storage.
“An automated evaluation gate compares the candidate with the currently approved model on a fixed metric and business guardrails, including segment-level performance and serving compatibility. The gate persists its report and reasons. If it passes the versioned approval policy, the pipeline registers the existing model artifact with links to its source run, data manifest, feature contract, evaluation report, owner, and approval status. Registration does not change serving traffic; release promotion is a separate controlled workflow.”
Three anti-patterns are worth naming if prompted about failure modes:
- Training from mutable “latest” tables, which makes later investigation impossible.
- Choosing the candidate solely by its best validation metric, which ignores baselines, segments, and operational constraints.
- Treating a model registry as a shared folder, with manual uploads and no link to the producing run or evaluation evidence.
Key takeaways
A reproducible training pipeline produces more than a model. It produces a connected evidence record:
- Pin code, data, environment, and procedure for every run.
- Track experiments centrally with searchable metadata and durable artifacts.
- Validate input data before training; make stages restartable without overwriting historical outputs.
- Evaluate candidates with a versioned policy: primary metric, baseline comparison, segment guardrails, and serving compatibility.
- Register only approved artifacts with links to their run, dataset, feature contract, evaluation report, ownership, and status.
- Keep registration separate from production deployment.
Next, you will design the release workflow that takes an approved registry version through automated checks, shadow or canary exposure, controlled promotion, and rollback.
Can't find a good explanation? Sign up and we'll make it for you
Sign up