Choosing Consistency Models for Features, Predictions, and Model Metadata

Welcome back. In the previous lesson, you distinguished batch, streaming, and online inference by the consuming workflow’s latency, throughput, temporal, and failure-handling requirements. We ended with the key observation that each mode needs consistency, but not necessarily the same consistency.

This lesson makes that operational. You will select consistency models separately for features, predictions, and model metadata rather than treating “strong consistency” as a universal requirement. The aim is to state a defensible contract in a system design: what must be fresh or ordered, what may be stale, what must be immutable, and what the system should do during a partition or failed rollout.

---

Consistency is an ML contract, not a datastore setting

A database may offer linearizable reads, replicas with eventual consistency, transactions, or versioned objects. Those primitives matter, but they are not the requirement itself. In an ML system, consistency has at least three distinct meanings:

1. Value consistency
   Does a read observe the required version of a feature or metadata record?

2. Temporal consistency
   Was the value actually available at the time the training or prediction decision was made?

3. Semantic and artifact consistency
   Do feature definitions, preprocessing logic, model artifacts, prompts, and policies refer to compatible versions?

A feature can be strongly consistent at the storage layer and still be unusable for training if it leaks future information. Conversely, an online feature can be a few minutes stale and still be correct for a support-priority model if the business accepts that staleness.

A useful framing is:

Choose the weakest consistency guarantee that prevents a materially incorrect or irrecoverable business outcome.

“Weakest” does not mean careless. It means avoiding an expensive global coordination requirement where a versioned snapshot, bounded staleness limit, or idempotent write is sufficient.

A practical vocabulary

Model or mechanism	What it gives you	Typical ML use
Linearizable read or write	A read reflects the most recently completed write in real time for an object or key.	Inventory, account balance, entitlement, or a release pointer whose stale value could create an unsafe action.
Serializable transaction	Concurrent operations appear equivalent to some serial order across multiple records.	Atomic promotion of a model, evaluation report, approval state, and deployment manifest.
Read-your-writes / session consistency	A client does not lose sight of its own acknowledged update.	A data scientist registers an evaluation result and immediately views the same experiment state.
Bounded staleness	A value may be old, but no older than an explicit limit.	Customer profile features within 15 minutes; support-ticket aggregates within five minutes.
Eventual consistency	Replicas converge if writes stop, but reads may be stale for an unbounded period.	Search indexes, noncritical dashboards, asynchronously replicated embeddings, and discovery metadata.
Immutable version plus atomic pointer	Artifacts never change after publication; a small mutable pointer selects the active version.	Model binaries, feature schemas, prompt packages, batch prediction outputs, deployment manifests.

Do not overgeneralize from CAP terminology. CAP concerns the behavior of a distributed system during a network partition. It does not prescribe one consistency choice for an entire ML platform, and it says nothing directly about feature leakage or training-serving parity.

The next sections apply these guarantees to the three ML assets that are commonly—and mistakenly—lumped together.

---

Features: select consistency from the decision’s time semantics

A feature is not merely “the latest value for an entity.” It is a statement about an entity at a specified time, produced by a named transformation from particular source data.

For a training example with label time $t_{\text{label}}$, the feature value must have been observable at the prediction time:

$$t_{\text{feature event}}\le t_{\text{label}}$$

That condition alone may not be enough. A source can backfill or correct a record after the original event. If the objective is to reproduce exactly what production could have known, feature retrieval also needs an availability-time constraint:

$$t_{\text{feature available}}\le t_{\text{label}}$$

This is point-in-time correctness. It is usually more important for offline training and retrospective evaluation than obtaining a strongly consistent read from the current feature store.

Four feature-consistency patterns

1. Point-in-time snapshot consistency for training and batch scoring

Use this when historical labels or a scheduled prediction run must correspond to a coherent cutoff.

For example, a day-ahead energy forecast training dataset must join weather forecasts, realized generation, market signals, and labels according to what was available at each forecast issuance time. Joining today’s corrected market data into a historical training row would produce an unrealistically favorable evaluation.

The contract should include:

• entity key and event timestamp;
• feature event timestamp;
• feature availability or created timestamp, where corrections exist;
• allowed lookback or TTL;
• transformation and feature-schema versions;
• treatment of missing features.

For batch scoring, the equivalent contract is usually snapshot consistency plus atomic publication. All scores in the published partition should refer to one input cutoff, one feature definition set, and one model version.

2. Bounded staleness for most online ML features

Many online models do not need a linearizable read of every feature. A ticket-routing model can generally use a customer tier that is up to 24 hours old and an open-incident count that is up to five minutes old. Requiring synchronous cross-region reads for both may turn an otherwise reliable $p95$ latency target into a fragile service.

State the maximum feature age explicitly:

$$\text{feature age}=t_{\text{decision}}-t_{\text{feature available}}\le S$$

where $S$ is the allowed staleness bound.

The service must then measure the age, not simply assume replication is prompt. If the store returns a value 40 minutes old when the limit is five minutes, the correct behavior may be to use a fallback model, return an explicit “insufficient data” result, or reject a high-risk action.

3. Strongly consistent authoritative reads for irreversible decisions

Use a linearizable or transactional read when the model’s action depends on a rapidly changing resource and an incorrect decision has direct side effects.

Examples include:

• inventory available before a recommendation immediately reserves an item;
• user authorization before an agent executes a privileged tool;
• account balance or credit limit before approving a transaction;
• a current policy flag that disables a model for a regulated customer segment.

Here, a feature store replica is often the wrong authority. The design may use low-latency replicated features for prediction context but read the critical field from the transactional source of truth immediately before the side effect.

4. Event-time state consistency for streaming features

A streaming anomaly detector that uses rolling ten-minute telemetry statistics needs more than “latest value” semantics. It needs a declared event-time policy, late-event tolerance, deduplication rule, and recovery behavior for stateful aggregates.

If an event is replayed after a failure, the feature state must not double-count it. This is usually addressed with a combination of durable stream-processing state, event identifiers, and idempotent downstream writes. The business requirement is often effectively-once feature and action behavior, not a vague assertion that every external system is “exactly once.”

The architecture diagram is useful because it separates two questions:

• Can the platform provide a common, versioned definition of the feature?
• What temporal and read-consistency contract does this particular decision require?

The first should be standardized. The second is specific to the model and the harm caused by a stale value.

---

Predictions: preserve identity, ordering, and publication semantics

Predictions are often treated as disposable API responses. That is appropriate for some cases, such as a transient ranking score used to render a page. It is inadequate when predictions drive workflows, audits, retraining data, or state-changing actions.

The relevant question is:

Is this prediction merely a response, an immutable record, or a command that causes a side effect?

A synchronous response needs request-level coherence

For online inference, each accepted request should be attributable to:

• one validated request schema;
• one feature schema and feature-value set;
• one loaded model artifact;
• one preprocessing and policy version;
• one response or explicit error.

During a canary deployment, different requests may legitimately use different model versions. That is not inconsistency if routing is explicit and each response records the version that produced it. What is dangerous is an untraceable mixture: preprocessing from version $A$, model weights from version $B$, and threshold policy from version $C$.

A robust request log contains at least a request ID, timestamp, model version, feature-contract version, decision policy version, and outcome. Whether raw feature values can be logged depends on privacy and security constraints; the identifiers and hashes often suffice for lineage.

A stored prediction should usually be immutable

For batch scoring, streaming scores, and decisions that may be reviewed later, treat prediction output as an immutable fact:

$$\text{prediction key}=(\text{entity or event ID},\text{prediction time},\text{model version})$$

This permits:

• reproducible analysis of what the system decided;
• comparison between model versions;
• safe backfills without overwriting history;
• recovery from interrupted jobs;
• reconstruction of training and evaluation cohorts.

A mutable table such as current_risk_score can still be useful for downstream applications. But it should be a materialized “latest view,” derived from immutable prediction records or atomically replaced by a complete batch output.

Side effects require idempotency, not only reliable inference

Suppose a streaming malware pipeline produces a high-confidence detection that opens a remediation ticket. A failure can occur after the inference result is computed but before the pipeline records that the ticket was successfully created. Retrying may create duplicates.

The proper consistency contract is not “the model must run exactly once.” It is:

• score events at least once if loss is unacceptable;
• assign a deterministic decision or idempotency key;
• make the ticket-creation sink deduplicate or upsert on that key;
• record whether the action was accepted, rejected, or pending.

A reasonable key could include the source event ID, model version, and action type. Including model version is often important: a later model may legitimately generate a different action for the same file or transaction.

Atomic publication matters more than row-by-row freshness

For a nightly batch prediction table, consumers should not see 60% of today’s scores mixed with 40% of yesterday’s scores unless the product explicitly supports partial availability. Prefer one of these patterns:

• write a new immutable partition and atomically update a manifest;
• validate a new table and atomically swap the consumer-facing view;
• publish a completion marker only after row-count and quality checks pass.

This is a snapshot consistency requirement for the prediction dataset. It does not require every prediction to be globally serialized as it is computed.

---

Model metadata: strong control plane, resilient data plane

Model metadata includes model versions, artifact digests, training datasets, code revisions, evaluation reports, approval states, aliases, prompts, safety policies, and deployment manifests. These records control what may be served. Their consistency requirements are therefore stricter than those of many high-volume feature reads.

A practical design separates:

• the control plane, which registers, evaluates, approves, promotes, and deploys artifacts;
• the data plane, which loads an approved artifact and serves inference traffic.

The control plane should generally fail closed. If the system cannot determine whether a model was approved, it should not silently deploy it.

Immutable versions, controlled aliases

Model version $42$ should not be modified in place after registration. Its artifact digest, signature, training-data reference, code revision, and evaluation summary should remain stable.

A mutable alias such as production, champion, or candidate is different. It is a pointer that selects a version. Updating that pointer is a high-impact control-plane operation and usually warrants a linearizable compare-and-set or serializable transaction.

A promotion transaction should validate that:

• the target model version exists and its artifact digest is verified;
• its evaluation report satisfies the release gate;
• required approvals are present;
• the intended environment is correct;
• the alias has not changed unexpectedly since the release process began;
• the audit record is appended atomically with the alias update.

This prevents a race in which two deployment processes both believe they promoted their candidate successfully.

Do not put the registry in the request path

It would be counterproductive for every inference request to synchronously query a strongly consistent model registry. Serving should instead work from a locally loaded, verified deployment manifest.

A resilient pattern is:

1. The release controller resolves the approved alias and writes an immutable deployment manifest.
2. Serving instances verify the artifact digest and load the referenced model, policy, and schema bundle.
3. The service atomically switches new requests to the loaded bundle.
4. Requests record the precise bundle and model version used.
5. If the registry becomes unavailable, already deployed and verified models continue serving according to a predefined rollback and expiry policy.

This yields strong consistency where it matters—promotion and authorization—without coupling inference availability to registry availability.

Metadata consistency is also lineage consistency

A registered model is not adequately identified by model_name:version alone. At minimum, attach or reference:

Metadata field	Why it matters
Immutable artifact digest	Proves the bytes deployed are the bytes evaluated.
Model signature and feature schema	Prevents incompatible request or feature vectors.
Training dataset or snapshot reference	Supports reproducibility and incident investigation.
Code and environment revision	Captures preprocessing, dependency, and runtime behavior.
Evaluation dataset and report version	Shows what quality evidence supported promotion.
Approval and policy state	Makes governance decisions auditable.
Deployment manifest and rollout ID	Connects registry state to actual serving infrastructure.

For LLM applications, the same principle applies to prompt templates, retrieval configuration, tool schemas, guardrail policies, and evaluation criteria. A model weight change is not the only production change that can alter behavior.

---

Selecting a model under realistic scenarios

The table below illustrates that one system commonly combines several consistency models.

Scenario	Features	Predictions	Model metadata
Nightly churn or energy forecast batch	Point-in-time snapshot at a defined cutoff; versioned transformations.	Immutable output partition; atomic publication after validation.	Immutable model and schema bundle; approved release alias resolved before the run.
Online support-ticket routing	Request text exact; account tier bounded to 24 hours; open-ticket count bounded to five minutes.	Per-request coherent response, with traceable model and policy versions.	Promotion strongly controlled; serving instances use cached, verified manifest.
Transaction or entitlement decision	Strong read from authoritative balance, inventory, or authorization source for the critical field; other features may be bounded stale.	Idempotent decision record and idempotent side effect under client retry.	Strongly controlled approval and rollback; fail closed if an unapproved bundle would be selected.
Streaming telemetry or malware events	Event-time windows, explicit late-data policy, deduplicated state updates.	Immutable event score; effectively-once alert or ticket creation through idempotency key.	Deployment version attached to every event result; rollout state is auditable.
Semantic search and document embeddings	Eventual replication often acceptable, but document version and access-control metadata must be respected.	Query result need not be persisted unless used for audit or feedback.	Index and embedding-model versions must be traceable; index publication should be atomic at collection level.

The important senior-level move is to explain why the row has those choices. For example:

“The routing model can use a five-minute stale workload feature because queue changes do not create an irreversible action. It cannot use an unversioned feature definition, because that would invalidate the trained model’s input contract. The model registry promotion is strongly controlled, but inference instances do not query the registry per request; they serve a verified deployment manifest and record its version.”

That statement covers staleness, semantic compatibility, control-plane integrity, and data-plane availability without claiming that the whole system must be strongly consistent.

---

A review template for design discussions

When asked to choose a consistency model, avoid answering with only a technology choice. Write a compact contract for each asset:

1. Name the asset and its authority
   Is it a derived feature, an authoritative transactional value, a model artifact, or a prediction record?

2. Name the reader and decision
   Who consumes it, and what happens if the value is stale, missing, reordered, duplicated, or incompatible?

3. Specify time semantics
   Is this “as of event time,” “latest within five minutes,” “latest committed value,” or “one daily snapshot”?

4. Specify version semantics
   Which feature schema, model digest, prompt package, policy, or deployment manifest is valid?

5. Specify partition and retry behavior
   Serve stale data within a bound, fall back, reject, queue, retry, or fail closed?

6. Specify observability
   Measure feature age, replication lag, missing-feature rate, model-bundle version, promotion conflicts, duplicate actions, and stale-read fallbacks.

This discipline prevents a common design failure: selecting a highly available feature store, a model registry, and an inference endpoint independently, then discovering that their combined behavior cannot be explained or reproduced.

---

Key takeaways

• Consistency in ML systems includes value freshness, historical time correctness, and semantic compatibility of versioned artifacts.
• For historical training and batch work, use point-in-time joins and, where relevant, availability-time filtering to prevent future information and backfills leaking into the past.
• For online features, bounded staleness is often the right contract; reserve strongly consistent authoritative reads for decisions where stale state can cause an irreversible or unsafe action.
• Treat operational predictions as immutable facts, publish batch outputs atomically, and use idempotency to achieve effectively-once business actions under retries.
• Treat model metadata as a strongly controlled control plane: immutable artifacts, auditable versioning, and transactional promotion of aliases or deployment manifests.
• Keep the registry out of the request path. Serving should use a locally verified, versioned model bundle and annotate every prediction with its lineage.

Next, you will use these ideas to identify training-serving skew: the gap that appears when offline feature construction, online feature retrieval, preprocessing, or artifact versioning no longer describe the same prediction contract.