Choosing the Right Message Processing Guarantee

Hello. In the previous lesson, you compared event notification with event-carried state transfer and saw why a consumer-owned replica needs versioning and duplicate-safe updates. That naturally leads to the delivery question: what should happen if an event is lost, or if the same logical event is processed twice?

This lesson gives you a decision framework for selecting at-most-once, at-least-once, or effectively-once processing. The goal is not to memorize broker terminology, but to state the business consequence of loss and duplication, identify the failure boundary, and choose a proportionate design—exactly the reasoning expected in a Technical Lead interview.

---

Delivery semantics are choices about failure

A message flow has several uncertain moments:

• A producer sends a message but times out before receiving confirmation. Did the broker store it, or not?
• A consumer performs its database update but crashes before acknowledging the message.
• The broker redelivers because it never observed the acknowledgement.
• An operator replays a dead-lettered event long after the original incident.
• Two consumer instances receive duplicate copies concurrently.

The central fact is uncomfortable but useful:

After a timeout or crash, a distributed system often cannot know whether an operation happened. Retrying prevents loss, but creates the possibility of duplication.

The three processing choices make different trade-offs.

Processing choice	What you accept	What you try to prevent	Typical use
At-most-once	A message may be lost.	Reprocessing/redelivery.	Low-value, disposable signals.
At-least-once	A message may be processed more than once.	Silent loss.	Most broker consumers where duplicates are harmless or tolerable.
Effectively-once	The transport may redeliver.	Repeated business effects for one logical message.	Financial actions, resource creation, user notifications, durable projections.

The distinction between delivery and business processing matters. “Exactly once” is frequently used too broadly. A broker may guarantee something precisely within the broker, yet have no control over an email provider, a payment gateway, or your PostgreSQL database.

For that reason, use effectively-once processing as the practical architectural term:

• the message may arrive more than once;
• the consumer recognizes the same logical message;
• the intended business result becomes visible once within a clearly stated boundary.

For example, an OrderPlaced event may be delivered twice to Shipping, but Shipping creates only one label for that order. That is effectively-once processing for label creation—not magical “exactly once delivery across the Internet.”

---

The consumer crash window explains the trade-off

Kafka’s design documentation presents the key distinction through the order in which a consumer saves its position and performs its processing. Although Kafka calls the saved position an offset commit, the same idea applies to a RabbitMQ acknowledgement.

At-most-once: acknowledge before doing the work

An at-most-once consumer records its progress first, then performs the business action.

Step	Result if the process crashes now
1. Receive message	Broker still considers it pending.
2. Commit offset or acknowledge	Broker considers it handled.
3. Update application state	If this step never happens, the message is lost from the application’s perspective.

This deliberately prioritizes avoiding repeat processing over avoiding loss.

A realistic example is a high-volume, best-effort telemetry signal such as UiHoverObserved. Losing a few observations does not alter a customer’s account, inventory, or money movement. You may prefer an approximate metric over the latency, storage, and retry costs of reliable processing.

Do not select at-most-once merely because high throughput sounds attractive. A lost PaymentAuthorized, OrderCancelled, or CustomerConsentRevoked event can create incorrect business state that may be difficult to detect later.

Also, committing before processing only addresses one consumer-side redelivery window. It does not prove that the producer never emitted two logical copies. Stable event identities are still important.

At-least-once: do the work before acknowledging

An at-least-once consumer processes successfully first and only then confirms progress.

Step	Result if the process crashes now
1. Receive message	Broker still considers it pending.
2. Apply application update	Business effect may now exist.
3. Commit offset or acknowledge	If this does not happen, the broker redelivers.

If the crash occurs between steps 2 and 3, the next consumer processes the same message again. That is not a broker bug; it is the price paid to avoid silently losing work.

At-least-once is normally the right transport and retry posture when loss matters. But by itself it is not automatically safe. You must explicitly ask:

If this handler runs twice for the same logical event, what happens?

If the answer is “we increment the balance twice” or “we send two customer emails,” at-least-once delivery needs an idempotency design.

Effectively-once: at-least-once transport, duplicate-safe outcome

Effectively-once processing keeps the loss-resistant ordering of at-least-once:

1. Receive the message.
2. Apply the business outcome and record that the message was processed atomically.
3. Acknowledge only after that transaction succeeds.

If the consumer crashes before the database transaction commits, neither the state change nor the processed marker exists, so redelivery safely retries. If it crashes after commit but before the acknowledgement, redelivery finds the marker and produces no second business effect.

This is the core pattern behind an idempotent consumer, sometimes called an inbox pattern.

---

Why duplicates are normal, not exceptional

The following short segment shows the common acknowledgement-related failure sequence. It is worth watching because it corrects the instinct to treat duplicate delivery as a rare edge case.

The Azure Architecture Center gives the same broader view: producer retries, missing acknowledgements, and consumer failures after a durable write can all cause duplicates.

---

The idempotent-consumer boundary

The diagram below depicts the essential effectively-once pattern. The consumer writes a row identifying the processed message and updates its own application data in one database transaction.

The decisive property is atomicity. Consider a Shipping consumer handling OrderPlaced.

begin transaction

insert into processed_message
    (consumer_name, event_source, event_id, processed_at)
values
    ('shipping-label-creator', 'urn:acme:orders', 'E-1842', now());

insert into shipping_label
    (order_id, label_status)
values
    ('O-4821', 'CREATED');

commit transaction

The table should enforce a unique constraint such as:

unique (consumer_name, event_source, event_id)

The previous lesson’s event envelope included source and id. Together, they identify the producer’s logical event more safely than a broker delivery tag or a receive timestamp. A RabbitMQ delivery tag can change on redelivery; a Kafka offset is specific to a topic partition and may not identify the same business event across replay or forwarding.

Including consumer_name is important. The same OrderPlaced event should legitimately be processed once by Shipping and once by Analytics. A shared table keyed only by event_id would incorrectly let Shipping’s success suppress Analytics’ work.

Concurrent duplicates must be settled by the database

Avoid this unsafe sequence:

1. Query whether the event ID exists.
2. If absent, create the shipping label.
3. Insert the processed-event row.

Two consumer instances can both observe “absent” before either writes. Both may then create a label.

A database unique constraint is the arbiter. Both transactions attempt to claim the same processed-message key; only one may commit. The losing transaction rolls back, recognizes a duplicate-key violation, and treats the event as already processed. It can then acknowledge the delivery.

This is one reason a simple in-memory Set<String> is not production deduplication: it is lost on restart, is not shared by instances, and cannot participate in the business database transaction.

---

Natural idempotency versus a deduplication inbox

An inbox table is powerful, but it is not always required. First ask whether the operation can be made naturally idempotent.

Compare these handlers for CustomerShippingProfileUpdated:

Handler design	Duplicate result	Idempotent?
shippingAddress = incomingAddress	The same absolute state is written again.	Usually yes.
Upsert a read-model row by customerId, only when event version is newer	Same or stale event makes no change.	Yes.
deliveryCount = deliveryCount + 1	Count increases again.	No.
Send “your order shipped” email	Customer receives it again.	No, unless email provider honors an idempotency key.
Charge a credit card	Customer may be charged again.	No, unless payment provider enforces an idempotency key.

The snapshot-style event-carried state transfer from the previous lesson is often naturally idempotent:

{
  "id": "E-1842",
  "source": "urn:acme:customers",
  "type": "com.acme.customer.shipping-profile-updated.v1",
  "data": {
    "customerId": "C-881",
    "version": 42,
    "shippingAddress": {
      "line1": "18 Market Street",
      "city": "Leeds",
      "postalCode": "LS1 4AB",
      "country": "GB"
    }
  }
}

A projection can update its row only when the incoming version exceeds the stored version. Reapplying version 42 leaves the projection in the same state. This protects against duplicates and stale arrivals, though it does not by itself protect a separate downstream side effect such as sending an email.

A useful rule is:

Prefer a naturally idempotent state transition where it represents the business truth. Use a durable inbox when the handler creates a non-repeatable effect or when natural idempotency cannot be established confidently.

---

External side effects are the hard case

A local PostgreSQL transaction can atomically commit:

• a processed-message marker, and
• data in the same PostgreSQL database.

It cannot atomically include a third-party payment API or email provider unless that system participates in a distributed transaction—which is usually impractical and undesirable.

For an external call, propagate the same stable idempotency key:

payment provider idempotency key = source + ":" + event id

The receiving system must persist that key and return the original result on retries. If the provider does not support idempotency, you have an unresolved ambiguity:

• The consumer could crash after the provider charged the card but before it records success locally.
• On redelivery, you cannot know whether retrying would charge again.

In that situation, “exactly once” is not a configuration option. You must change the integration contract, use a provider that supports idempotency, introduce reconciliation, or redesign the workflow.

This boundary statement is particularly valuable in interviews:

“I can guarantee one local database outcome by committing the inbox marker and state update together. For a payment provider, I propagate the same idempotency key and rely on the provider to deduplicate. Without that cooperation, I would provide reconciliation rather than claim end-to-end exactly-once.”

---

A selection framework for business requirements

Choose the semantic from the cost of the wrong outcome, not from Kafka versus RabbitMQ.

1. Is losing the message acceptable?

If yes, at-most-once may be sufficient.

Example: A live dashboard receives mouse-movement signals. Missing an occasional signal has no durable business consequence. Retrying every failed sample is not worth the load or complexity.

If no, start from at-least-once. You need retry and redelivery rather than silent discard.

2. Is duplicate processing harmless, tolerable, or unacceptable?

• Harmless: A search index receives an absolute product snapshot and upserts it by product ID and version.
• Tolerable: A diagnostic analytics stream may accept a small degree of duplicate raw telemetry, with approximate aggregates.
• Unacceptable: A handler creates an account, dispatches a physical shipment, sends a customer-facing notification, or initiates a payment.

When duplicates are unacceptable, add idempotency and select effectively-once processing for the relevant business outcome.

3. Where does the side effect occur?

Side-effect location	Feasible effectively-once approach
Same consumer database	Processed-message marker and business data in one transaction.
Consumer-owned read model	Upsert by business key and version; inbox if necessary.
Another internal service	Propagate event identity or idempotency key; downstream service deduplicates independently.
External provider	Use provider-supported idempotency key and record the resulting outcome.
Provider without idempotency support	Reconciliation, compensation, or a redesigned integration; do not overclaim guarantees.

4. How expensive is the protection?

Effectively-once processing has real costs:

• a durable deduplication store and a unique index;
• transaction and storage overhead;
• retention and cleanup for processed-message records;
• monitoring duplicate rates and constraint violations;
• idempotency propagation through downstream calls.

Those costs are appropriate for shipping labels and financial effects. They may be disproportionate for disposable telemetry.

---

Worked decisions

Scenario A: Update a search projection after a product change

Requirement: Catalog publishes product name, price, availability, and a monotonically increasing version. Search maintains a local query-optimized representation. It is acceptable for the projection to lag briefly, but a product must not disappear because a consumer crashes.

Choice: At-least-once transport with effectively-once projection outcome.

The consumer processes before acknowledging. It upserts product_search_view by productId only if the incoming version is newer. Duplicate snapshots and older out-of-order events do not alter the final projection. An inbox table may be added if the handler has additional non-idempotent work.

Scenario B: Record browser interaction telemetry

Requirement: A dashboard needs rough trends for page hover activity. Occasional missing observations are acceptable; repeated delivery is not worth the persistence and retry overhead.

Choice: At-most-once.

The team should document that the metric is approximate and should not reuse this stream for billing, fraud detection, or compliance reporting later. The semantic is a business decision that constrains future use.

Scenario C: Create a shipping label

Requirement: Every paid order must eventually have a label. Creating two labels can cause duplicate pickup, duplicate cost, and warehouse confusion.

Choice: Effectively-once.

Use at-least-once redelivery, a unique processed-event marker in the Shipping database, and a transaction that records the marker with the local shipment state. If label creation requires an external carrier API, pass the same idempotency key to that API and persist the returned carrier label identifier.

Scenario D: Send a payment-receipt email

Requirement: A receipt must be sent, but customers should not receive duplicate emails after consumer restarts.

Choice: Effectively-once, but only if the outbound email boundary participates in idempotency.

Persist an email-dispatch intent and the source event identity transactionally. A dispatcher calls the email provider using that identity as its idempotency key, then records the provider result. If the provider cannot deduplicate, design an auditable reconciliation process rather than assuming a broker acknowledgement solves the problem.

---

Interview-ready answer structure

When presented with a requirement, answer in this order:

1. State the business consequence of loss.
   “A missing OrderPlaced event could leave an order unfulfilled, so at-most-once is not acceptable.”

2. State the business consequence of duplication.
   “A duplicate can create a second shipping label, so plain at-least-once processing is unsafe.”

3. Choose the semantic and describe its boundary.
   “I would use at-least-once delivery with effectively-once label creation in the Shipping boundary.”

4. Name the concrete mechanism.
   “The consumer inserts (consumer, source, eventId) into an inbox table with a unique constraint in the same database transaction as shipment creation, then acknowledges after commit.”

5. Address external effects honestly.
   “For the carrier call, I propagate the same idempotency key. If the carrier cannot honor it, I need reconciliation rather than claiming exactly once.”

That response shows reliability reasoning, database correctness, and operational realism—not just familiarity with Kafka terminology.

---

Key takeaways

• At-most-once accepts loss to avoid retry or redelivery overhead; use it only when missing work is genuinely acceptable.
• At-least-once prevents silent loss by retrying and redelivering, so duplicate processing must be considered normal.
• Effectively-once means one observable business result for a logical message, even though its delivery may be repeated.
• The core effectively-once pattern is a stable message identity, a unique deduplication constraint, and an atomic transaction containing both the processed marker and local business update.
• Natural idempotency—such as applying an absolute snapshot by key and version—can reduce the need for inbox bookkeeping.
• Kafka transactions can provide strong guarantees for Kafka-to-Kafka processing, but no broker alone can guarantee exactly-once effects in arbitrary databases or external APIs.

Next, you will narrow the same reasoning to ordering scope: determining which events must stay ordered, for which business key, and why global ordering is usually both unnecessary and expensive.