Idempotency Keys and Deduplication for At-Least-Once Consumers

Hello. In the last lesson, you defined where ordering matters: for example, order-lifecycle events must be applied in sequence per orderId, while unrelated orders can proceed independently. That protects the sequence of meaningful state transitions, but it does not prevent the same delivered event from being processed twice.

This lesson addresses the complementary reliability requirement for at-least-once delivery: design a stable idempotency key and a durable deduplication strategy so a consumer can receive a message repeatedly while producing the same business result as one successful processing attempt. The design applies equally to Kafka consumers, RabbitMQ consumers, and later CDC consumers.

---

Idempotency is a consumer property

At-least-once delivery deliberately favors duplication over loss. A broker cannot safely assume that processing succeeded merely because it sent a message. A consumer might update its database and then crash before committing its Kafka offset or acknowledging its RabbitMQ delivery. On restart, the broker redelivers the message.

The dangerous interval is therefore:

1. The consumer performs a durable business update.
2. The consumer fails before the broker records the acknowledgement.
3. The same message is delivered again.

For a payment consumer, an unprotected duplicate can create a second debit. For fulfilment, it can create a second shipping label. For an email consumer, it can send the same customer email again.

CodeOpinion’s short explanation is useful for visualizing the acknowledgement failure window and the resulting redelivery.

The target is not literally “run the Java method once.” Failures make that impossible to promise. The target is:

Reprocessing the same logical message must not create an additional business effect.

For a consumer function $f$, the intended property is:

$$f(f(x))=f(x)$$

That definition needs care. Setting an order’s status to CANCELLED repeatedly may be harmless. But setting the status and publishing a cancellation email every time is not idempotent unless the email action is also protected.

---

Choose an identity for the logical event

The first design decision is the idempotency key: the value that means, “this is the same logical operation or event as before.”

For an event-driven consumer, the default answer is a producer-assigned immutable event ID:

{
  "id": "018f30b5-4df1-7baf-9bdb-9f8b89f7b91d",
  "type": "com.acme.orders.order-placed.v1",
  "source": "urn:acme:orders",
  "subject": "order/O-104",
  "correlationId": "checkout-9a31",
  "data": {
    "orderId": "O-104",
    "version": 1,
    "totalAmount": 249.99
  }
}

Here, id is the event identity. It must be generated once when the producer creates the logical event and must survive:

• producer retries;
• broker redelivery;
• relay or outbox retries;
• dead-letter replay;
• forwarding through another service;
• a consumer restart.

Do not generate a fresh event ID each time a publisher retries. That makes the same logical event appear new to every downstream deduplication system.

Do not confuse the IDs

A production event often carries several identifiers. They serve different purposes.

Identifier	Meaning	Suitable as a deduplication key?
eventId / message ID	Identity of one immutable published fact	Usually yes
correlationId	Connects work across a request or workflow	No; many events can share it
causationId	Identifies the event or command that caused this event	No; one cause can produce several valid events
orderId	Identity of the business entity	Usually no; an order has many valid events
Kafka topic-partition-offset	Position in one Kafka log	No; tied to one transport and changes on replay or republishing
RabbitMQ delivery tag	Identity of one delivery attempt on a channel	No; changes on redelivery
payload hash	Fingerprint of content	Rarely; identical payloads can represent distinct valid business actions

A useful interview correction is:

A Kafka key such as orderId is usually an ordering and partitioning key, not an idempotency key. An order can validly have OrderPlaced, OrderPaid, OrderCancelled, and OrderRefunded events.

Event identity versus business-operation identity

Sometimes the business action has its own stronger uniqueness rule. Consider shipping-label creation:

• The OrderPlaced event has an eventId.
• The shipping label represents one business operation: “create the initial label for order O-104.”

If the producer accidentally emits two distinct OrderPlaced events with different event IDs for the same order, event-ID deduplication alone will not catch that producer defect. The Shipping database should also enforce its business invariant:

One active initial shipping label per order

This is usually represented as a unique business constraint, such as a unique index on shipping_label.order_id for the relevant label type.

Use both layers where the cost of duplication is material:

1. Message deduplication prevents the same event ID from being handled twice.
2. Business uniqueness constraints prevent two different messages from creating an invalid duplicate business object.

---

Scope deduplication to the logical subscriber

One event may be consumed legitimately by several services:

• Shipping creates a label.
• Analytics updates a metric.
• Notifications sends a customer email.
• Fraud Detection evaluates risk.

They must not share one global “processed” record. Shipping processing an event must not cause Notifications to skip it.

Instead, deduplication belongs to the logical consumer or handler:

deduplication key = subscriberId + messageId

Examples of stable subscriber IDs:

shipping.create-label.v1
billing.capture-payment.v1
warehouse.reserve-stock.v1

A subscriber ID is not a pod name, container ID, or Kafka consumer instance ID. All instances of shipping.create-label.v1 represent the same logical subscriber and must share the same deduplication state.

The Microservices.io pattern gives the foundational database design: a processed-message record is written in the same transaction as the application’s state change.

---

The durable deduplication record

For a typical Spring Boot service with PostgreSQL, a dedicated table is explicit, auditable, and applicable across many handlers.

create table processed_message (
    subscriber_id varchar(150) not null,
    message_id uuid not null,
    processed_at timestamptz not null default current_timestamp,
    primary key (subscriber_id, message_id)
);

The primary key is the critical part. It creates an atomic race arbiter in PostgreSQL: only one transaction can successfully claim a particular (subscriber_id, message_id) pair.

A useful implementation detail is to avoid this unsafe pattern:

1. Query whether a processed-message row exists.
2. If it does not exist, perform the business work.
3. Insert the processed-message row.

Two consumers can perform the query before either has inserted anything. Both then decide the message is new and both can perform an irreversible side effect.

Instead, claim the message first through a database write that enforces uniqueness.

PostgreSQL provides a clean way to do that:

insert into processed_message (subscriber_id, message_id)
values (:subscriberId, :messageId)
on conflict do nothing;

The returned update count has a direct meaning:

Result	Interpretation	Consumer action
1 row inserted	This transaction claimed a new message	Perform local business updates
0 rows inserted	Another successful transaction already claimed it	Skip business work and acknowledge it as a duplicate

For concurrent attempts with the same key, PostgreSQL resolves the conflict safely. The second attempt may wait while the first transaction completes. If the first transaction rolls back, another attempt can claim and process the message. If it commits, the other attempt observes that it is a duplicate.

---

Put claiming and business state in one transaction

The central rule is:

The deduplication record and the consumer’s local database changes must commit or roll back together.

A compact Spring design can use JdbcTemplate for the atomic claim and JPA or JDBC for business updates. The key point is the transaction boundary, not the persistence library.

@Service
public class ShippingService {

    private static final String SUBSCRIBER = "shipping.create-label.v1";

    private final JdbcTemplate jdbcTemplate;
    private final ShippingLabelRepository shippingLabelRepository;

    @Transactional
    public ProcessingResult handle(OrderPlaced event) {
        int claimed = jdbcTemplate.update("""
            insert into processed_message (subscriber_id, message_id)
            values (?, ?)
            on conflict do nothing
            """,
            SUBSCRIBER,
            event.id()
        );

        if (claimed == 0) {
            return ProcessingResult.DUPLICATE;
        }

        shippingLabelRepository.save(
            ShippingLabel.createInitialLabel(event.orderId())
        );

        return ProcessingResult.PROCESSED;
    }
}

The message listener calls handle() and acknowledges the broker delivery only after this method has returned successfully. The same idea applies to a manually acknowledged Kafka listener and a RabbitMQ consumer using manual acknowledgements.

The resulting failure behavior is deliberate:

Failure point	Database result	Broker result	Safe outcome after redelivery
Before the transaction commits	Claim and business updates roll back	No acknowledgement	A later attempt can claim and process it
After the transaction commits but before acknowledgement	Claim and business updates persist	No acknowledgement	Redelivery finds the processed record and skips work
Duplicate arrives after an earlier success	Processed record already exists	It still needs acknowledgement	Consumer skips work and acknowledges duplicate
Business update fails	Entire transaction rolls back	No acknowledgement	Retry can attempt processing again

This is effectively-once processing of local database state under at-least-once delivery. The broker may deliver several times; the consumer’s committed local result appears once.

A Spring JPA caution: flush before an irreversible action

With Hibernate, calling save() does not necessarily issue SQL immediately. Hibernate can defer the insert until transaction commit. In a concurrent duplicate scenario, that delay can allow both threads to reach an external side effect before one eventually loses the unique-key race.

The Lydtech Consulting article explains this timing risk and why forcing the deduplication insert to reach the database before a non-transactional side effect narrows it.

If you use JPA rather than a direct INSERT ... ON CONFLICT, saveAndFlush() can force the insert before later work. However, handling a DataIntegrityViolationException inside an already-failed transactional context requires care: in many configurations the transaction is marked rollback-only. A direct PostgreSQL conflict-safe insert is often simpler for a straightforward claim-or-skip design.

---

A database transaction cannot make an external side effect disappear

The strategy above protects changes in one database transaction. It does not magically make a remote HTTP call, an email, or a payment-provider request atomic with your local database.

Consider this tempting design:

1. Claim the event in the database transaction.
2. Call a payment provider.
3. Commit the local transaction.

If the payment provider succeeds but the service crashes before commit, the local claim rolls back. On redelivery, the service calls the provider again. Conversely, committing before the provider call risks recording a payment that was never sent.

For an external payment or irreversible API call, design another idempotency boundary:

• Give the external provider a stable operation idempotency key, such as paymentAttemptId.
• Persist a payment-attempt or command-intent record with a unique business constraint.
• Have the provider return the same result for repeated requests carrying that key.
• Record and reconcile uncertain outcomes rather than assuming a timeout means failure.

For example, a payment provider may receive:

Idempotency-Key: payment-attempt-7c4d...

The key represents the business operation “capture this particular payment,” not merely the Kafka delivery.

Similarly, if a consumer changes local data and must publish a new outbound event, do not treat a successful database commit plus a later broker publish as atomic. The transactional outbox pattern addresses that separate boundary; you will build it concretely in the CDC modules. For now, remember the design principle:

Deduplicate the incoming message and make local state changes atomically; use a separate reliable publication mechanism for outbound effects.

---

Decide whether a dedicated table is necessary

Not every handler needs a processed_message table. A pure state assignment can be naturally idempotent:

Set order status to CANCELLED

Repeated application may lead to the same final state. Even then, examine the full handler:

• Does it update a timestamp on each duplicate?
• Does it emit another event?
• Does it call another service?
• Can an older event overwrite a newer state?
• Does it violate a business invariant under concurrency?

In the ordering lesson, you saw version-aware updates. A projection update can combine version handling with idempotency:

update order_projection
set status = :incomingStatus,
    version = :incomingVersion
where order_id = :orderId
  and version < :incomingVersion;

That protects against stale state updates, but it answers a different question from message deduplication:

• Deduplication: “Have I handled event E-123?”
• Versioning: “Is this state change newer than what I have?”
• Business uniqueness: “May a shipping label already exist for this order?”

Robust consumers often use more than one of these controls.

---

Retention, replay, and operations

A processed-message table is state. If every consumer processes one million messages per day, storing all keys forever has an operational cost.

Choose retention based on the meaning of replay:

Situation	Suitable retention approach
Normal broker redelivery and short retry windows	Retain deduplication records longer than the maximum redelivery and retry horizon
Dead-letter replay may occur months later	Retain records through that operational replay window
A consumer can rebuild a projection from the beginning	Use a new subscriber identity or rebuild into separate projection storage
Financial or irreversible effects	Retain the business-operation uniqueness record for the required audit and legal period
High-volume, low-risk telemetry	Use a bounded retention policy if delayed replay is explicitly acceptable

Deleting records after 30 days is not inherently right or wrong. It means: “If this event is replayed after 30 days, we permit it to be treated as new.” That may be acceptable for a disposable analytics projection and unacceptable for a payment capture.

At minimum, monitor:

• duplicate messages skipped, grouped by topic or queue and consumer;
• unique-constraint or claim-conflict rates;
• processing failures and retry counts;
• growth and retention of processed_message;
• the age of the oldest unprocessed or dead-lettered message;
• mismatches where the same message ID arrives with conflicting event type, source, or payload.

A repeated ID with materially different contents is not a normal duplicate. It suggests a producer bug, identity reuse, or message corruption. Quarantine and investigate it rather than silently choosing one payload.

---

An interview-ready design answer

For a consumer that creates shipping labels from OrderPlaced events, a concise technical-lead answer could be:

“The broker provides at-least-once delivery, so I assume an event can be redelivered after its database work succeeds but before the acknowledgement is persisted. The producer assigns an immutable event ID once per logical event and preserves it across retries. Shipping uses a durable processed_message table keyed by subscriberId and eventId, so all Shipping instances share deduplication state while other services still process the event independently. In one PostgreSQL transaction, the handler atomically claims the event with a unique insert and writes the shipping-label state. If the claim already exists, it skips the work and acknowledges the duplicate. I also enforce a business uniqueness constraint on the initial label per order, because distinct duplicate events could still represent an invalid duplicate operation. For external calls, I propagate a business operation idempotency key to the provider; a local database transaction alone cannot guarantee exactly-once external side effects.”

---

Key takeaways

• At-least-once delivery means a consumer must expect redelivery after work may already have succeeded.
• Use a stable producer-assigned event ID as the default idempotency key; do not use correlation IDs, delivery tags, Kafka offsets, or an entity ID by default.
• Scope deduplication by logical subscriber: (subscriberId, messageId).
• Claim the message with a database-enforced unique insert before performing business work; do not rely on a preliminary lookup.
• Commit the processed-message record and local business updates in the same transaction.
• A committed transaction followed by failed acknowledgement is safe: redelivery becomes a detected duplicate.
• Message-level deduplication, version checks, and business uniqueness constraints solve different problems and often belong together.
• Database idempotency does not make external API calls exactly once; external operations need their own stable idempotency key and recovery strategy.
• Deduplication retention is a replay policy, not merely a table-cleanup task.

Next, you will step back from individual mechanisms and assess the broader trade-offs in an event-driven design: consistency, latency, availability, and operability.