Choosing Kafka for Integration Scenarios

Welcome to Kafka. Over the next two weeks, you will move from the architectural reasons for using Kafka to hands-on Java and Spring Boot development, then into production design and operations. This first module establishes the mental model needed for the lab work that follows.

This lesson answers a question that prevents many poor integrations: is Kafka actually the right tool for this interaction? You will distinguish three communication needs—queries, commands, and events—and use them to decide among direct request/response calls, traditional message queues, and Kafka event streams.

---

Start with the interaction, not the technology

Services communicate for different reasons. The technology should follow the meaning of the interaction.

There are three useful categories:

Interaction	What the sender means	Typical example	Usually needs an immediate answer?
Query	“Tell me something.”	“What is the current price of product 42?”	Yes
Command	“Do this work.”	“Generate this invoice.”	Usually no
Event	“This happened.”	“Order 718 was placed.”	No

A query asks for current information. A command directs a particular receiver to perform work. An event reports an immutable fact from the past: it should not contain an instruction disguised as a fact.

For example:

• GetCustomerCreditStatus is a query.
• CreateShipment is a command.
• ShipmentCreated is an event.

The difference sounds semantic, but it determines coupling, failure behavior, scaling, and whether the data can later serve new consumers such as analytics or monitoring.

---

Direct request/response: best when the answer is needed now

With direct request/response, one service calls another through HTTP/REST, gRPC, or an RPC framework. The caller waits for a response or timeout before it can finish its work.

This is often exactly right.

Suppose a Spring Boot checkout service must display the current product price before a customer confirms a purchase. It needs an answer at that moment. Sending an event to Kafka and waiting for another service to eventually react would make no sense: the customer is waiting and the UI needs a concrete result.

Direct requests are strongest when:

• the caller requires a current answer synchronously;
• there is a clear, bounded dependency on one service;
• the call volume and latency budget are reasonable;
• failure should be visible immediately to the caller.

However, direct calls create two forms of coupling:

1. Spatial coupling: the caller must know which service to contact and which API contract to use.
2. Temporal coupling: both services must be available at roughly the same time.

A single direct call can be entirely sensible. The danger appears when one request triggers a long chain: checkout calls orders, which calls inventory, which calls shipping, which calls promotions. The user-visible latency becomes dependent on every link, and an outage or slowdown in one downstream service can cause failures to propagate upward.

The goal is not “eliminate REST.” It is to reserve direct requests for genuine queries and immediate interactions, rather than using them as a default integration mechanism for every state change.

The cost of asynchronous decoupling

Kafka reduces temporal coupling, but it does not make distributed systems magically consistent. When an Order service publishes OrderPlaced, the Fulfillment service may process it milliseconds later, seconds later, or later still during a recovery.

That is eventual consistency: separate services converge on a shared business reality over time, rather than in one synchronous operation.

This is often appropriate for fulfillment, notifications, analytics, search indexing, and many background workflows. It is usually not appropriate when an immediate, authoritative answer is mandatory—for example, deciding whether a payment authorization succeeded before telling a customer the purchase is complete.

---

Traditional queues: best for directed, consumable work

A traditional message queue is designed primarily as a work buffer. A producer places a message on a queue; one consumer instance receives and handles that message. Once successfully handled, the message is normally acknowledged and removed.

Consider image processing after a user uploads a photo:

• A web application creates an instruction: GenerateThumbnail(photoId).
• A pool of image workers handles the work.
• Each job should be performed by one worker, not independently by analytics, billing, fraud detection, and five other systems.
• If workers are temporarily busy, the queue absorbs the backlog.

That is a natural queue-shaped problem. The message is a command addressed to the image-processing capability. The important question is “has this unit of work been completed?”

Queues provide valuable properties:

• asynchronous work distribution;
• buffering during bursts;
• competing consumers for parallel execution;
• acknowledgement and retry mechanisms;
• often sophisticated per-message routing, priorities, or delays.

But a queue is not usually the best canonical record of business history. If an event is consumed and deleted, a new consumer cannot normally start later and reconstruct what happened simply by reading the queue from the beginning.

It is also important not to turn this into an absolute rule. Kafka can be used for worker-style workloads, and some queue technologies support fan-out or retention. The useful distinction is about the dominant model:

• A queue treats a message chiefly as work to be completed once.
• Kafka treats a record chiefly as a retained fact that independently interested consumers may read at their own pace.

---

Kafka: a shared, retained stream of facts

Kafka is an event-streaming platform. A producer writes records to a topic, and Kafka retains those records according to a configured policy. Consumers read the topic and track their own position in it rather than causing records to disappear when they read them.

This makes one published business event reusable.

In this diagram, the frontend does not need to make separate calls to Hadoop, security, analytics, recommendations, and monitoring. It publishes the fact that a job view occurred. Each downstream system decides whether and how to react.

Kafka fits especially well when several of these conditions apply:

• A service emits a continuous sequence of domain events.
• Multiple independent consumers need the same events.
• Consumers may be added later without changing the producer.
• Events must remain available for a retention period.
• A consumer may need to replay history after a bug fix, outage, or logic change.
• High throughput or horizontal consumer scaling matters.
• The same operational data can support application behavior, monitoring, and analytics.

A useful example is an e-commerce system publishing OrderPlaced, OrderCancelled, and OrderShipped events. Potential consumers include:

• fulfillment, which creates and tracks shipments;
• inventory, which reserves or releases stock;
• notifications, which sends customer updates;
• fraud detection, which evaluates orders;
• analytics, which measures conversion and fulfillment behavior;
• a data platform, which builds reporting datasets.

Each is a different concern with an independent pace of change. Adding fraud detection should not require changing the Order service into a coordinator that knows about fraud detection.

Kafka achieves this through consumer groups. Each consumer group can read the topic independently. Within one group, members share the work; across different groups, each group can process the full relevant stream. We will examine partitions, offsets, and group assignment in detail in later lessons.

---

A practical decision framework

When evaluating an integration scenario, ask the questions in this order.

1. Is the sender asking for an answer now?

If yes, start with a direct request. A product page asking for an up-to-date price, a UI validating a coupon, or a service retrieving a customer’s current account status are query-shaped interactions.

Kafka can distribute data that helps a service construct a local view, but it is not a replacement for every synchronous query. Building and maintaining a local view introduces delay and operational responsibility.

2. Is the sender directing one capability to do work?

If yes, a traditional queue is often a strong default. The sender knows the intended work category, and one worker should handle each job.

Examples include:

• send one email;
• render one uploaded video;
• run one report;
• invoke a third-party API subject to rate limits.

Kafka may still be chosen if the command history must be retained and consumed by multiple independently evolving systems, but that is an additional requirement—not an automatic benefit.

3. Is the sender announcing a fact that many parties may care about?

If yes, Kafka is a strong candidate. The producer should publish the fact without knowing every future consumer.

This is the difference between:

• Command: ReserveInventory(orderId)
• Event: OrderPlaced(orderId, items, customerId)

The first identifies an intended action. The second makes business history available. Inventory may decide to reserve stock, while analytics counts the order and notifications prepare a confirmation—all from the same event.

4. Do we need replay, auditability, or a new consumer later?

If you need to reprocess last week’s events after fixing a defect, backfill a new search index, train an analytical model, or investigate a sequence of business actions, retained streams are valuable. This is one of Kafka’s defining advantages.

A queue can help work get done today. Kafka can help today’s work get done while preserving a source that other consumers can use tomorrow.

5. Can the business tolerate eventual consistency?

Kafka consumers process asynchronously. Design the user experience and business rules around that fact. For example, after an order is accepted, a UI may show “Order received; fulfillment is being prepared” rather than claiming that inventory, shipping, and notifications have all completed atomically.

If a workflow requires one immediate all-or-nothing outcome across services, Kafka alone is not the answer. You need to reconsider service boundaries, use a carefully designed synchronous operation, or model the workflow with explicit compensating actions.

---

Comparing the three choices

Dimension	Direct request/response	Traditional message queue	Kafka event stream
Primary meaning	Query or immediate operation	Command / unit of work	Domain event / retained fact
Sender knows receiver	Yes	Usually yes, at least by work destination	No need to know individual consumers
Timing	Synchronous	Asynchronous	Asynchronous
Consumer model	Caller gets one response	One consumer handles a queued message	Multiple consumer groups can independently read events
Data after processing	Usually stored or overwritten in service databases	Often removed after acknowledgement	Retained according to topic policy
Replay	Requires separate history or reconstruction	Usually limited	Core capability
Typical strengths	Simplicity, immediate answers	Job distribution and buffering	Fan-out, replay, high-volume streaming, data reuse
Main risk	Availability and latency chains	Treating commands as durable shared history	Added operational and consistency complexity

The table is a decision aid, not a prohibition. Real architectures commonly use all three patterns. A checkout service might synchronously query pricing, enqueue a slow third-party document-generation task, and publish OrderPlaced to Kafka for downstream business reactions.

---

Avoid “Kafka everywhere”

Kafka is powerful because it makes event history broadly available, not because it is a universal transport layer. Choosing it has real costs:

• You must define stable event contracts and evolve them carefully.
• Consumers can lag, fail, or process records more than once, so processing must be resilient.
• Services become eventually consistent.
• Kafka clusters require monitoring, security, capacity planning, and operational discipline.
• A simple one-off request can become unnecessarily indirect if placed on a stream.

Use Kafka where those costs buy meaningful decoupling, durable history, replay, or broad reuse. Do not use it merely because two services need to exchange a small piece of information.

A concise rule of thumb is:

Use direct calls to ask for information now, queues to assign work, and Kafka to publish facts that should remain independently consumable over time.

---

Takeaways

You now have the architectural lens for the rest of the course:

• Direct request/response is appropriate for immediate queries and tightly bounded synchronous operations, but it couples availability and latency across services.
• Traditional queues are well suited to asynchronous, point-to-point commands and worker-pool task distribution.
• Kafka is appropriate for durable, replayable event streams that multiple independent consumers can process at their own pace.
• Kafka’s decoupling and replayability come with eventual consistency and operational responsibilities.
• A realistic system often combines all three patterns rather than choosing one exclusively.

Next, we will move from architecture to Kafka’s mechanics: tracing a single record from a producer into a topic partition and through a consumer group.