When Should an API Wait for Kafka?

An API receives an event. It validates the request, publishes a message to Kafka, and returns success. That sounds straightforward until you ask one uncomfortable question: at exactly what point has the API earned the right to say the event was accepted?

Before the broker receives it? After the producer sends it? After Kafka acknowledges it? After a consumer processes it? After the database commits the resulting state?

Those are different guarantees. Yet an HTTP response can easily flatten them into the same reassuring 2xx.

I ran into this decision while building FitLife, an event-driven personalization case study. User interactions such as class views are published to Kafka and later incorporated into recommendation generation.

The tempting implementation was simple:

request
  ↓
validate
  ↓
start Kafka publish
  ↓
return success

Fast. Asynchronous. Clean.

Also capable of lying.

The uncomfortable gap between “sent” and “accepted”

Suppose the API starts a Kafka publish and immediately returns success. Then the broker is unavailable, the connection fails, or the producer cannot complete delivery.

From the caller's perspective, the request succeeded. From the system's perspective, the event may never have crossed the broker-acknowledged ingestion boundary at all.

That distinction matters because HTTP success is part of the system's contract.

An asynchronous workflow does not eliminate acknowledgement semantics. It moves the boundary where you have to define them.

In the current FitLife implementation, the producer is configured with Acks.All, EnableIdempotence = true, and MaxInFlight = 5. The endpoint waits for ProduceAsync to complete before returning. That gives the response a narrower meaning than “the business operation finished”: the producer completed delivery under the configured broker acknowledgement semantics.²

client
  ↓
POST /api/events
  ↓
validate versioned event
  ↓
publish to Kafka
  ↓
wait for configured broker acknowledgement
  ↓
return 200 + event ID  ← current implementation
  ↓
consumer processing continues asynchronously

200 OK, 202 Accepted, and the contract underneath them

The current endpoint returns 200 OK after the Kafka publish completes. I originally reached for 202 Accepted because downstream processing is still asynchronous, but the HTTP semantics are more nuanced than “async means 202.”

RFC 9110 describes 202 Accepted as intentionally noncommittal: the request has been accepted for processing, but it might or might not eventually be acted upon. It also says the response representation ought to describe current status and point to a status monitor or estimate when possible.¹ FitLife does not currently expose an event-status resource, so switching the status code alone would not create a stronger end-to-end guarantee.

Those are still different events. The status code matters, but the more important engineering work is defining exactly what the response means and avoiding claims that the system has crossed a stronger boundary than it actually has.

A surprising amount of distributed-system confusion comes from using one word — success — for several different states.

Acknowledgement creates another problem: retries

Waiting for Kafka reduces one ambiguity, but distributed systems rarely let you remove ambiguity for free.

Imagine this sequence:

1. Client sends the event.
2. The API publishes it successfully.
3. Kafka acknowledges it.
4. The HTTP response is lost.
5. The client times out.
6. The client retries.

Did the original event succeed? Yes. Does the client know that? No.

If the retry creates a new logical event, downstream processing can now happen twice. That is why acknowledgement and retry design have to be considered together.

The endpoint accepts an optional caller-supplied EventId. If the caller supplies one, the API preserves it; otherwise the server generates a GUID. That distinction is crucial.

SQL-enforced deduplication makes broker redelivery of the same event ID safe. But the lost-response scenario above is only safe as a client retry if the caller generated and retained the ID before the first request and sends that same ID again. If the server generated the ID and the response was lost, the caller never learned it; a retry without that ID becomes a new logical event.

caller-owned EventId
        → ambiguous HTTP retry identity

producer idempotence
        → producer retry / ordering guarantees

SQL UNIQUE(EventId)
        → consumer redelivery deduplication

That is not magical exactly-once execution. It is a set of explicit duplicate boundaries.

At-least-once changes the question

Message processing often gets discussed as if duplicate delivery were itself an architectural failure. I think the more useful question is: what happens if the same logical event is observed more than once?

In FitLife, the durable concurrency boundary is a unique filtered SQL index on Interactions.EventId. The consumer does an early existence check, but it does not rely on that check for correctness: if concurrent deliveries race, the unique constraint is the final authority and duplicate-key violations are treated as a duplicate rather than a processing failure.

consume event
  ↓
attempt to persist EventId + interaction
  ↓
SQL UNIQUE(EventId)
  ├─ insert succeeds → new interaction
  └─ duplicate key   → treat as already processed
  ↓
continue with idempotent cache invalidation when required

Reliability often comes from making repetition safe, not from assuming repetition can be prevented.

Networks retry. Clients retry. Libraries retry. Operators replay. Consumers restart. A design that requires “this must only ever arrive once” is demanding a property that the surrounding system may not be able to guarantee.

The consumer needs a failure policy too

Successfully placing an event on Kafka does not guarantee that the consumer will successfully process it. A dependency can fail temporarily. A message can trigger a persistent processing error. A code defect can repeatedly crash the handler.

If the consumer retries forever, one poison event can stall progress. If it immediately skips every failure, temporary dependency failures can turn into loss.

The current consumer disables auto-commit, processes one record, and commits its source offset manually afterward. Known contract failures such as malformed JSON or an unsupported schema go directly to the dead-letter path. Other thrown processing exceptions are retried up to three times with a configurable linear delay (250 ms × attempt by default). The implementation does not currently classify thrown exceptions into “transient” and “fatal” categories, so “three attempts” is a pragmatic demo policy rather than a claim that it has perfectly identified transience.

After retries are exhausted, the consumer publishes a metadata-only record to user-events-dlq containing the source topic, partition, offset, message key, event ID, disposition, and attempt count. The original event payload is not included, so the DLQ is diagnostic evidence rather than a self-contained replay queue.

consume
  ↓
process
  ↓
success? ─────────────────────────────→ commit offset
  │
  no
  ↓
retry, bounded
  ↓
still failing?
  ↓
publish poison metadata to DLQ
  ↓
wait for DLQ acknowledgement
  ↓
commit source offset

The dead-letter path therefore has an acknowledgement boundary of its own. If DLQ publication fails, the processing call throws and the source offset is not committed.

There is still a residual failure window: DLQ publication and source-offset commit are two separate operations, not one Kafka transaction. A crash after the DLQ record is acknowledged but before the source offset commits can cause the source record to be delivered again and dead-lettered again. “Publish to DLQ, then commit” trades a loss window for a duplicate window; it does not make the pair atomic.

The cost: broker health becomes caller-visible

Waiting for Kafka acknowledgement has an obvious downside: the caller now experiences broker latency.

If Kafka slows down, the endpoint slows down. If Kafka cannot accept the event, the endpoint cannot pretend everything is fine.

That coupling is real. It would be misleading to describe the design as “decoupled” without acknowledging it. The downstream processing is asynchronous, but the ingestion endpoint still depends on the broker being available enough to accept work.

When I would not wait

There are cases where waiting for broker acknowledgement is the wrong trade.

Consider low-value UI analytics. A user clicks a cosmetic control. Losing one telemetry event is unfortunate but irrelevant to the user's primary task. Blocking that task because an analytics broker is unavailable would invert the system's priorities.

For non-critical tracking, I would rather let the primary action succeed and handle telemetry failure separately.

There are other architectures

Waiting for the broker directly is not the only way to build a reliable ingestion path. For a more complex production system, I would also consider a transactional outbox.

In that model, the API persists business state and an outgoing event together in one durable transaction. A separate publisher later transfers the event to Kafka.

request
  ↓
database transaction
  ├─ business state
  └─ outgoing event
  ↓
commit
  ↓
return

later:
outbox publisher → Kafka

Now Kafka availability does not necessarily sit directly on the request path. But the system has gained an outbox table, publisher lifecycle, lag, cleanup, monitoring, and another set of failure modes.

That can be an excellent trade. It is still a trade.

Infrastructure rarely deletes complexity. More often, it relocates complexity to a boundary you prefer.

The useful question is: what does success mean?

The implementation decision that started this was small: await the Kafka publish.

But the more useful lesson was about contracts. When an API returns success in front of an asynchronous system, I now want to ask:

• What exactly has succeeded?
• Which boundary has actually been crossed — socket write, broker acknowledgement, replicated log, or durable business state?
• What are the producer acknowledgement and replication settings?
• Can the caller safely retry after an ambiguous HTTP outcome?
• Who owns the logical event ID, and will it survive a lost response?
• What happens if the consumer sees the same event twice?
• What happens when processing repeatedly fails?
• Is dead-letter publication atomic with source-offset advancement?
• Which dependency failures should be visible to the caller?
• Which failures should be isolated from the user's primary action?

Those questions matter more than whether a particular code path contains await. They define what the system is actually promising.

Reliability is easier to reason about when every acknowledgement has a precise meaning.

In this case, waiting for Kafka means accepting more latency and availability coupling at the API boundary. In return, the API can make a narrower, testable statement: the publish completed under the configured Kafka acknowledgement semantics.

That statement is intentionally smaller than “the event is durable everywhere” or “the business operation completed.” The smaller promise is the point. It is one the implementation can actually defend.