Kafka Fundamentals Primer

You do not need deep Kafka knowledge to follow the main article. But a handful of terms make its central distinction much easier to reason about:

socket write
≠
broker acknowledgement
≠
replicated log
≠
consumer processing
≠
committed business state

Those are five different things. An HTTP 2xx can flatten all of them into one word. This primer gives names to the boundaries between them.

1. Kafka is an append-only log

Kafka is easiest to reason about as a distributed, append-only log. Producers append records; consumers independently track how far they have read. Reading a record does not remove it.

The vocabulary:

• Record — the unit Kafka stores: key, value, headers, timestamp.
• Topic — a named stream of records, such as user-events.
• Partition — a topic is divided into partitions, each an independent append-only sequence.
• Offset — a record's position within its partition. Reading is "give me everything from offset N onward."
• Broker — a Kafka server holding partitions. A cluster is several brokers.

A simplified partition:

user-events, partition 0

offset     41        42        43        44
           ↓         ↓         ↓         ↓
        [view]    [click]   [signup]   [view]

After successfully processing offset 43, a consumer can commit offset 44 — effectively saying, "resume with 44 if I restart." Committed offsets are the next offset to consume, not the last one completed.

Two consequences matter for the article.

Ordering is per-partition, not per-topic. Kafka guarantees order within a partition. It does not inherently tell you whether partition 0 / offset 12 happened before or after partition 2 / offset 97.

Replay is normal. Records are not deleted because they were consumed; Kafka retains and cleans them according to the topic's configured policy. In an at-least-once design, consumers must be prepared to see the same record again: consumers restart, rebalances happen, and operators deliberately replay data. This is why the article treats duplicate handling as a design requirement rather than a defect to be engineered away.

Record, event, and message

These words get used interchangeably. Kafka documentation generally says record; application code often says event or message.

The distinction that matters is between the Kafka record and the logical event it represents. Two records can represent the same logical event. That gap is where retries and deduplication live.

2. How records get into Kafka

A producer is the client that writes records to Kafka.

HTTP endpoint
    ↓
Kafka producer
    ↓
user-events topic

Calling the producer does not by itself mean the record has reached Kafka. There is a period during which the producer has begun sending but does not yet know whether Kafka accepted the record. The acknowledgement configuration determines when the producer considers the send complete. That gap is exactly what the main article is about.

Message key

Records can carry a key. Producers normally use the record key to choose a partition, allowing records with the same key to be routed consistently to the same partition. That preserves partition-local ordering for related events:

key = user-123

ClassViewed
ClassBooked
ClassCancelled

A key is not the same thing as the article's EventId. A key is about partitioning and ordering. An EventId identifies a particular logical event for deduplication.

Replication, leaders, and followers

Kafka can keep multiple copies of each partition.

Partition 4

Broker A — leader
Broker B — follower
Broker C — follower

The leader handles writes for the partition; followers replicate the leader's log. If the leader fails, a sufficiently up-to-date replica can take over. Replication factor is how many replicas are configured.

ISR: in-sync replicas

Kafka tracks the in-sync replica set (ISR): the replicas currently considered caught up with the leader. This set shrinks and grows as brokers lag or recover.

acks, and what acks=all actually says

Producers control how much acknowledgement they require before a send completes.

acks=0 — send and continue. The producer does not learn whether delivery failed.

acks=1 — the leader writes the record and acknowledges. There is a window in which the leader has acknowledged a record that no other replica holds yet.

acks=all — the leader waits for the required acknowledgements from the partition's current in-sync replicas. This is Kafka's strongest producer acknowledgement setting.

The relationship worth memorizing:

replication.factor
    = how many replicas are configured

ISR
    = which replicas are currently in sync

acks=all
    = all current ISR members must acknowledge

min.insync.replicas
    = minimum ISR size Kafka requires before allowing
      an acks=all write to succeed

min.insync.replicas sets the durability floor for a successful acks=all write. If the ISR holds three members, acks=all waits for all three even when the minimum is two. If the ISR falls below the minimum, the write fails rather than succeeding against a degraded set.

This is why topology matters. With three appropriately configured replicas, an acknowledgement can represent replicated storage. With a single replica, acks=all provides no replication advantage over waiting for that one leader, because no additional copy exists. acks=all is an acknowledgement policy, not a declaration of universal durability — which is precisely the caveat the article attaches to its single-broker local topology.

And even a fully replicated acknowledgement says nothing about:

consumer ran
database committed
cache updated
business workflow completed

3. Producer retries and idempotence

Distributed systems retry. Suppose a producer sends a batch, Kafka stores it, and the acknowledgement is lost. The producer cannot distinguish "Kafka never stored it" from "Kafka stored it and I never heard back." A reasonable producer retries, and without protection that retry creates a duplicate record.

enable.idempotence addresses this class of duplicate. The idempotent producer tags record batches with a producer ID and per-partition sequence numbers; the broker uses that identity and sequence information to recognize retried writes it has already stored. Kafka requires compatible settings, including acks=all and max.in.flight.requests.per.connection no greater than 5.

max.in.flight.requests.per.connection is how many unacknowledged batches can be in transit at once. Without idempotence, multiple in-flight requests plus retries can reorder records. With idempotence enabled and supported max.in.flight settings, Kafka preserves producer ordering.

The scope is the important part. Producer idempotence recognizes a producer retry. It knows nothing about a client that lost an HTTP response and issued a new request. That is why the article separates three distinct duplicate boundaries:

Kafka producer idempotence
→ recognizes a producer retry

caller-owned EventId
→ recognizes an ambiguous HTTP retry

SQL UNIQUE(EventId)
→ recognizes a consumer-side duplicate effect

Three mechanisms, three different problems. None of them substitutes for another.

4. How consumers make progress

A consumer reads records and performs work: update SQL, invalidate a cache, send an email.

Consumer group — consumers sharing a group.id. Kafka distributes partitions among the members:

Topic:
P0  P1  P2  P3

Consumer group:
C1 → P0, P1
C2 → P2, P3

Within a consumer group, a partition is assigned to one consumer at a time. That scales processing while preserving partition ordering.

Rebalance — reassignment of partitions when a member joins, leaves, or is presumed dead. Work done but not committed before a rebalance can be redone by the new owner.

Committed offset — how a group records progress. Note that a producer acknowledgement says something about getting a record into Kafka, while a committed offset says something about how far a consumer group claims to have progressed through Kafka. Neither says anything about an external SQL transaction.

The ordering of processing and committing decides your delivery semantics:

• Commit after processing → at-least-once. Crash in between and you reprocess.
• Commit before processing → at-most-once. Crash in between and you lose the record.
• Automatic offset behavior → with the default settings in the Confluent .NET client, an offset can become eligible for commit before application processing completes, creating a loss window on failure. Explicit commit after processing is what gives you at-least-once.

An ordinary Kafka offset commit does not make an external side effect, such as a separate SQL transaction, atomic with that commit. Without another coordination pattern, you therefore choose which failure window you prefer.

This also explains a subtlety in the article: its retry limit is bounded within one processing invocation. A rebalance or restart before the source offset commits can hand the uncommitted record to another consumer, which will process from the last committed offset again. The durable safety property is idempotent state handling, not a global cap on executions.

5. Delivery semantics

At-most-once — processed zero or one times. No duplicates, possible loss.

At-least-once — processed one or more times. No loss, possible duplicates.

Exactly-once — meaningful only when you specify the boundary. Kafka can provide atomic consume-transform-produce behavior within Kafka using transactions, but that does not automatically make arbitrary side effects in an external SQL database exactly once. Kafka's own design documentation treats publishing guarantees and consuming/output guarantees separately, and that separation is the whole point.

For most systems the practical design is:

at-least-once delivery
+
idempotent processing
+
deduplication

The record may arrive more than once; its business effect stays safe.

Idempotence

An operation is idempotent when repeating it produces the same effective result as performing it once.

SET account.status = "active"     -- naturally idempotent
account.balance += 100            -- not, without identity

The common technique in event processing is to give the event an identity and let the database enforce it:

UNIQUE(EventId)

If the same logical event arrives twice, only one insert can win. That is the pattern at the article's SQL boundary, and the reason a duplicate-key violation there is treated as a duplicate rather than a failure.

Deduplication needs identity

EventId = A
EventId = A     → recognizable duplicate

EventId = A
EventId = B     → two logical events, as far as the system knows

Which is why who owns the ID, and when is a design decision rather than an implementation detail. An idempotency key known to and retained by the caller before the first attempt is what makes an ambiguous HTTP retry recognizable as the same operation. In FitLife, that means a caller-supplied EventId. An ID minted by the server only during the request cannot do that job if the response carrying it is lost, because the caller never learned it.

6. Failure handling

Poison message — a record that repeatedly fails processing: malformed data, unsupported schema, an invariant violation, an application bug. Retrying a temporary database outage makes sense. Retrying malformed JSON forever does not. A consumer needs a failure policy, not merely a retry loop.

Head-of-line blocking — in a sequential partition-processing loop, repeatedly retrying one poison record blocks progress on later records in that partition. This is the concrete reason poison-message handling exists.

Dead-letter queue (DLQ) — a destination for records that cannot be processed after the chosen failure policy runs out.

consume
↓
process
↓
fails
↓
retry, bounded
↓
still fails
↓
publish to DLQ
↓
advance source offset

A DLQ stops one permanently failing record from blocking a processing path indefinitely. But it is not automatically a replay mechanism. What it contains matters: a DLQ carrying full payloads can serve as a self-contained replay source; a metadata-only DLQ records where the failure occurred but depends on the original source record still being retained if the payload later needs to be recovered. The article's implementation is the second kind, deliberately.

7. Distributed-system failure shapes

The Kafka terminology explains mechanics. These concepts explain why the article's questions are hard.

Partial failure

In an in-process function call, success and failure can look binary. Across a network they do not.

API          Kafka
 |             |
 |---send----->|
 |             | stores record
 |<---ack------|
 X connection dies

Kafka succeeded. The API may not know. The client knows even less. Different components hold different knowledge about the same operation, and there is usually a third outcome alongside success and failure:

UNKNOWN

That state drives retry design.

Ambiguous outcome

The specific case where an operation succeeded but the caller cannot tell. A lost HTTP response is the canonical example: Kafka has the event, the client sees a timeout. Retrying is sensible, and the system needs a way to recognize that the retry represents the same logical operation. Hence caller-owned event IDs.

Non-atomic pair

Two operations that must both happen, with a crash window between them.

do A
then do B

Whenever you see that shape, ask: what happens if the process dies after A but before B?

The article's DLQ path is exactly this. Publishing the DLQ record and committing the source offset are two operations, not one. A crash between them causes the source record to be redelivered and dead-lettered again. Ordering them "publish, then commit" trades a loss window for a duplicate window; it does not make the pair atomic.

Dual write

The same shape applied to two different systems with no shared transaction — a database write and a Kafka publish. Whichever you do first, a crash in between leaves the two disagreeing. The article's endpoint sidesteps this by only writing to Kafka on the request path.

Critical path and coupling

The critical path is the chain of work that must finish before the caller gets a response.

validate
↓
wait for Kafka
↓
respond

Kafka is on the critical path here. Its latency becomes request latency; its availability becomes the endpoint's ability to report success. Awaiting ProduceAsync puts completion of the Kafka publish on the HTTP response's critical path.

Note that asynchronous and decoupled are not synonyms. Downstream processing can be fully asynchronous while the ingestion endpoint remains operationally coupled to the broker during publishing. Asynchronous work does not remove dependencies; it changes where they sit.

8. Architectural tools

Kafka transactions

Kafka can bind consuming, producing, and committing offsets into a single transaction, with consumers reading in read_committed mode. This is the documented basis for its exactly-once processing semantics within Kafka.

That is relevant to the article's DLQ window, because publishing the DLQ record and committing the source offset are both Kafka operations. When the source and DLQ are in the same Kafka cluster, Kafka transactions can close this particular Kafka-to-Kafka atomicity gap. The implementation described in the article does not use it, which is why its crash window is real rather than hypothetical.

Transactional outbox

An alternative for the database-to-Kafka boundary. Instead of requiring Kafka to succeed during the request:

request
  ↓
SQL transaction
  ├─ business change
  └─ outbox event
  ↓
commit
  ↓
response

later:
outbox publisher → Kafka

The business change and the intent to publish are stored together in one local transaction, so Kafka can be briefly unavailable without blocking the request. The trade is that the system now owns outbox publishing, retry behavior, lag, cleanup, monitoring, and its own duplicate handling. The complexity is relocated to a boundary you prefer rather than removed.

9. The mental model to keep while reading

Rather than asking "did the event succeed?", ask:

1. Did the client reach the API?
2. Did the producer attempt the send?
3. Did Kafka acknowledge the record?
4. How many replicas does that acknowledgement represent?
5. Did the consumer receive the record?
6. Did the consumer's business operation succeed?
7. Did its external state commit?
8. Did the consumer commit its Kafka offset?
9. If any acknowledgement was lost, can the operation safely repeat?

Those are nine different questions. The status code alone does not tell you which boundaries were crossed; the API contract has to define that.

"success" is not one universal state

It is a statement made at a particular boundary, under a particular set of guarantees. Reliability design starts by naming that boundary precisely.

References

• Apache Kafka — Design (delivery semantics, idempotent producer, transactions)
• Apache Kafka — Producer Configs (acks, enable.idempotence, max.in.flight.requests.per.connection)
• Apache Kafka — Topic Configs and Broker Configs (min.insync.replicas at the topic and broker level, replication factor)
• Confluent — .NET client overview and IConsumer API (offset commit behavior; Commit(result) commits result.Offset + 1)