Your database and Kafka are lying to each other

Your database and your message broker have trust issues, and you probably haven’t noticed because they’re both being polite about it. Here’s what’s happening behind your back.

Your code tells the database, “save this payment.” The database says, “done.” Your code tells Kafka, “publish this event.” Kafka says, “sure.” You walk away thinking everything is fine, except sometimes Kafka is lying, or the database is, or both. And you won’t find out until a customer asks why their transaction vanished.

This is the dual-write problem. If you write to a database and a message queue in the same operation, you have it. You just might not know yet.

In Nigerian fintech this isn’t academic. Picture an outward NIP transfer: you debit the customer’s wallet in your core banking DB and publish an outflow.completed event so reconciliation, notifications, and the ledger service all react. If the debit commits but the event never reaches Kafka, you’ve moved money that the rest of your system doesn’t know about. That’s the bug that shows up as a “missing” transaction three days later in a settlement report.

What’s actually going wrong

Here’s typical code. Looks innocent enough:

@Transactional
public void processOutflow(OutflowRequest request) {
    outflowRepository.save(outflow);
    kafkaTemplate.send("outflow-completed", event);
}

Two things happen, and they’re supposed to either both succeed or both fail. They won’t. The @Transactional annotation governs the database connection only. Kafka isn’t enrolled in it. There’s no shared transaction log between Postgres and a Kafka broker, so the database can commit while the Kafka send is still in flight, or vice versa.

Two ways this bites:

The database commits, then the network hiccups, and Kafka never gets the message. The payment exists; downstream services have no idea.

Or Kafka accepts the message, then the database transaction rolls back. Now you’re broadcasting events about a payment that doesn’t exist.

There’s no annotation that makes these two coordinate. Distributed transactions over XA / two-phase commit technically exist, but Kafka doesn’t support them in any form you’d want to run in production, and they drag in their own coordinator failures and lock contention. So pretending a database and a broker can commit atomically together is wishful thinking. You need a different shape.

Enter the outbox

The outbox pattern solves this by refusing to make the two systems trust each other. You funnel everything through the one system that gives you real, battle-tested transactions, your database, and relay outward from there.

The flow:

1. Write your business data to the database.
2. In the same transaction, write the event to an outbox table in the same database.
3. A background process reads pending rows from the outbox and delivers them to Kafka.
4. Only after Kafka confirms delivery does the row get marked delivered.

Steps 1 and 2 are a single database transaction, so they genuinely succeed or fail together. That’s the whole trick. If Kafka is down, events pile up in the outbox and go out when it recovers. Nothing is lost in the gap.

A note on flavours: what I’m describing is the polling publisher, a job that queries the outbox on a schedule. There’s also a log-based variant where something like Debezium tails the database’s write-ahead log and publishes changes with no polling at all. CDC scales better and adds zero query load, but it’s another moving piece to run and reason about. For most teams the polling approach is easier to operate and debug, so that’s what I’ll build here.

Here’s the table:

@Entity
@Table(name = "outflow_transaction_outbox")
public class OutboxEvent {

    private String aggregateType;   // what kind of event
    private String aggregateId;     // reference ID (e.g. session/transaction ref)
    private String eventType;       // what to do with it
    private String payload;         // the serialized event

    private OutboxStatus status;    // PENDING, PROCESSING, PROCESSED, FAILED

    private int retryCount;
    private int maxRetries;
    private LocalDateTime scheduledAt;
    private String failureReason;
}

Status does the heavy lifting. PENDING is waiting to be sent. PROCESSING means a worker has it. PROCESSED means Kafka confirmed. FAILED means we gave up and a human should look.

One promise to be clear-eyed about up front: this gives you at-least-once delivery, not exactly-once. A worker can deliver to Kafka and then die before it marks the row PROCESSED; on the next sweep the row still looks unsent and goes out again. That’s unavoidable without distributed transactions, and it’s fine, provided your consumers are idempotent. Dedupe on aggregateId plus eventType, or carry an idempotency key. Everything below assumes you’ve done that. If you take one thing from this article, take that: outbox makes delivery reliable, idempotent consumers make it correct.

Now the interesting part: what breaks when you actually run this.

Multiple workers fighting over the same row

You deploy three instances for redundancy. Each runs a scheduled job hitting the outbox every 30 seconds. All three notice the same pending row at the same time. All three process it. All three send to Kafka. The customer gets three debit alerts, your reconciliation counts the event thrice, and accounting opens a ticket.

The fix is row-level locking with skip logic, a database feature built for exactly this.

SELECT * FROM outflow_transaction_outbox
WHERE status = 'PENDING'
  AND (scheduled_at IS NULL OR scheduled_at <= NOW())
ORDER BY created_at ASC
LIMIT 50
FOR UPDATE SKIP LOCKED;

Instance A grabs and locks rows 1-50. Instance B runs the identical query, sees those rows are locked, skips straight past them, and takes 51-100. SKIP LOCKED is the load-bearing part: without it, Instance B blocks waiting for A to finish instead of moving on.

The claim check, and why it’s not just belt-and-braces

Before processing, each worker tries to “claim” the row with a conditional update:

@Query("UPDATE outflow_transaction_outbox SET status = 'PROCESSING' " +
       "WHERE id = :id AND status = 'PENDING'")
int claimEvent(@Param("id") String id);

int claimed = repository.claimEvent(event.getId());
if (claimed == 0) {
    // someone else got here first
    return;
}
processEvent(event);

This looks redundant next to the row lock. It isn’t. FOR UPDATE SKIP LOCKED only holds the lock for the lifetime of the transaction that ran the SELECT. The instant that transaction ends, the lock is gone. But a Kafka send can take a while, and if you fetch a batch in one transaction and then process each row in its own transaction. You often want that so one slow event doesn’t hold a lock over the whole batch. The original lock no longer protects you. The conditional UPDATE ... WHERE status = 'PENDING' is what survives that gap: if two workers reach the same row, only one update matches, the other gets back 0 and backs off. The claim check is the part that’s actually still standing during a long send.

When a worker dies mid-sentence

A worker grabs a row, sets it PROCESSING, starts the Kafka call, and then the process dies: deploy restart, OOM kill, the pod gets evicted. The row is now stranded. It’s PROCESSING, so no other worker will touch it, but processing never finished. It sits there forever unless you do something.

So before each cycle, run a recovery sweep:

@Query("UPDATE outflow_transaction_outbox SET status = 'PENDING', " +
       "failure_reason = 'Recovered from stuck PROCESSING state' " +
       "WHERE status = 'PROCESSING' AND last_modified_at < :threshold")
int recoverStuckEvents(@Param("threshold") LocalDateTime threshold);

@Scheduled(fixedRate = 30000)
public void processOutbox() {
    LocalDateTime fiveMinutesAgo = LocalDateTime.now().minusMinutes(5);
    int recovered = repository.recoverStuckEvents(fiveMinutesAgo);
    if (recovered > 0) {
        log.warn("Recovered {} stuck events", recovered);
    }
    // normal processing follows
}

If a row has been PROCESSING for over five minutes, assume the worker that claimed it is dead, reset it to PENDING, and let someone else try. This re-delivery is exactly the at-least-once behaviour from earlier. The dead worker might have reached Kafka before it fell over, which is why idempotent consumers aren’t optional. Set the threshold comfortably above your worst-case processing time, or you’ll re-queue events that are still legitimately in flight. Five minutes suits most cases; bump it if you call slow external rails.

When Kafka just isn’t feeling it today

Brokers restart. Networks partition. The cluster gets overloaded and times out. This is normal in production, not a bug.

The wrong reaction is to mark the row FAILED and quit on the first error. Most of these clear themselves in minutes. The other wrong reaction is to retry immediately in a tight loop, hammering a broker that’s already struggling. The right answer is exponential backoff:

public LocalDateTime calculateNextRetryTime() {
    // 30s, 1m, 2m, 4m, 8m, 16m...
    long delaySeconds = 30L * (long) Math.pow(2, this.retryCount);
    return LocalDateTime.now().plusSeconds(delaySeconds);
}

Thirty seconds, then a minute, two, four, eight, sixteen. After roughly half an hour of trying, it’s probably not transient, mark it FAILED for a human.

private void handleTransientFailure(OutboxEvent event, String reason) {
    event.incrementRetryCount();
    if (event.hasReachedMaxRetries()) {
        event.setStatus(OutboxStatus.FAILED);
        event.setFailureReason("Gave up. Last error: " + reason);
    } else {
        event.setStatus(OutboxStatus.PENDING);
        event.setScheduledAt(event.calculateNextRetryTime());
        event.setFailureReason(reason);
    }
}

The fetch query already respects the schedule, so anything booked for the future just waits its turn:

WHERE status = 'PENDING'
  AND (scheduled_at IS NULL OR scheduled_at <= NOW())

When retrying is hopeless

Some failures will never resolve no matter how long you wait: a customer ID that doesn’t exist, an amount below the minimum, a malformed request, a resource that’s been deleted. Retrying those just burns cycles and delays every event queued behind them.

So split your error codes:

// give up immediately
private static final List<String> PERMANENT = List.of(
    "50001",  // not found
    "50003",  // invalid request
    "90001",  // bad payment code
    "90002",  // bad customer ID
    "90003",  // below minimum
    "90004"   // above maximum
);

// try again later
private static final List<String> TRANSIENT = List.of(
    "11011",  // timeout
    "50002",  // still processing
    "99999"   // generic error
);

if (PERMANENT.contains(responseCode)) {
    event.setStatus(OutboxStatus.FAILED);
    event.setFailureReason("Permanent failure: " + responseCode);
} else {
    handleTransientFailure(event, responseCode);
}

Be conservative about what you call permanent. An unknown or generic code should default to transient. It’s far cheaper to retry something that turned out to be permanent than to silently drop something that would have recovered on the next attempt. In payments, dropping is the expensive mistake.

The one that almost got me

Look closely:

kafkaTemplate.send(TOPIC, key, payload);
event.setStatus(OutboxStatus.PROCESSED);

kafkaTemplate.send() is asynchronous. It returns a Future and hands control straight back. The message is still in flight, not acknowledged. If Kafka fails after that call but before the broker actually persists the record, you’ve stamped the row PROCESSED for a message that never landed. It’s gone, and there’s no retry, because you told the system it was done.

Wait for the receipt:

try {
    kafkaTemplate.send(TOPIC, key, payload).get();  // blocks for broker confirmation
    event.setStatus(OutboxStatus.PROCESSED);
    event.setProcessedAt(LocalDateTime.now());
} catch (Exception e) {
    handleTransientFailure(event, e.getMessage());
}

The .get() blocks until Kafka confirms delivery; only then do you mark it done. It’s slower, and you shouldn’t skip it. Pair it with producer config that makes the confirmation meaningful: acks=all so a partition leader change can’t quietly lose the record, and enable.idempotence=true so a producer retry under the hood doesn’t write a duplicate.

There’s a cost to be aware of here. That blocking .get() sitting inside a @Transactional method holds a database connection open for the entire duration of the Kafka round-trip. Under load, with a slow broker, you can starve your connection pool waiting on the network. If you see pool exhaustion, that’s the place to look. Moving the send outside the transactional boundary, or processing each row in its own short transaction, keeps connections from being held hostage to broker latency.

A word on ordering

Polling doesn’t give you ordering for free. With LIMIT 50 spread across three instances, two events for the same account can reach Kafka out of order. For plenty of payment flows that’s tolerable, but if a downstream consumer assumes per-account ordering, debit before the matching credit, say, you have to enforce it.

The standard move is to use aggregateId as the Kafka message key, so every event for one account lands on the same partition and stays ordered relative to its siblings. Remember the ceiling, though: ordering only holds within a partition, so the number of partitions caps how many accounts you can process in parallel while preserving order. If you also need ordering on the producing side, serialize processing per aggregateId: one in-flight event per account at a time. Don’t reach for any of this unless you actually need it; it costs throughput.

The whole thing together

@Scheduled(fixedRate = 30000)
public void processOutbox() {
    recoverStuckEvents();

    List<OutboxEvent> events = repository
        .findAndLockPendingEvents(LocalDateTime.now(), 50);
    if (events.isEmpty()) return;

    for (OutboxEvent event : events) {
        processEvent(event);
    }
}

@Transactional
public void processEvent(OutboxEvent event) {
    int claimed = repository.claimEvent(event.getId());
    if (claimed == 0) return;

    event = repository.findById(event.getId()).orElse(null);
    if (event == null) return;

    try {
        executeAction(event);
        kafkaTemplate.send(TOPIC, event.getAggregateId(), buildPayload(event)).get();
        event.setStatus(OutboxStatus.PROCESSED);
        event.setProcessedAt(LocalDateTime.now());
    } catch (PermanentException e) {
        event.setStatus(OutboxStatus.FAILED);
        event.setFailureReason(e.getMessage());
    } catch (Exception e) {
        handleTransientFailure(event, e.getMessage());
    }

    repository.save(event);
}

Keep an eye on it

The outbox won’t monitor itself. One query gets you most of the way:

SELECT
    status,
    COUNT(*) AS count,
    MAX(retry_count) AS max_retries,
    MIN(created_at) AS oldest
FROM outflow_transaction_outbox
GROUP BY status;

What to watch:

Signal	What it means
FAILED count > 0	Someone needs to look at these
PENDING rows older than an hour	Processing is backed up
High retry counts on PENDING	Something is consistently failing downstream
PROCESSING count stays high	Stuck workers; recovery may be misconfigured

Put it on a dashboard and alert on it. A healthy outbox is near-empty almost all the time. If it’s growing, something is wrong upstream or down.

One bit of housekeeping that’s easy to forget: prune PROCESSED rows. A busy outbox that never deletes delivered events balloons, and every locking query above slows with it. A nightly batch that deletes or archives PROCESSED rows older than a few days keeps the hot table small and the SKIP LOCKED scans fast.

So yeah, your systems were lying

Not maliciously. A database and a message broker are separate systems with separate guarantees, and expecting them to commit atomically together was never realistic.

The outbox fixes the relationship by never asking them to trust each other. Everything goes through the database, then gets relayed out. If the relay fails, you retry. If it keeps failing, a human looks. It’s not clever. A table, a background job, and some careful state management, but it keeps your money movements and your events in sync even when things go wrong. And in payments, things go wrong. At least now you’re ready for it.