Fault-tolerant event-driven architecture for enterprise scale.
Design resilient event streams with transactional outbox consistency, idempotent consumers, schema contract governance and dead-letter handling, so services stay decoupled without losing or double-applying events.
Design targets · not measured results from past client work
Target · Message Reliability99.999%
Replicated durable logs and at-least-once delivery, with idempotent consumers, designed to avoid lost or double-applied events.
Target · Event Latency<5ms
Producer publish to consumer acknowledgement, for a well-sized cluster under normal load.
Target · Events Per Second10M+
Horizontally scaled Kafka clusters designed for sustained traffic bursts.
Target · Cascading Timeouts0
Asynchronous decoupling removes synchronous dependency chains, the usual source of timeout cascades.
The short answer
What is event-driven architecture?
Event-driven architecture is a design in which services communicate by publishing and reacting to events, meaning records that something happened, such as an order being paid. A producer publishes an event, a broker or router delivers it, and any number of consumers react in their own time. Because services only know the event, not each other, they can scale, fail and be deployed independently. The price is new work: handling duplicates, ordering, schema changes and failures that no longer show up as a simple error response.
Producers
Services that publish events when state changes, without waiting for anyone to react.
Broker or router
Kafka, RabbitMQ or a managed bus that stores or routes events and delivers them to interested consumers.
Consumers
Services that react to events, and must cope with duplicates, retries, ordering limits and changing schemas.
Robral provides event-driven architecture consulting and engineering for enterprise platforms: topic and broker design, transactional outbox, idempotent consumers, schema governance and the failure tests that prove it, delivered first as a 14-day proof of concept.
Core engineering pillars
Three pillars. One fault-tolerant messaging backbone.
Each pillar is independently deployable. Applied together they are designed to address the most common failure modes in distributed event systems.
01
Topology & Throughput
Event Broker & Topic Design
Kafka and RabbitMQ cluster topologies with partition strategies that preserve ordering where it matters, replication sized for durability targets, and in-sync replica settings that let the cluster tolerate broker failures without losing acknowledged writes.
Topic partition layout aligned to consumer group concurrency
In-sync replica settings with min.insync.replicas enforced
Retention policies per topic class (compacted vs time-bounded)
Schema-aware producer serialization with Avro / Protobuf
Rolling Kafka upgrades planned to avoid consumer downtime
02
Contract Governance
Schema Registry & Contracts
Confluent Schema Registry with compatibility checks enforced in CI. Breaking changes are blocked before they reach producers, which prevents silent data corruption across service boundaries. Schemas are versioned, discoverable and used automatically by consumer deserialization.
Confluent Schema Registry with compatibility enforcement
Breaking-change detection in CI / pre-deploy gates
Schema IDs carried with every message
Avro and Protobuf schema generation from domain models
Consumer migration tooling for backward-incompatible schemas
03
Audit & Replay
Event Sourcing & CQRS
An immutable event store as the system of record. State is derived by replaying ordered events rather than mutating rows. Independent CQRS projections per read use case keep reads fast without touching the write path, and replay supports audit trails and point-in-time state.
Immutable event store — state derived from event replay
Independent CQRS projections per read query pattern
Audit trail with point-in-time state hydration
Snapshots for efficient aggregate reconstruction at scale
Re-hydrate new microservices from historical event streams
Delivery guarantees
At-most-once, at-least-once or effectively-once: which do you need?
Every messaging design chooses what happens when something fails between send and processing. Knowing the trade-off is what keeps events from being lost or applied twice.
Guarantee
What can go wrong
How it is achieved
Best for
What you must build
At-most-once
Events can be lost, never duplicated.
Fire-and-forget sends, or commit the consumer position before processing.
Metrics, logs and other data where a gap is acceptable.
Nothing special, but accept the loss.
At-least-once
Events are never lost but can arrive more than once.
Acknowledged writes (acks=all), retries, and committing the consumer position after processing. This is the Kafka default.
Most business events: orders, payments, inventory.
Idempotent consumers, so a repeated event has no extra effect.
Effectively-once
Duplicates may still be delivered, but their effects are applied once.
At-least-once delivery plus idempotent producers, an outbox, and inbox or key-based deduplication. Kafka transactions cover Kafka-to-Kafka processing.
Financial and stock changes where double-applying is costly.
Deduplication keys, transactional boundaries and tests that replay and crash.
Kafka documents at-most-once, at-least-once and exactly-once, with idempotent producers and transactions inside Kafka. Effects on external systems, such as a database or a payment call, still need idempotent handling. See the references below.
Platform capabilities
Every messaging primitive, engineered for production.
From dead-letter queues and idempotency to CDC pipelines and lag monitoring, the full event delivery stack.
Dead-Letter Queues & Circuit Breaking
Retries with exponential backoff and jitter, poison-message isolation, and circuit breakers that stop a blocked consumer from backing up the system. Failed messages land in dead-letter topics with context metadata for controlled replay.
Brokers usually deliver at least once, so consumers must tolerate duplicates. Event IDs are recorded in an inbox table in the same transaction as the state change, so a retry or replay is detected and skipped.
Events are written to an outbox table in the same database transaction as the domain change. Debezium change data capture streams committed outbox rows to Kafka, which removes the dual-write problem. Delivery is at least once, so consumers stay idempotent.
Consumer group lag tracking with per-partition alert thresholds. Trends surface backpressure before it becomes latency spikes, and lag depth can drive scaling of consumer groups.
Lag AlertingPer-Partition MonitoringBackpressure DetectionAuto-Scaling
Engineering notes
The outbox and the idempotent consumer.
Two small pieces of SQL remove most of the data-consistency problems in event-driven systems.
Transactional outbox and inbox (PostgreSQL, illustrative)
-- Producer: state change and event in ONE transaction
BEGIN;
UPDATE orders SET status = 'PAID' WHERE id = :order_id;
INSERT INTO outbox (id, aggregatetype, aggregateid, type, payload)
VALUES (gen_random_uuid(), 'order', :order_id, 'OrderPaid', :payload);
COMMIT; -- Debezium streams committed outbox rows to Kafka
-- Consumer: skip events already handled
BEGIN;
INSERT INTO inbox (event_id) VALUES (:event_id)
ON CONFLICT (event_id) DO NOTHING;
-- 0 rows inserted: duplicate, roll back and acknowledge
-- 1 row inserted: apply the change in this same transaction
COMMIT;
Outbox rows are only ever inserted. Delivery is at least once, so the consumer-side inbox is what makes processing safe to repeat. The column names above follow the defaults of the Debezium outbox event router.
Failure test checklist
Kill the consumer after processing but before committing the offset; the event is reprocessed with no double effect.
Redeliver the same event twice; the state changes once.
Crash the producer between the database write and the publish; the outbox still delivers the event.
Publish events out of order for one key; the consumer handles or rejects stale versions.
Send a malformed event; it goes to the dead-letter topic and the consumer keeps moving.
Publish a breaking schema change; CI rejects it before deployment.
Stop a consumer for an hour; lag alerts fire, and catch-up does not overwhelm downstream systems.
Replay a topic into a fresh consumer; the result matches the original state.
Settings that matter
The Kafka settings behind reliable delivery.
Exact settings are where most reliability problems hide. These are the ones we check first.
Producer durability
Use acks=all with enable.idempotence=true, so acknowledged writes are replicated and retries cannot create duplicate log entries. Set these explicitly rather than relying on client defaults.
acks=allenable.idempotence=trueRetries
Replication and in-sync replicas
A replication factor of 3 with min.insync.replicas=2 lets a topic lose one broker while still accepting acknowledged writes, and refuses writes when too few replicas are in sync.
replication.factor=3min.insync.replicas=2Unclean election off
Consumer offsets
Turn off auto-commit and commit offsets after the work is done. A crash then causes a repeat, never a gap, which is why the consumer must be idempotent.
enable.auto.commit=falseCommit after processingIdempotent handlers
Transactional reads
When producers use Kafka transactions, consumers should read with isolation.level=read_committed so they never see messages from aborted transactions.
Agree a compatibility mode per topic and enforce it in CI. Confluent Schema Registry defaults to backward compatibility; long-lived topics with replay often need stricter, transitive modes.
BACKWARDFULL_TRANSITIVECI gate
Retries and dead letters
Separate retry topics with increasing delay, a capped attempt count, and a dead-letter topic that keeps the original payload, error and attempt history for replay after a fix.
Retry topicsBackoff + jitterDLQ with headers
What you receive
An event streaming topology. In 14 days.
Illustrative dashboard · not a client screenshot
01
Event Streaming Topology Blueprint
Kafka or RabbitMQ cluster design, topic partition strategy, consumer group topology and retention policy configuration tailored to your throughput targets.
02
Schema Governance Runbook
Confluent Schema Registry setup, Avro/Protobuf schema definitions for your domain events, compatibility gate configuration and CI enforcement hooks.
03
Phased Migration Roadmap
Prioritized sequence for moving existing synchronous API calls to event-driven patterns, with risk assessment, rollback strategies and consumer cutover playbooks.
Delivered in 14 days: a topology blueprint, a schema governance runbook and a phased migration roadmap.
Choosing a partner
How to choose an event-driven architecture partner.
Six questions to ask any firm, including us, before you commit.
01
Delivery guarantees
Ask which guarantee they design for and how duplicates are handled. Look for at-least-once delivery with idempotent consumers, and an honest view of what exactly-once does and does not cover.
Red flag: "exactly-once, no duplicates ever"
02
Consistency at the database boundary
Ask how a state change and its event stay in step. Look for a transactional outbox with change data capture, or an equivalent, not two separate writes.
Red flag: write to the database, then publish
03
Failure handling
Ask what happens to a message that keeps failing. Look for retry topics, dead-letter queues, replay tooling and alerts.
Red flag: infinite retries on the main topic
04
Schema governance
Ask how a producer team ships a schema change safely. Look for a registry, compatibility checks in CI and a consumer migration path.
Red flag: schemas agreed in a document
05
Operations
Ask how lag, throughput and broker health are watched, and who is on call. Look for per-partition metrics, alerts and runbooks.
Red flag: no lag monitoring
06
Ownership and handover
Ask what you own at the end. Look for code, infrastructure as code, runbooks and training handed over, with acceptance criteria agreed up front.
Red flag: black-box delivery
Questions and answers
Event-driven architecture: common questions.
What is event-driven architecture?
It is a design where services communicate through events, records that something changed. Producers publish events, a broker or router delivers them, and consumers react independently. AWS describes it as using events to trigger and communicate between decoupled services. The benefit is loose coupling and independent scaling; the cost is handling duplicates, ordering, schema changes and asynchronous failures.
Does Kafka guarantee exactly-once delivery?
Kafka supports exactly-once processing inside Kafka, using idempotent producers and transactions, and consumers can read with isolation.level=read_committed. Its default delivery is at least once. When a consumer writes to an external system such as a database or a payment API, the effect is only applied once if that write is idempotent or part of the same transaction.
What is the transactional outbox pattern?
A service writes the event to an outbox table in the same database transaction as its state change, so either both happen or neither does. A separate process, often Debezium change data capture, reads committed outbox rows and publishes them to the broker. This avoids the dual-write problem. Delivery is at least once, so consumers must handle duplicates.
What is an idempotent consumer?
It is a consumer that produces the same result no matter how many times it receives the same event. A common way is an inbox table: record the event ID in the same transaction as the state change, and skip the event if the ID already exists. Debezium outbox messages carry an event ID header that consumers can use for this.
When should I use Kafka and when RabbitMQ?
Kafka is built around a replicated, replayable log, which suits high-throughput streams, event sourcing and many independent consumers reading the same history. RabbitMQ is a message broker with flexible routing and acknowledgment, which suits task queues and request-style workflows. Many platforms use both. The choice depends on throughput, replay needs and routing patterns.
How do you handle schema changes without breaking consumers?
Register schemas in a registry such as Confluent Schema Registry, choose a compatibility mode per topic, and check every change in CI before deployment. Add optional fields rather than renaming or removing them, and migrate consumers before producers when a breaking change is unavoidable.
What do you receive from a 14-day proof of concept?
A streaming topology blueprint (brokers, topics, partitions, retention), a schema governance runbook with CI hooks, and a phased roadmap for moving your synchronous calls to events, including rollback and consumer cutover. It is meant to show the design working on one of your flows, not to replace the full build.
