Apache Kafka, 9092 from the host and 29092 inside the compose network. Three topics, three dead-letter topics, and one message envelope that every service understands.
Compose starts the broker process and stops there. Everything above that line is the team's work: creating the topics with the contracted names, partition counts, replication factor and retention, writing down why those numbers are right, and configuring every producer and consumer. Auto-creation is switched off, so a team that has not created its topics gets an error rather than a one-partition topic with default retention that silently behaves almost correctly.
The catalogue in docs/contracts/kafka-topics.md is binding. Renaming a topic or changing
a field breaks every consumer in the platform, including ones another team wrote.
The key decides the partition, and the partition decides the ordering guarantee. Kafka orders messages within a partition and gives no ordering across partitions.
orders and trade-events are keyed by accountId because the
ordering that matters is per account. Two orders on the same account must execute in the order they were
accepted, or a sell can be processed before the buy that made it possible. Two orders on different
accounts have no relationship and can run in parallel.
market-data is keyed by symbol because the ordering that matters is per
instrument. A consumer must never see an older quote for AAPL after a newer one. Never key
by order identifier: every message lands on its own partition and per-account ordering is gone.
Three partitions lets three executor instances run in one group, which is enough to demonstrate a
rebalance and enough to show that a group cannot usefully exceed the partition count. Six on
market-data reflects its higher message rate. Partitions can be increased later but never
decreased, and increasing them rehashes keys, so an account's history splits across partitions from that
point. Pick the number in Sprint 7 and record why.
| Component | orders | trade-events | market-data |
|---|---|---|---|
| Trade REST API | produce | consume, to update read models | not used at the core |
| Trade Executor | consume, group trade-executor | produce | produce, from the scheduled poller inside it |
| Analytics service | not used | consume, optional, group analytics-loader | not used |
| Portfolio and P&L module | not used | consume, group portfolio-service | consume, group portfolio-service |
| Watchlists and price alerts module | not used | not used | consume, group watchlist-service |
| Customer notifications module | not used | consume, group notification-service | not used |
| Customer preferences module | not used | not used | not used |
| Trade advice and signals module | not used | consume, group advice-service | consume, group advice-service |
| Automated strategy execution module | not used | consume, group strategy-service | consume, group strategy-service |
| Frontend | never | never | never |
Group ids stay distinct per extension module even though the modules run in one process. A group id names a logical consumer, and separate ids keep each module's offsets independent: a redeployed notifications consumer must not move the watchlist consumer's position in the stream. Two consumers sharing a group id split the partitions between them, and each sees only part of the stream, which presents as messages going missing at random.
The platform runs at-least-once. Producers retry, consumers commit offsets after processing, and duplicates therefore happen. Plan for them; do not try to eliminate them.
| Setting | Value | Why |
|---|---|---|
acks | all | Wait for the in-sync replicas. A local single-broker cluster makes this cheap and a real cluster makes it necessary. |
enable.idempotence | true | Removes duplicates caused by a producer retry. It does not remove duplicates caused by an application retrying after a crash. |
max.in.flight.requests.per.connection | 5 or fewer | Keeps ordering inside a partition while a retry is outstanding. |
enable.auto.commit | false | Commit after processing. Committing first loses the message on a crash; committing after reprocesses it, and reprocessing is survivable when the handler is idempotent. |
group.id | explicit, per logical consumer | A default or a shared group id is the most common cause of "messages disappearing". |
auto.offset.reset | earliest | A new consumer group reads the retained history rather than only what arrives after it starts. |
Every consumer with a side effect must survive seeing the same eventId twice. Two
mechanisms are acceptable: a processed-events table keyed on eventId and written inside the
same transaction as the side effect, or a guarded state transition. The Trade Executor uses the second,
and demonstrating it is the Sprint 7 acceptance check.
Kafka transactions can give exactly-once between topics. This platform does not use them, because the side effects here are database writes rather than topic writes, and the guarded transition gives the same outcome with less machinery. Be able to explain that choice.
.DLT topics.Local development runs plaintext with no authentication, because TLS, SASL and ACLs on a single-node
broker teach configuration rather than architecture. Document what you would configure in production:
TLS between clients and brokers, SASL for client authentication, and per-topic ACLs so that only the
Trade REST API can write to orders and only the Trade Executor can read it. Document it as
a plan, and do not claim it is implemented.
Never put a credential, a full name, an email address or an API key in a message payload. Topics are retained for days and read by services that have no need for that data.