Messenger Ravens and Royal Couriers: Building with Messages and Events

Not every message requires the messenger to wait at the castle gate.

A traveler arrives at the castle gate with an urgent request. The guard carries it to the steward, the steward consults the royal records, and the traveler remains outside until someone returns with an answer. This arrangement works when the traveler needs an immediate response, and the castle can provide one quickly. It becomes fragile when the work takes several minutes, the responsible official is unavailable, or five hundred travelers arrive at once. The problem is not that the road is poorly built. The problem is that every participant must remain available throughout the conversation.

In the previous chapter of The Kingdom in the Clouds, we examined the roads that connect cloud-native services. REST APIs, gRPC calls, service discovery, gateways, and load balancers allow distant parts of a system to communicate. Those roads answer an essential architectural question: how can one service reach another? They do not answer the equally important question of whether the sender should wait for the receiver. Once communication crosses a network boundary, waiting becomes an engineering decision rather than an invisible default.

Messages and events give distributed systems another way to coordinate. Instead of requiring two services to be available at the same moment, they place a durable intermediary between them. A message can wait until a consumer is ready, whereas an event can announce that something happened without requiring every interested service to respond immediately. This approach can absorb bursts of traffic and allow services to evolve independently. It can also introduce delayed results, duplicate delivery, unfamiliar debugging problems, and uncertainty about when the wider system has finished reacting.

The mature question is not whether asynchronous communication is better than synchronous communication. Neither model deserves the throne in every kingdom. The useful question is which parts of a workflow require an immediate answer and which merely require dependable progress. That distinction shapes nearly every decision that follows.

The Waiting Envoy: Synchronous Communication

A synchronous request creates a direct conversation between a caller and a receiver. A checkout service asks an inventory service whether an item is available, then waits for the response before continuing. This model is easy to understand because the control flow remains visible: request, processing, response. Errors can often be returned directly, and the caller knows whether the requested operation succeeded. When the response determines what must happen next, waiting may be entirely appropriate.

The cost is temporal coupling. Both services must be available during the same interval, and the caller must remain active until the receiver answers or a timeout expires. If the receiver calls another service, which calls two more services, the original request becomes dependent on an expanding chain of network conversations. Latency accumulates, and every dependency creates another opportunity for partial failure. A road may be open while the distant castle is overwhelmed, restarting, or waiting on a castle of its own.

Consider an order endpoint that performs every downstream action before returning:

</> C#

public async Task<OrderResult> PlaceOrderAsync(
    PlaceOrderRequest request,
    CancellationToken cancellationToken)
{
    var reservation = await inventoryClient.ReserveAsync(
        request.Items,
        cancellationToken);

    var payment = await paymentClient.AuthorizeAsync(
        request.CustomerId,
        request.Total,
        cancellationToken);

    await emailClient.SendConfirmationAsync(
        request.CustomerId,
        request.Email,
        cancellationToken);

    return new OrderResult(
        reservation.Id,
        payment.AuthorizationId);
}

The sequence looks orderly, but the customer-facing request now depends on inventory, payment, and email remaining responsive. Inventory and payment may belong on the critical path because the application cannot honestly confirm an order without them. Email usually does not. If the mail provider takes twelve seconds to respond, the order may succeed while the customer sees a timeout and tries again. The code has allowed a secondary notification to determine the apparent success of the primary business operation.

The order of statements in code should not automatically reflect the system’s dependency structure. Some work must be completed before the caller receives an answer, but other work only needs to occur afterward with reasonable reliability. Treating every step synchronously makes implementation straightforward at the expense of runtime independence. Experienced engineers identify the point at which the business decision is complete, then ask whether the remaining work can be left by messenger.

The Courier Station: Message Queues

A message queue allows a producer to submit work without requiring a consumer to process it immediately. The broker accepts the message, stores it in accordance with its durability guarantees, and makes it available to an eligible consumer. The producer can continue once it knows the broker accepted the message. Consumers process queued work at their own pace, separating the arrival rate from the processing rate. The courier station does not make the journey instantaneous, but it prevents every sender from occupying the road until delivery is complete.

This separation is valuable when traffic arrives in bursts. Imagine that a ticketing system normally processes 20 purchases per minute but processes several thousand during a major announcement. A synchronous notification service must scale quickly enough to match that arrival rate, or it will force purchasing requests to wait. A queue can hold notification work while a stable group of consumers drains the backlog. The system has not eliminated load; it has converted an immediate capacity requirement into a manageable period of delayed processing.

That buffering creates operational obligations. Engineers must monitor queue depth, message age, processing rate, failure rate, and dead-letter volume. A queue with ten thousand messages may be healthy if consumers are draining it rapidly, or disastrous if the oldest message has waited six hours. Autoscaling based solely on queue depth can also mislead, as a poison message may fail repeatedly without reflecting useful demand. A queue is not a cupboard where unfinished work can be hidden. It is a visible inventory of promises the system has not yet fulfilled.

Message queues typically deliver each message to a single consumer within a competing group. If five workers listen to the same order-processing queue, the broker normally assigns each message to one of them rather than all five. Adding consumers can increase throughput until another constraint becomes dominant, such as database capacity, third-party rate limits, or partition contention. The broker can regulate delivery, but it cannot repeal the limits of the systems behind the consumers.

Sealed Instructions: Commands as Messages

A command expresses an intention for a particular capability to perform work. Examples include generating an invoice, resizing an uploaded image, or sending an order confirmation. The sender knows what kind of work it wants performed, even if it does not know which process will perform it. Commands are therefore directed communication. They resemble sealed instructions handed to the courier station for delivery to the appropriate guild.

A useful command contains the information required to perform the operation, along with metadata that supports reliable processing. That often includes a unique message identifier, a correlation identifier, the creation time, and a contract version. It should avoid carrying a convenient copy of every related business object because those copies become stale and make contracts difficult to evolve. The message should express the work without becoming a traveling replica of the producer’s database.

For example, the order service could move email delivery beyond the customer-facing request:

</> C#

public sealed record SendOrderConfirmation(
    Guid MessageId,
    Guid OrderId,
    Guid CustomerId,
    string EmailAddress,
    DateTimeOffset RequestedAt,
    int SchemaVersion);

public async Task ConfirmOrderAsync(
    Order order,
    CancellationToken cancellationToken)
{
    await orderRepository.MarkConfirmedAsync(
        order.Id,
        cancellationToken);

    var command = new SendOrderConfirmation(
        Guid.NewGuid(),
        order.Id,
        order.CustomerId,
        order.EmailAddress,
        DateTimeOffset.UtcNow,
        1);

    await messageSender.SendAsync(
        command,
        cancellationToken);
}

The customer no longer waits for the email provider, but the design has gained a dangerous gap. The database update may succeed, but the send operation may fail, leaving a confirmed order without a confirmation command. Reversing the operations merely reverses the inconsistency: the command may be sent before the database commit fails. The code demonstrates the architectural goal, but production reliability requires a deliberate strategy such as a transactional outbox. Asynchronous communication moves work out of the request path; it does not absolve the system of the responsibility of proving that accepted work will survive the journey.

Command messages also require clear ownership. If several unrelated services consume the same command and interpret it differently, the sender can no longer reason about what the instruction means. A command should normally have one logical handler, even when multiple worker instances compete to execute it. Events follow a different rule because they describe history rather than assign work. That difference between requesting an action and announcing a fact is where event-driven architecture truly begins.

Bells Across the Realm: Events as Facts

An event records something that has already happened. An order was confirmed, a payment failed, an account was created, or a shipment left the warehouse. Unlike a command, an event does not instruct a particular service to perform a specific action. It announces a fact that other parts of the system may find meaningful. Once the bell rings across the kingdom, each guild decides whether that news concerns its responsibilities.

This distinction influences ownership. The service that owns the underlying business state publishes the event, but it does not need to know every subscriber that may react. An OrderConfirmed event might be consumed by notification, analytics, fulfillment, loyalty, and fraud-monitoring services. New consumers can be added without modifying the order service, provided they understand the published contract. The producer owns the fact, while each consumer owns its reaction to that fact.

That independence is the foundation of loose coupling, but loose coupling does not mean the services have no relationship. Consumers still depend on the meaning, structure, availability, and timing of the events they receive. A producer that renames fields or subtly redefines what “confirmed” means can break consumers without changing the direct API. The dependency has moved from a synchronous endpoint to an asynchronous contract. It has not disappeared into the clouds.

Commands and events should therefore be named differently. ConfirmOrder is a request that may succeed or fail, while OrderConfirmed states that the transition has already occurred. A command may be rejected because inventory disappeared or payment authorization failed. An event should not describe a hoped-for outcome that remains uncertain. Naming events in the past tense helps prevent architecture from confusing intention with history.

The Royal Proclamation: Publish-Subscribe

Publish-subscribe communication allows one event to reach multiple independent subscribers. The publisher sends the event to a topic, exchange, or event stream rather than addressing each consumer directly. The broker then delivers it according to configured subscriptions and routing rules. Every interested service can receive its own copy and process it independently. One proclamation can therefore begin several workflows without forcing the announcing service to coordinate them.

Suppose an order service publishes OrderConfirmed. The fulfillment service reserves warehouse work, the notification service sends a receipt, and the analytics service records the conversion. If analytics is temporarily unavailable, fulfillment can continue because subscribers do not share a single synchronous call chain. When the analytics consumer returns, it can process accumulated events from its subscription. The order service does not need to wait for every guild to acknowledge the news.

This model also supports extensibility. Months later, a loyalty service can subscribe to the event and award points without requiring another deployment of the order service. The producer remains focused on order rules rather than accumulating knowledge about marketing, reporting, and customer engagement. That separation allows teams to change their own capabilities with fewer coordinated releases. New reactions can be attached to established facts.

The benefit can be overstated, however. Adding a subscriber may be technically easy while remaining operationally expensive. Every consumer adds code, deployment, monitoring, data retention, schema compatibility, security, and failure-handling responsibilities. A single event can quietly become the foundation of many business processes, making its contract more consequential than a public API. Event-driven systems reduce coordination during execution only when disciplined engineering supports them.

The Keeper of the Aviary: Event Brokers

An event broker manages the movement of messages between producers and consumers. Depending on the platform, it may provide durable queues, topics, routing rules, acknowledgments, retries, partitions, retention, and dead-letter destinations. Some brokers emphasize work distribution, while others preserve event streams that consumers can replay. These differences affect how systems recover, scale, and reason about history. Choosing a broker based solely on popularity is like selecting a royal courier because the horse has impressive armor.

Queue-oriented brokers often suit discrete units of work that one eligible consumer should handle. Stream-oriented platforms commonly retain events for a configured period and allow several consumer groups to read the same history at different speeds. The choice should begin with workload behavior rather than product reputation. Engineers should ask whether replay is required, whether ordering matters, how traffic will be partitioned, and how long messages must remain available. Throughput, message size, security, operating experience, and acceptable delay also matter.

Managed brokers can reduce infrastructure work, but they do not transfer responsibility for message design. The provider may operate servers, replace failed nodes, and expose metrics, yet the application team still defines retry behavior, retention, permissions, and consumer semantics. Cloud services change who maintains the aviary, not who decides which messages deserve delivery. The most damaging messaging failures often come from ambiguous application behavior rather than unavailable broker nodes.

The Raven That Returns Twice: Delivery Guarantees

Distributed messaging forces engineers to confront an uncomfortable truth: delivery is rarely as simple as sent once and processed once. A consumer may finish its database update and crash before acknowledging the message. Because the broker did not receive the acknowledgment, it resends the message. From the broker’s perspective, retrying is safer than silently losing work. From the consumer’s perspective, the same command has returned with suspiciously familiar feathers.

Many systems therefore provide at-least-once delivery. A message should reach the consumer, but duplicates are possible. At-most-once delivery avoids duplicate attempts by accepting that some messages may be lost if failure occurs at the wrong moment. Claims of exactly-once delivery usually apply within carefully defined technical boundaries. They do not automatically ensure that every external API call, database update, and business effect occurs exactly once.

Reliable consumers must consequently be designed for idempotency. An idempotent operation can be attempted more than once without producing an unintended additional effect. Setting an order status to confirmed is easier to make idempotent than incrementing a reward balance or charging a credit card. The first operation describes a desired state, while the others create cumulative effects. When cumulative work is unavoidable, the consumer needs a durable way to recognize messages it has processed.

A message identifier can support that recognition:

</> C#

public async Task HandleAsync(
    OrderConfirmed message,
    CancellationToken cancellationToken)
{
    if (await processedMessages.ExistsAsync(
        message.MessageId,
        cancellationToken))
    {
        return;
    }

    await transactionRunner.ExecuteAsync(async () =>
    {
        await loyaltyAccounts.AddPointsAsync(
            message.CustomerId,
            message.RewardPoints,
            cancellationToken);

        await processedMessages.RecordAsync(
            message.MessageId,
            DateTimeOffset.UtcNow,
            cancellationToken);
    });
}

The deduplication record and business update belong in the same database transaction. If the points are committed but the identifier is not, a retry can award them again. If the identifier is recorded before the points are committed, the retry may be discarded even though the business work never finished. The pattern does not prevent redelivery; it makes redelivery safe. Duplicate messages are a common occurrence in distributed systems, not evidence that the broker betrayed the crown.

External side effects make the problem harder because a local transaction typically cannot include a payment provider or an email service. The consumer may need an idempotency key supported by the provider, a local state machine, or another outbox to reliably schedule the external action. Exactly once across an entire distributed workflow is usually a business design constructed from smaller guarantees, not a switch hidden in the broker configuration. Every boundary must be examined on its own terms.

Scrolls That Outlive Their Scribes: Event Contracts

Events frequently outlast the code that first published them. Messages may wait during an outage, remain in a replayable stream for months, or be stored for auditing. Producers and consumers also deploy independently, so several contract versions may exist in production at once. A field change that appears harmless in one repository can make old messages unreadable or break a slower consumer. Event contracts must therefore be treated as durable interfaces.

Additive changes are generally safer than destructive ones. A producer can often add an optional field while preserving existing fields and their meanings. Removing a field, changing its type, or redefining its semantics requires a deliberate migration. Consumers should tolerate fields they do not recognize, and producers should not assume that every consumer upgrades immediately. Compatibility tooling can enforce structure, but it cannot detect a quiet change in business meaning.

Events should also carry enough context to be traced. A message identifier distinguishes one message from another, while correlation and causation identifiers connect related steps in a wider workflow. Timestamps, producer names, and schema versions provide further diagnostic context. These fields may appear ceremonial during local development, but during a production incident, they become the map showing how a single decision traveled across the realm.

Good event design resists both starvation and excess. An event containing only an entity identifier may force every consumer to call the producer, recreating synchronous coupling behind asynchronous decoration. An event containing a complete database record exposes internal structure and distributes data that consumers may not need or be authorized to receive. The right payload communicates the fact and the stable information legitimate consumers need. A message should carry enough provisions for its journey without attempting to transport the entire castle.

The Ledger and the Courier: Publishing Reliably

The earlier order example exposed a gap between updating business data and sending a command. A transactional outbox closes that gap by writing both the business change and an outgoing message record in the same database transaction. A separate publisher reads pending outbox records, sends them to the broker, and marks them as published. If publishing fails, it can try again without losing track of the work that remains. Duplicate publication is still possible, so consumers must remain idempotent, but the system no longer depends on two unrelated operations succeeding together.

The outbox pattern adds storage, background processing, cleanup, and monitoring. Those costs are justified when losing a message would leave important business state unfinished. They may be unnecessary for low-value telemetry or notifications that can be safely regenerated. Reliability should be proportional to consequence. Architecture becomes expensive when every message is treated like a royal decree, but it becomes dangerous when a royal decree is treated like tavern gossip.

News That Travels Slowly: Eventual Consistency

Asynchronous communication means different parts of the system can temporarily hold different views of reality. An order may be confirmed while the analytics dashboard still shows the previous total and the loyalty account has not received its points. This is eventual consistency: the system accepts a period of disagreement while messages travel and consumers update their own state. The model improves independence and availability, but users and engineers must understand which facts may lag and for how long.

Eventual consistency is not permission to be vaguely correct someday. Each workflow needs an acceptable consistency window, a way to detect stalled progress, and a recovery path when processing cannot complete. A confirmation page may display order data immediately while labeling reward points as pending. An operations dashboard may alert when the oldest unprocessed message is more than 5 minutes old. A useful system makes temporary uncertainty visible instead of disguising it as certainty.

Ordering adds another complication. Brokers may preserve order only within a queue, partition, session, or message group, and parallel consumers can complete work in a different sequence from the one in which it began. If OrderCancelled arrives before an earlier OrderConfirmed After the event finishes processing, a careless consumer may restore the obsolete state. Partitioning related events by order identifier, recording entity versions, or rejecting stale transitions can protect sequence-sensitive workflows. Global ordering is rarely worth its cost when only events concerning the same business entity must remain ordered.

The Undeliverable Scroll: Retries and Dead Letters

Retries are appropriate for transient failures such as brief network interruptions, broker throttling, or temporarily unavailable dependencies. They should use bounded attempts and increasing delays to prevent a failing service from being attacked by its own recovery mechanism. Permanent failures, including invalid data or incompatible schemas, will not become healthy through determination alone. After an appropriate number of attempts, the message should move to a dead-letter destination for investigation or controlled replay.

A dead-letter queue is not a graveyard that makes failed work disappear. Teams need ownership, alerts, diagnostic context, and a documented process for deciding whether a message should be repaired, replayed, compensated, or discarded. Replaying messages without correcting the cause sends the same doomed courier down the same broken road. Failure handling is part of the business workflow, not merely broker configuration.

Lanterns Along the Route: Observing Asynchronous Work

Traditional request logs are insufficient when a single customer action produces messages processed across several services and over minutes. Correlation identifiers should travel with the workflow so logs, traces, and metrics can reconstruct its path. Teams should measure publish failures, processing latency, retries, duplicate detection, queue depth, oldest message age, and dead letter volume. Distributed tracing can connect producers, brokers, and consumers, although asynchronous boundaries require deliberate propagation of trace context.

Observability should answer a business question as well as a technical one: which accepted promises remain unfulfilled? A healthy broker does not prove that confirmations were sent, points were awarded, or shipments were scheduled. Infrastructure metrics show whether the roads exist; application metrics show whether the kingdom’s work is arriving. That distinction often separates a system that merely runs from one that can be trusted.

Choosing the Messenger: Engineering Judgment

Use synchronous communication when the caller needs an immediate answer, the operation belongs on the critical path, and the dependency can meet the required latency and availability. Use asynchronous communication when work can continue later, when traffic needs buffering, when multiple consumers should react independently, or when temporary receiver unavailability should not block the sender. Many reliable systems combine both models. An order might synchronously validate inventory and authorize payment, then asynchronously begin fulfillment, notifications, analytics, and loyalty processing.

The strongest design does not maximize the number of events. It places waiting only where waiting protects a real business decision. Every asynchronous boundary should earn its complexity through resilience, scalability, independence, or extensibility. Every synchronous dependency should justify the availability and latency it imposes on the caller. Communication style should follow business semantics rather than architectural fashion.

Beyond the Castle Gate: Building a Distributed Realm

Messages and events allow services to coordinate without remaining locked in the same moment. Queues buffer work, publish-subscribe distributes facts, and idempotent consumers make redelivery safe. Transactional outboxes protect the passage from committed state to outgoing messages, while schemas, monitoring, and dead-letter handling keep asynchronous workflows understandable. Removing the wait does not remove responsibility for the outcome.

This week’s theme, The Distributed Realm, begins with distance changing the meaning of communication. Once messages can arrive later, different services may temporarily disagree about what is true. On Wednesday, the next chapter of The Kingdom in the Clouds, When Two Kings Claim the Throne: Consistency in Distributed Systems, will examine how distributed systems balance consistency, availability, replication, and competing versions of reality. The couriers have carried the news across the kingdom; now we must decide which ledger tells the truth.