System Design 101 : Observers

All system design problems reduce to managing how time, wrongness, and pressure are perceived by their observers.

An observer is anything that forms expectations about the system and reacts when those expectations are violated.

It expects something to happen. It changes behavior when that expectation isn't met.

If something doesn't react to unmet expectations, it isn't an observer. It's state. A log file holds whatever is written to it regardless of what that content represents. A database row persists, indifferent to meaning. These store. They don't expect.

Four modes of observation emerge from this definition.

Initiators start an action and wait for acknowledgment.

They expect to know whether their action counted. They detect violation when they act and hear nothing back, or hear back too late to matter.

They react by retrying. They care about confirmation. Failure is acceptable. Ambiguity is not.

Recipients consume results they did not initiate.

They expect what they receive to hold together. They detect violation when data contradicts itself, references things that don't exist, or cannot be assembled into meaning.

They react by delaying, rejecting, or working around it. They care about coherence. Staleness is acceptable. Nonsense is not.

Supervisors watch the system for health.

They expect the system to keep moving and failures to announce themselves. They detect violation when queues grow unboundedly, processes stall, or silence replaces expected signals.

They react by intervening: restarting, rolling back, alerting. They care about visibility. Degradation is acceptable. Invisibility is not.

Dependents build their correctness on top of yours.

They expect stable meaning over time. They detect violation when responses change meaning, state transitions skip steps, or historical data shifts silently.

They react by cascading failure or compensating incorrectly. They care about semantic stability. Failure is acceptable. Surprise is not.

Consider placing an order. Someone expresses intent to exchange value, the system acknowledges that intent, and other parties act on that acknowledgment.

The initiator is the party expressing intent. They click a button and wait. They expect to know whether the order registered. When the screen hangs indefinitely or an email confirms what the website denies, they retry. An order can be delayed or canceled later. Not knowing whether it exists is the failure they cannot tolerate.

The recipients are parties who must act on the order without having placed it. The warehouse preparing items. The inventory system reserving stock. The shipping provider arranging delivery. They expect orders that make sense: valid product IDs, positive quantities, parseable addresses. When an order references a product that doesn't exist or arrives twice with conflicting details, they delay or reject. They don't care who placed it. They care that they can act on it.

The supervisors are anything watching order flow rather than participating in it. Operations dashboards. Automated monitors. Systems comparing throughput across stages. They expect orders to keep moving and problems to surface. When queues grow without draining or orders sit unprocessed for hours, they intervene. A few incorrect orders are tolerable. Silent accumulation is not.

The dependents are anything building future behavior on order data. Financial reconciliation. Analytics. Fraud detection. Compliance. They expect an order to mean today what it meant yesterday. When a "completed" status starts appearing on canceled orders, or a currency field sometimes contains text, their downstream computations break. They often discover this long after the damage is done.

The same customer occupies different modes at different moments.

When clicking "Place Order," they care about confirmation. When viewing order history, they care about coherence. Same person. Different expectation shape.

This has design consequences. If you optimize order placement for consistency by delaying acknowledgment until all systems agree, initiators retry and create duplicates. If you optimize order history for speed by showing data before it's consistent, recipients see orders appear and disappear.

Each mode requires different optimization. Most failures happen when you optimize one mode at the expense of another without recognizing the tradeoff.

Consider leader election. It has no users.

Nodes compete for leadership. One wins, others follow. If the leader fails, a new election occurs.

The initiators are nodes proposing candidacy and clients submitting writes. Both act and wait for acknowledgment. Both retry when they hear nothing back.

The recipients are follower nodes receiving log entries and clients reading state. Both consume what they didn't initiate. Both reject or distrust when what they receive is incoherent.

The supervisors are followers watching for heartbeats and external monitors watching cluster health. Both expect regular signals. Both intervene when silence replaces signal.

The dependents are any systems building on the leader election guarantee. Distributed locks depending on "exactly one leader." Replicated databases depending on "one primary accepts writes." When two nodes both act as leader, these systems cascade failure outward.

The taxonomy applies where there are no users. It classifies how any entity relates to any system.

Time, wrongness, and pressure manifest differently for each mode.

Same system. Four meanings for each dimension.

The observer question precedes everything because it determines what time, wrongness, and pressure mean. You cannot design for "low latency" without knowing whose latency. You cannot design for "acceptable failure" without knowing acceptable to whom.

Evaluating Trade Offs

There is no such thing as a neutral system design decision.

Every choice privileges one observer and taxes another. You cannot satisfy all equally. You can only choose who pays and whether you know you're choosing.

An initiator places an order and needs to know it counted. The system can acknowledge before processing completes, or wait until processing finishes.

Acknowledge early and initiators get certainty. But when acknowledgment is slow, they retry. Two orders now exist. The recipient sees both, each valid-looking, with no way to tell which represents intent.

Wait for processing to complete and acknowledgment slows. Initiators retry anyway, or abandon.

Both paths tax someone. Early acknowledgment: recipients pay through duplicates. Late acknowledgment: initiators pay through uncertainty.

If you know recipients pay, you build deduplication. Idempotency keys. Reconciliation workflows. The cost is absorbed by design.

If you don't know, you discover the cost through double charges, manual order merging, and refund spikes.

A warehouse needs to know order state to begin work. The system can show state as it changes, or wait until state is settled.

Show state early and recipients can act quickly. But dependents also read this state. An analytics pipeline computes revenue from orders marked "shipped." When state reverts due to an error, the recipient adapts. The dependent's computation is already wrong.

Wait until settled and recipients are delayed. Preparation starts late.

Both paths tax someone. Early state: dependents pay through instability. Late state: recipients pay through delay.

If you know dependents pay, you provide event logs that preserve history alongside current state. Dependents build on stable records while recipients work with current reality.

If you don't know, you discover the cost through mismatched reports, failed audits, and questions about why yesterday's numbers changed.

An initiator places an order. The system confirms immediately and queues it for processing. The initiator leaves satisfied.

The queue backs up. The processor stalls. The order sits unprocessed.

The initiator believes their order is being handled because they received confirmation. The supervisor sees no alerts because the system didn't fail—it just stopped making progress. The order exists for the initiator. It doesn't exist for the supervisor.

Days later, the customer calls asking where their order is.

If you know supervisors pay when confirmation is decoupled from processing, you monitor queue depth. You alert on orders that sit too long.

You build visibility into the gap between "confirmed" and "processed."

If you don't know, you discover the cost through support calls about orders that vanished.

The system is under heavy load. Orders keep arriving. The system keeps accepting them.

Initiators see a responsive system. They submit orders and receive acknowledgment. From their view, everything works.

Behind acceptance, processing capacity is fixed. Orders enter faster than they complete. A queue grows. Nothing signals this to initiators or supervisors.

The queue grows until processing delays become visible to customers. By then, thousands of orders are backlogged.

If you know supervisors pay when the system accepts without signaling capacity, you build backpressure. You slow acceptance when queues grow.

You surface capacity limits before they become crises.

If you don't know, you discover the cost through sudden mass delays and emergency shutdowns.

An order has an error. A support agent fixes it by editing the record directly. The order now shows correct information. The immediate problem is solved.

An analytics pipeline ran last night. It computed revenue from the original record. Today, someone queries the same data. The numbers don't match. Yesterday's shipped orders total differs from today's.

The support agent saw a problem and fixed it. The dependent assumed records don't change after the fact. Both acted reasonably within their own frame.

If you know dependents pay when records can be edited, you preserve history. Corrections become new events rather than overwrites.

The current state reflects the fix. The historical record shows what happened and when it changed.

If you don't know, you discover the cost through reconciliation failures and auditors asking why the numbers keep shifting.

The archetypes predict these collisions.

For any design choice, ask: which archetype gains? Then ask: which archetype pays?

Then ask: is the cost absorbed by design, or will it surface as failure?

You cannot eliminate these tensions. You can see them early enough to choose consciously.

Conscious tradeoffs come with mitigation. Unconscious tradeoffs come with surprises.

The next question is how each mode experiences time, and why temporal misalignment is where invisible failures hide.