Subscribe for more posts like this →

Every system has more users than you think

Observers

When you build a feature, you think about users.

The person clicking the button. The customer reading the dashboard. The admin running the report.

But there’s a category of “user” that doesn’t appear in your requirements, doesn’t show up in your analytics, and never files a support ticket — until the day it does, and by then the damage is already done.

A monitoring agent that expects your service to respond within 10 seconds. A downstream pipeline that expects your data schema to stay stable. A compliance system that expects your logs to be complete. A mobile client that expects the record it cached this morning to still mean the same thing tonight.

None of these are users in the traditional sense. But they all have one thing in common with your users:

They formed an expectation about your system. And they will react when you violate it.

This is what an observer is.

Not just a human. Not just someone with a login. Anything — human or machine — that formed an expectation about your system and will change its behavior when that expectation isn’t met.

The health check that pages your on-call engineer is an observer. The fraud detection model that learned what “normal” looks like is an observer. The partner API consuming your webhook is an observer. The SLA your sales team promised a customer is an observer with a lawyer.

Most systems are designed for the obvious observers — the ones in the requirements doc. They discover the hidden ones through incidents.

The cache that served stale data to a downstream service. The schema change that silently broke three pipelines. The deletion that removed a record six systems were still relying on.

The incident wasn’t caused by bad code. It was caused by an observer nobody had mapped.

Writing software is easy when you know who’s watching. The hard part is that most observers never introduced themselves.


Fault Lines — byte-size system design, one concept at a time.

Read more