The instinct, when a system goes wrong in a way no one saw coming, is to add more monitoring. More dashboards, more charts, more metrics. The result is often a wall of graphs that nobody reads and that still does not answer the question that matters when something breaks: what is actually happening, and to whom.
Good instrumentation is not about volume. It is about whether the data you collect can answer the questions you will actually ask, usually under pressure, usually at the worst possible time. Most instrumentation fails that test not because there is too little of it, but because it measures what is easy to measure rather than what is worth knowing.
Instrument questions, not activity
The useful test for any metric is simple: what decision does it inform? A number that no one would act on differently regardless of its value is decoration, however interesting it looks on a chart.
Vanity metrics are the clearest example. Total page views, cumulative sign-ups, lifetime requests served: numbers that rise reassuringly and tell you almost nothing about whether the system is working for the people using it right now. The metrics that earn their place answer operational questions. Is this workflow completing? How long are people waiting? Where are they failing to get what they came for?
Track decisions and failures, not just events
The most valuable thing to instrument is rarely the happy path. It is the points where things go wrong, or where the system makes a choice.
When something fails, capture it with enough context to act on. Knowing that errors rose is of limited use; knowing which operation failed, for which kind of input, and what the system did next is what lets someone fix it. Record the decisions the system makes, too, especially the automated ones. When a process branches, retries, or rejects something, the reasoning behind it is what you will need when the outcome is questioned later.
And measure what the user experiences, not what the server reports. A request the backend counts as successful can still be a slow, broken experience for the person waiting on it.
Structure events so they can be asked questions
A log line written for a human to read in the moment is hard to query in aggregate later. Capture events as structured data instead, with consistent fields for the entities and actions involved, and you can filter, group, and correlate them after the fact, which is when most real questions arrive.
The discipline that makes this pay off is consistency: the same identifier for the same thing across services, so you can follow a single user’s journey or a single transaction end to end rather than reconstruct it from fragments. Instrumentation you cannot join together answers only the narrowest questions.
Alert on symptoms, not causes
A system can emit hundreds of warning conditions, most of which never affect anyone. Alerting on all of them trains the team to ignore alerts, and the one that matters arrives in the same flood as the ones that do not.
The alerts worth waking someone for are the ones tied to something a user is feeling: requests failing, latency past the point of usability, a workflow that has stopped completing. Internal causes are useful for diagnosis once you are already looking; they are a poor trigger for looking in the first place.
Instrumentation, done this way, is less about collecting more and more about collecting deliberately: the same restraint that keeps a data warehouse from sprawling and that any system needs before its data is ready for AI. Building the visibility that lets a business see what its systems are actually doing is part of what our data solutions practice does.
