Notes from decommissioning a system I designed in 2020. The parts that aged well, the parts that did not, and the line between the two.
In Q1 of this year, we finished decommissioning a service I designed in 2020. Six years of production traffic. Somewhere in the range of 400 million requests over its lifetime. At peak it was handling 12,000 requests per minute with a p95 latency under 80ms.
Decommissioning a system you designed is a particular kind of feedback loop. The decisions you made under uncertainty are now visible in the record. Some held up. Some did not.
What aged well
The interface contract. The API contract I defined in 2020 was conservative. I resisted adding fields that were "probably useful" in favor of fields that were definitely needed. That restraint meant that when consumers of the service needed to upgrade, the migration surface was small. We had one breaking change in six years, and it was well-telegraphed.
The operational runbook. The original runbook was detailed to the point where it felt excessive at the time. Incident response procedures, rollback steps, known failure modes and their mitigations. That runbook was updated dozens of times over the life of the service, but the structure held. Every engineer who ran an incident on that service could navigate it without asking someone who had been there from the start.
The decision to not abstract prematurely. In 2020, there was a push to build a more generic version of the service that could handle a wider class of problems. I pushed back and built something narrowly scoped to the problem we actually had. The generic version would have been more complex, harder to test, and harder to reason about under load. The narrow version did one thing well for six years.
What did not age well
The data model. The original schema made assumptions about the shape of the data that held for about two years. After that, as the business evolved, we spent a significant amount of engineering time working around those assumptions rather than changing them. Changing the schema on a live service with that traffic volume is expensive. The lesson is not that I should have predicted the future — I could not have. The lesson is that I should have been more explicit about which assumptions the data model was making, so that when those assumptions became invalid, the path to changing them was clearer.
The monitoring. I instrumented the happy path well. I instrumented error rates and latency. What I did not instrument was the business logic — the downstream outcomes that indicated whether the service was doing what it was supposed to do, not just whether it was running. We found several cases over the years where the service was technically healthy by infrastructure metrics while producing subtly incorrect outputs. Those were caught by users, not by monitoring.
The dependency on a vendor that changed its pricing model. In 2020, one of our upstream dependencies was priced in a way that made the architecture economical. By 2023, that vendor had changed its pricing structure and the same traffic cost approximately 3x more. We adapted, but it required an architectural change we had not planned for. I do not have a clean lesson here — vendor pricing changes are hard to predict. What I should have done is modeled the cost sensitivity more explicitly at design time, so we understood how exposed we were to that kind of change.
The line between the two
Looking at what aged well versus what did not, the pattern I see is this: the things that aged well were things I was explicit about. The API contract was written down. The runbook was written down. The decision to build narrow instead of generic was written down in the design doc, with the reasoning.
The things that did not age well were things I was implicit about. The data model assumptions were not documented — they were just the shape of the schema. The monitoring gaps were not visible because I had not written down what I expected the monitoring to cover. The vendor cost sensitivity was not modeled because I assumed the pricing structure was stable.
Explicit decisions can be revisited and updated as circumstances change. Implicit decisions become invisible constraints that are only discovered when they cause problems.
What I would do differently
I would write down the assumptions. Not just the decisions — the assumptions the decisions depend on. "We are choosing this data model because we expect X. If X turns out to be false, this model will need to change in Y way." That kind of documentation is almost never written. It would have saved a meaningful amount of engineering time on this service.
I would instrument the business logic from the start, not just the infrastructure. The question "is the service running" is easy to answer. The question "is the service doing what it is supposed to do" requires different instrumentation and different alerting.
The service ran for six years and handled its job. That is a reasonable outcome. The decommissioning took less than a quarter with no incidents. That is also a reasonable outcome. The lessons from it are not catastrophic failures — they are the incremental improvements I would carry into the next design.
