A measured retrospective on twelve internal AI experiments. Three shipped, two are still running, the rest got a quiet funeral.
In 2025, my team ran twelve AI experiments. Three shipped to production. Two are still running as internal tools we depend on. Seven got cancelled — some early, some after significant investment. This is a retrospective on what made the difference.
I am writing this in December, with the benefit of a full year of data. The observations are specific to our context — an enterprise software environment with regulated customers, a platform team mandate, and a bias toward shipping incrementally rather than betting on large transformations.
The three that shipped
Automated QA triage. We trained a classifier to route incoming QA reports to the right engineering team without human triage. The model is not perfect — accuracy is around 87% — but 87% automated routing against a baseline of 100% manual is a meaningful efficiency gain. The key to this one was a clear, measurable definition of success before we started. We knew what good looked like, we could measure it, and we could improve against the measurement.
Release notes generation. We built a tool that drafts release notes from git commit messages and linked tickets. An engineer reviews and edits the draft before publishing. The tool does not replace judgment — it replaces the blank page. Time savings are approximately 40 minutes per release cycle. Adoption was near-instant because the output was immediately useful and the workflow fit naturally into what engineers were already doing.
Internal knowledge retrieval. We built a RAG system over our internal documentation — runbooks, architecture docs, ADRs. Engineers can ask questions in natural language and get answers with source citations. This is still the most-used internal tool we have shipped. It gets approximately 200 queries per week. The maintenance cost is low because the underlying documents are already being maintained.
The two still running
Both are internal tools in active use that have not been through a formal "ship" decision — they are just things people depend on. I count these as successful experiments that have not yet completed their arc.
One is a prompt testing harness that the AI platform team uses for evaluating prompt changes before they go to production. The other is a latency profiler for agent traces that started as a weekend project and became something three teams now rely on.
The seven that did not ship
The meeting summarizer. This felt like an obvious win. Transcribe meetings, generate summaries, save everyone time. It failed in practice because the summaries were accurate but not useful — they captured what was said, not what was decided or what the follow-up actions were. The distinction between a summary and a useful summary is significant, and we underestimated it. We cancelled after six weeks.
The customer intent classifier. We wanted to classify inbound support tickets by underlying intent to improve routing beyond simple keyword matching. The model performed well in testing. In production, it had a significant false positive rate on an edge case category that was underrepresented in our training data. By the time we identified and corrected for it, the team that had sponsored the project had moved to a different priority. Cancelled at week ten.
The code review assistant. This one was cancelled early — week three — when we discovered that the developers who would have benefited most from automated code review were the developers least likely to trust it. The adoption problem was not technical. We were solving a problem that the people who had the problem did not want solved in this particular way.
The remaining four had variations of the same failure mode: technically functional, practically not adopted. Either the workflow integration was too awkward, the output quality was insufficient for the specific use case, or the people who would have used the tool found workarounds that were good enough.
What the successful experiments had in common
Looking at the five experiments that worked, three patterns stand out.
A measurable baseline. In each case, we knew what we were replacing, how long it took, and what the error rate was. "Better than what we have now" is not a measurable baseline. "87% vs. 100% manual routing" is.
A natural workflow fit. The tools that got adopted fit into workflows that already existed. Release notes are already written; we made the writing faster. Engineers already look things up in documentation; we made the lookup better. We did not ask people to change their behavior — we made their existing behavior less effortful.
A narrow scope. None of the five successful experiments were trying to solve a broad problem. They were trying to solve one specific, defined problem. The experiments that failed often started with a broader ambition and tried to narrow it during execution. Narrowing scope under pressure is harder than starting narrow.
What I would carry forward
Small bets at high frequency beat large bets at low frequency in this domain. The cost of a failed six-week experiment is a team's attention for six weeks. The cost of a failed six-month project is much higher, in direct time and in opportunity cost.
Define success before you start. If you cannot write down what success looks like in a way that a neutral observer could evaluate, you are not ready to run the experiment.
Adoption is a feature, not an afterthought. A tool that works perfectly and nobody uses is a failed experiment. Ask the people who would use it whether they would use it, before you build it.
The funeral for a cancelled experiment should be quick and undramatic. Seven experiments cancelled in a year is not a problem. It is a sign that you are running experiments at an appropriate rate and cutting losses appropriately.
