Two working projects: a practical blueprint for reviewable recommendations and explainable model spend
Overview
A data import fails. The dashboard it feeds is now out of date. You have an error message, a few logs, and several runbooks that might explain what happened.
Before you can fix anything, you need to work out where to start.
That’s the problem I’ve been exploring with RunbookOps AI and its companion LLM Cost and Evaluation Gateway, which I call CostOps Gateway.
RunbookOps helps turn an incident report into a recommendation someone can review. The gateway handles the model calls, budget checks, and usage records behind it. Together, they form a local solution framework for exploring AI-assisted incident triage without losing sight of the evidence—or the bill.
Both projects are available on GitHub if you’d like to explore the code, try the demos, or adapt the approach:
What happens when an incident comes in?
Imagine an overnight import stops after a supplier adds an unexpected column. No new records reach the reporting tables, and the morning dashboard is waiting on that load.
A useful triage tool should help the reviewer connect the failure to the relevant guidance and decide what to investigate next.
RunbookOps starts by retrieving relevant passages from approved local runbooks. It sends that evidence along with the incident report to the model, then checks that the response contains the expected fields.
The result includes the incident category, severity, supporting evidence, recommended next step, and whether human review is needed. The console also exposes the retrieved material and the reason for the review decision.
The person reviewing it can inspect how the recommendation was assembled. The application does not carry out the repair.
For this project, all incidents are synthetic. That lets me test failures and edge cases without using customer or production incident data.
Why build a separate gateway?
I wanted the triage application to stay focused on the incident. Model selection, spending limits, retries, and usage accounting have their own concerns—and could be useful to other applications too.
That became the gateway’s job.
Before calling a provider, it selects a route, estimates the bounded cost, and reserves budget. It also manages bounded retries and records usage in a local SQLite ledger. After execution, it reconciles the reservation with the available usage information.
RunbookOps can call the gateway over HTTP. The CostOps dashboard then provides a view of the recorded usage and estimated costs.
This separation makes it easier to investigate both sides of the workflow: the recommendation someone received and the execution that produced it.
Five lessons I’ll take into the next project
1. A good recommendation needs evidence people can inspect
A clear answer is helpful. A clear answer with relevant runbook evidence is easier to evaluate.
That’s why the console exposes retrieved passages, cited runbooks, and routing reasons. Reviewers need enough context to challenge a recommendation when something doesn’t fit.
There’s still more work to do here. Checking that a response has the right fields doesn’t establish that its explanation is correct. NIST’s Generative AI Profile discusses how generated content can sound confident while being wrong. Testing whether the cited evidence actually supports the recommendation is an important next step.
2. Human review belongs in the workflow from the beginning
Some incidents are too ambiguous to handle automatically. Others involve missing evidence, potentially corrupt source data, or sensitive access and audit concerns.
For those cases, the system needs to explain why it is asking for help and give the reviewer something useful to work with.
I’ve come to see appropriate escalation as a result worth measuring. A system that correctly recognizes uncertainty can save someone from following an unsupported recommendation.
3. Test on examples you didn’t use to build the system
It’s easy to become familiar with your own demo cases.
The project separates 20 development incidents from 20 held-out synthetic incidents across five runbook categories. The published evaluation reports:
- Incident category accuracy: 95%
- Runbook match rate: 95%
- Severity accuracy: 100%
- Review-routing accuracy: 100%
For category prediction, that means 19 correct cases out of 20. The average response time was approximately 8.7 seconds.
Those results give me a baseline to improve. Twenty synthetic cases are a small test set, and the case-level results are more useful than treating the percentages as proof of production reliability.
4. Cost comparisons need their assumptions attached
The gateway includes a synthetic scenario representing roughly 12,000 incident-triage requests. It compares routed execution with a baseline that uses the stronger model for every automatable request.
- Strong-model-only baseline: $1,593.62
- Routed execution: $671.73
- Estimated difference: $921.89
- Percentage difference: 57.8%
Projection based on local pricing assumptions.
It also leaves out human-review labor. Sending more cases to people may reduce model spending while adding work elsewhere.
The quality evaluation and cost projection are separate exercises, too. The projection does not establish that every routing policy delivers equivalent results.
For a real pilot, I would want to compare total cost per accepted recommendation, including retries, unsuccessful attempts, review effort, and operating overhead.
5. Reviewer feedback needs somewhere to go
RunbookOps lets a reviewer accept, edit, or reject a result, correct its labels, and propose a runbook update.
Those decisions are saved locally and can feed a backlog of improvements and future evaluation cases.
That gives a correction a purpose beyond the incident in front of the reviewer. It can help identify missing guidance or expose a behavior that should be tested before the next release.
The feedback does not automatically retrain the model or change the live workflow.
Why spend governance matters here
AI spending is becoming a routine responsibility for FinOps teams. The FinOps Foundation’s 2026 report says 98% of respondents manage AI spend, compared with 63% in 2025 and 31% in 2024.
These are community-survey results, with different respondent groups each year—not a measure of all businesses. Still, they provide useful context for building cost visibility into an AI workflow early.
The Foundation’s principle, “Business value drives technology decisions,” is a useful test for this project. I want to understand whether a recommendation helps someone reach an acceptable decision, how long that takes, and what it costs to get there.
Want to try the approach?
Start locally with the synthetic incidents and mock providers.
Pick a sample incident, run triage, and inspect the recommendation alongside its evidence. Then try an over-budget request or a provider failure and see how the application handles it. Save a reviewer correction and follow it into the learning backlog.
The setup instructions are in the RunbookOps README and gateway README.
For a later pilot, I’d choose one low-risk queue and use authorized historical incidents. The system could suggest recommendations while staff retain responsibility for decisions. That would let the team compare usefulness, review effort, response time, and cost before expanding its role.
What’s under the hood?
Both projects use Python. RunbookOps uses Streamlit for the console, Pydantic for structured responses, and the OpenAI Responses API for optional provider-backed analysis. Its runbooks are Markdown files, and incident and reviewer records use JSONL.
The gateway adds FastAPI and Uvicorn for the service interface, SQLite for usage records, YAML routing policies, and a Streamlit cost dashboard. Both projects use pytest and Ruff.
They are working local applications. A hosted production version would need additional identity controls, tenant isolation, shared durable storage, monitoring, and billing reconciliation.
Where I’d take it next
My priorities are to broaden the evaluation set, test whether recommendations are supported by their citations, and measure the full cost of reaching an accepted result.
I’d also strengthen change approval so that updates to prompts, runbooks, models, and routing policies come with a versioned evaluation record and a rollback option.
If you’re exploring a similar workflow, the code is there to inspect and build on. I’d be interested in what you would change—for example, setting a rule that all incidents of a particular type will always route to a human reviewer, etc.