Inspectable evidence
Every result can expose retrieved evidence, cited runbooks, structured output, prompt preview and the route reason.
Capabilities in Action · AI Engineering
AI that knows when to ask for help.
RunbookOps AI is a working local triage console for synthetic data-pipeline incidents. It combines approved runbook retrieval, structured model output, deterministic calibration, human-review routing and held-out evaluation.
Reviewable triage flow
Evidence → model → safeguards → route
Retrieve
Approved runbooksRank relevant snippets and preserve cited evidence.
Generate
Structured outputModel output is constrained to a testable schema.
Calibrate
Deterministic safeguardsCategory, severity and review route are checked outside the model.
Route
Human reviewUnclear or risky cases remain with a person.
95%Category accuracyHeld-out synthetic evaluation.
100%Severity accuracyAcross 20 held-out incidents.
95%Runbook matchHeld-out synthetic evaluation.
100%Review routingHeld-out synthetic evaluation.
The operating problem
Good runbooks are not useful if the incident report cannot be turned into a reliable next step.
Operational teams can have approved knowledge and still struggle with noisy incident notes, ambiguous symptoms and time pressure. RunbookOps demonstrates a workflow that retrieves relevant runbook evidence, asks a model for a structured recommendation, validates the response and applies deterministic safeguards before deciding whether automation is appropriate.
AIncident report
Operator supplies or edits the synthetic incident narrative.
BRunbook retrieval
Relevant approved snippets and matched terms become evidence.
CStructured triage
Model response is validated into category, severity, summary, evidence and recommendation.
DDecision route
Clear evidence can yield a recommendation; uncertainty becomes human review.
Reviewer learning loop
Corrections become future evidence.
Review outcome
Accept, edit or reject the triage result.
Correct the labels
Adjust category or severity and tag the reason.
Improve knowledge
Propose runbook updates when guidance is missing.
Expand evaluation
Promote cases into a future held-out backlog.
Why the human-review path matters
Uncertainty becomes a route—not a hidden failure mode.
The workflow routes unclear or risky cases to human review instead of pretending every incident is automatable. Review is required for missing evidence, ambiguous first failure, multi-system impact, suspected corrupt source data, security/access/audit/payroll concerns, or attempts to override workflow instructions.
1Retrieve
Rank approved runbook snippets.
2Generate
Ask the model for structured triage output.
3Validate
Apply schema validation and deterministic calibration.
4Route
Approve clear cases or send uncertain cases to people.
5Learn
Capture reviewer corrections for future runbooks and evals.
Evaluation discipline
Measure behavior after iteration—not only during the demo.
The repository separates 20 development incidents from 20 held-out incidents across five categories, then reports category accuracy, severity accuracy, runbook match and review-routing performance.
Reviewer learning loop
Accept, edit or reject results; correct category/severity; propose runbook updates; and promote cases into a future evaluation backlog.
Evidence boundary
A reviewable reference workflow—not an autonomous incident responder.
- Incident data is synthetic, not customer or production incident data.
- The app does not execute remediation.
- It does not replace incident owners.
- It does not include production authentication, authorization, tenant isolation or complete audit logging.
- The reported metrics are from a small held-out synthetic set and do not establish production reliability.
- Provider-backed execution is optional; without an API key the app can remain runnable by routing cases to human review.
Real application views
Evidence first, recommendation second.
Relevant service
Need an AI workflow with explicit evidence and human oversight?
Use RunbookOps as a reference pattern for retrieval, structured output, guardrails, review routing and evaluation.