RunLore: the SRE buddy that investigates incidents — and learns from every fix

An alert fires. And the diagnosis begins: digging through recent changes in Git, Grafana dashboards, logs, maybe a runbook, cross-checking clues until a first hypothesis takes shape.

This investigation phase sits at the heart of SRE (Site Reliability Engineering) work. It runs on deep familiarity with the platform: knowing where to look for the right signals. Which is why the time it takes to trace a cause varies so much with the seniority of whoever picks it up. And the knowledge built up along the way? It's usually gone the moment the incident is closed.

I decided to take this on a few weeks ago by starting an open source project: RunLore. The initial idea was simple: a buddy that takes the data-gathering off your hands and points the investigation at the most likely root cause, fast. And one that, over time, learns the context we work in together: our platform, our constraints, our rules and standards.

To hold on to that knowledge, I picked OKF (Open Knowledge Format), a format designed precisely so an AI agent can read and write its knowledge efficiently. I'll come back to it in detail further down.

🎯 Goals

• 🤖 Get to know RunLore and how it works
• 🧠 Its three design choices: a single binary, the learning loop, and the human who makes the call
• 🛠️ Deploy it with Helm and watch it investigate a real incident on cloud-native-ref

🔥 Why this project: the hidden cost of investigation

The previous post on alerts ended on an obvious point: technical configuration alone isn't enough, you also need runbooks and incident response procedures. In real life, though, those runbooks are incomplete, and much of that knowledge only ever lives in a few people's heads.

When an alert fires, you replay the same manual sequence every single time:

• What changed? A deployment, a version bump, an infrastructure change, an expired certificate?
• What's wrong? Saturation, network, nodes, failing dependencies?
• Have we seen this before? The answer is usually "yes", but nobody can find where it was written down.

Correlating all these sources (Git, metrics, logs, network flows) to reach a diagnosis is exactly the kind of task an agent can take on. Several already do. But the part I care about is different: keeping a human in the loop on the decisions, on what the agent does as much as on what it learns.

🤖 RunLore in a nutshell

RunLore is an open source SRE agent (Apache-2.0 licensed) that runs inside your Kubernetes cluster as a single Go binary, deployed with Helm.
The idea is deliberately simple. An event comes in: an alert, a GitOps failure (continuous delivery driven from Git), and so on. RunLore investigates, answers two questions — what changed? and what's wrong? — then posts a root cause to your chat, with a confidence score, the evidence backing it, and the open questions it leaves to a human.

The diagram breaks down into three stages:

• Triggers — an alert, a GitOps failure or a webhook (e.g. PagerDuty) kicks off the investigation.
• Data sources — the GitOps history (the backbone of what-changed), metrics, logs, network flows, the cloud… the more you plug in, the stronger the answer.
• Notification channels — the verdict goes out to Slack, Matrix…

And all of it sits on top of a knowledge base that grows with every incident.

Three design choices

Agents that investigate incidents already exist, and we'll get back to them. What sets RunLore apart comes down to three choices, and it's their combination that matters:

1. 📦 A single binary, on your infra, running your models. One Go binary in your cluster. You keep control of your data and your models, and with a self-hosted LLM nothing leaves your perimeter. No SaaS, no lock-in.
2. 🧠 A memory that compounds. Every investigation feeds a knowledge catalog you own, in the OKF standard. The same incident, the next time around, gets an instant answer instead of a fresh investigation.
3. ✋ The human stays in charge. The agent reads and recommends, it never acts on its own (read-only → suggest → approve). And above all, nothing enters its memory without a PR reviewed by a human. When it doesn't know something, it hands the question back instead of guessing.

The next two sections dig into the two least obvious ones: the memory, then where the human fits in.

🧠 A memory that compounds

The autonomous alert → root cause → chat loop has become commonplace: plenty of tools do it. What's far less common is a knowledge base that consolidates into a catalog you control.

The learning loop

RunLore implements a four-stage cycle:

1. Retrieve — faced with an incident, it first looks for a trustworthy past answer in the knowledge base.
2. Capture — if nothing matches, it investigates live, records what it found, then watches what follows: did the incident actually resolve?
3. Curate — a finding that is both reliable and new enough becomes a Pull Request opened against your knowledge repo. This is the quality gate: an entry is reviewed the way code is reviewed.
4. Compound — once the PR is merged, the entry is re-indexed and immediately available. The same incident, next time, is recognized and served from the catalog, with no second investigation.

So it all hinges on Retrieve, which decides between two outcomes: serve a known answer instantly, or, with no trustworthy match, run the full loop — the one that feeds the catalog.

What makes this cycle genuine learning, rather than a notepad, is that outcomes loop back. RunLore keeps an outcome ledger: every time an entry is recalled, it records whether the incident actually resolved afterwards. Confidence is therefore derived from the observed resolution rate, not asserted by the model:

• an entry that resolves consistently → it gains confidence;
• an entry recalled without the incident ever resolving → it decays and stops being offered, until a new investigation corrects it.

The memory isn't frozen: it stays anchored to operational reality.

That same score has a human half: an optional pair of 👍 / 👎 buttons under every notification on Slack and Matrix. The click isn't cosmetic. One tap moves two levers:

• The quality of the diagnosis — a 👍 confirms it, a 👎 declares it wrong. The vote feeds the entry's confidence, not its content. And it is sometimes the only judge available: some triggers, a GitOps failure for instance, never emit a resolution signal.
• When the agent is allowed to repeat itself — a 👎 that sticks triggers a fresh investigation immediately, instead of re-serving a contested answer.

OKF: from a Karpathy idea to a standard

Google Cloud recently published a portable standard for what's known as the LLM-wiki: OKF (Open Knowledge Format, June 2026). The concept itself, though, is anything but new.

The idea comes from Andrej Karpathy: rather than re-deriving knowledge from raw documents on every query, you hand the model a tree of markdown files that it reads, extends and maintains like code. A format designed to be read efficiently by AI, while staying plain text, versioned and reviewable by a human.

I was already using this pattern locally: first in Obsidian, then through Tolaria, the tooling I built on top of it so an agent could read and write in my knowledge base. Making it RunLore's memory felt like the natural move.

So the memory lives in a Git repo you own: BM25-indexed, reviewed through PRs, fully traceable. And you don't have to start from a blank page. The catalog can be seeded on day one (constraints, architecture, team conventions), and incidents build on it from there.

✋ The human stays in charge

RunLore takes a deliberate stance: keep a human at the center of the decisions. LLM answers are still debatable often enough that chasing full automation is a trap. Here, you stay firmly in control:

• of how the knowledge base evolves — nothing enters it without your approval. Every reliable finding becomes a Pull Request that you review and merge. The agent proposes; the team decides what joins its memory.
• of judging diagnosis quality — you're the one who confirms or overrules it (👍 / 👎), and the agent is calibrated never to oversell: an honest unresolved when it doesn't know, and a confidence that can only go down at verification.
• of the domain knowledge that goes in — you seed and enrich the catalog with your context: company rules, team conventions, habits, platform specifics.

And the agent never acts alone. The posture is read-only → suggest → approve: it reads, correlates, recommends; a human validates.

Keeping a human in the loop, on what gets done as much as on what gets learned, isn't a self-imposed limit. It's what makes the agent usable at all.

👀 Here's what it looks like

Enough theory, here is RunLore at work. The incident below is a real one, investigated on cloud-native-ref, my reference repo: a complete platform on EKS combining Cilium, VictoriaMetrics, Crossplane and Flux.

🔍 The incident

A HarborRegistryDown alert fires. RunLore investigates and correlates several signals:

• the pod status: harbor-registry is failing with CreateContainerConfigError — couldn't find key username in Secret tooling/xplane-harbor-access-key;
• the Kubernetes events in the tooling namespace: a persistent Warning on the Crossplane resource AccessKey/xplane-harbor — LimitExceeded: Cannot exceed quota for AccessKeysPerUser: 2;
• the causal link: that failing Crossplane resource is the one meant to create the Secret the Harbor pod consumes.

The agent then identifies the root cause with high confidence: the Crossplane resource has hit the AWS IAM AccessKeysPerUser: 2 quota, which prevents it from creating the registry's credentials. It posts all of this to Slack, suggests a remediation (delete an unused access key, flagged reversible=false), and, importantly, lists what it doesn't know (the data gaps).

The notification is verdict-first: it leads with a clear actionability verdict — no action, action suggested, action required or inconclusive — before any of the details. You know at a glance whether you need to step in.

Once the incident is reviewed and the PR merged, here is the OKF entry that joins the catalog:

 1# file: harbor-registry-down-due-to-iam-access-key-quota-limit.md
 2---
 3type: Incident
 4title: Harbor Registry Down due to IAM Access Key Quota Limit
 5resource: tooling/harbor-registry
 6tags: [runlore, incident]
 7fingerprint: 2d6bd8279304b3e17a5d5e35a55fb0c115ffbeabde820af8cdd2494a4141a60b
 8---
 9
10## Decision
11- **why keep:** The Crossplane resource `AccessKey/xplane-harbor` has hit an AWS
12  IAM quota limit (`AccessKeysPerUser: 2`), preventing it from creating the
13  credentials required by the Harbor registry.
14- **confidence:** 95%
15
16## Investigate
17- pod_status shows harbor-registry failing with 'CreateContainerConfigError:
18  couldn't find key username in Secret tooling/xplane-harbor-access-key'
19- kube_events shows a persistent Warning on 'AccessKey/xplane-harbor':
20  'LimitExceeded: Cannot exceed quota for AccessKeysPerUser: 2'
21- The knowledge base article 'HarborRegistryDown' describes this exact scenario.
22
23## Resolution
24- An administrator should delete an old or unused access key, then reconcile
25  the `AccessKey/xplane-harbor` resource. (reversible=false)
26
27## Unresolved
28- The name of the specific IAM user that has reached its quota.
29- Which of the two existing access keys is safe to delete.

The Unresolved section shows that principle in action: the agent hands back anything that needs access or judgment it doesn't have. Here, which of the two access keys to delete.

⚡ Recurrence: an instant answer

Some time later, the same incident happens again. This time RunLore runs no investigation at all: it recognizes the incident, pulls the already validated answer out of the catalog (cause and resolution) and posts it to Slack instantly.

And it's radically cheaper: ~58,000 tokens for the first investigation, ~3,700 for this recall. Same answer, in a matter of seconds.

🔬 The alternatives

RunLore isn't landing in an empty field: RCA (Root Cause Analysis, automatically finding the root cause) is already crowded. Here's how I position it against what exists, starting with the closest:

The table is far from exhaustive: Aurora (open source, multi-cloud) plays in the same space, and a whole wave of SaaS agents (Datadog Bits AI SRE, Cleric, Resolve.ai…) goes after the same need, but as closed products. That puts them outside the ground I care about here: open and self-hostable.

Change-oriented RCA is clearly nothing new: commercial tools have been computing change diffs for a long time. The space RunLore actually occupies is the combination that open tools don't offer: that same change signal feeding an open, portable, reviewed catalog, operated by an agent that leaves the call to a human.

🛠️ Try it yourself

Want to try it? Installation is a Helm chart and a values.yaml. You'll need at least one data source, an LLM, a private GitHub repo for the knowledge base (with a dedicated GitHub App) and a notification target. Credentials go into a Kubernetes Secret; the values.yaml does the wiring.

 1# values.yaml (excerpt) — a standard example: Argo CD + Prometheus + VictoriaLogs + Slack
 2config:
 3  gitops:
 4    engine: argocd                   # or "flux"
 5  sources:
 6    alertmanager: {}                 # Alertmanager webhook
 7    gitops:
 8      enabled: true                  # also reacts to Argo CD failures
 9  model:                             # any OpenAI-compatible endpoint
10    provider: openai
11    model: glm-5.2
12    base_url: https://api.z.ai/api/paas/v4/
13    api_key_env: GLM_API_KEY
14  metrics:
15    url: http://kube-prometheus-stack-prometheus.monitoring.svc:9090   # Prometheus
16  logs:
17    url: http://victoria-logs-single-server.observability.svc:9428     # VictoriaLogs
18  cloud:
19    provider: aws
20    region: eu-west-3
21  catalog:
22    instant_recall: {enabled: true}  # instant recall from the KB — off by default
23  notify:
24    slack:                           # bot token: threaded detail + 👍/👎 buttons
25      bot_token_env: SLACK_BOT_TOKEN
26      signing_secret_env: SLACK_SIGNING_SECRET
27      channel: "#alerts"
28      feedback_buttons: true
29  forge:
30    kb_repo: your-org/runlore-kb     # the repo that receives the PRs that grow the knowledge base

1helm install runlore deploy/helm/runlore -n runlore --create-namespace -f values.yaml

All that's left is to route Alertmanager to http://runlore.runlore.svc:8080/webhook/alertmanager, and investigations start. I'm deliberately sticking to the essentials here: the complete getting started guide covers creating the GitHub App, the full values.yaml reference and the verification steps.

The example above is a standard stack, but you can plug in whatever you want: GitOps (Flux/Argo CD), metrics, logs, network flows, cloud, several LLMs and notifiers, each one pluggable. The full matrix, which evolves with each release, lives in the repo's README.

📊 Analyzing the agent's behavior

An agent that investigates your incidents has to be observable itself. RunLore exposes Prometheus metrics and provides a Grafana dashboard (in the repo, to import into your instance), organized around the learning loop, not just around technical health. Its first row answers a single question, "is the loop working?":

• Recall fire-rate & precision — how often it fires, and whether it picks the right entry.
• Resolution rate — the key signal: do the recalled answers really resolve the incident?
• Tokens saved by recall — the concrete gain against a full investigation.
• Invalid KB entries — knowledge that is degrading, worth keeping an eye on.

Many other indicators follow: cost (tokens), performance, health and errors.

It's also what will eventually settle the question this post closes on: does this memory actually pay off?

🧑‍💻 Built with agents, in full transparency

RunLore itself was built with my agentic coding tooling: Claude Code, skills, the superpowers framework and multi-agent reviews. It's a direct extension of what I explore in the Agentic AI series, in particular the post on coding agents and the one on tips and workflows. On the model side, Fable 5 did a lot of the specs/plans work, and Opus 4.8 most of the code.

And it was nothing like autopilot: even with those models, inconsistencies surfaced across iterations, especially around the learning loop, the subtlest part of the whole thing. I do write Go, but mostly DevOps tooling on relatively simple projects. RunLore's internal logic (recall, scoring, decay…) went well beyond what I would have written on my own, and agentic coding was essential here.

But I don't want to settle into the role of conductor, the one who writes the specs and drives the agents. I want to understand the internals, so I had a proper learning plan generated from the code (the concepts, the key functions, the decision points) to be able to talk about the project with confidence. That's a work in progress.

💭 Final thoughts

I'm well aware the project is very young, and since I'm its only contributor for now, that alone can be off-putting. But this isn't some "vibe-coded" throwaway: I've put real care into quality, performance and security, and I have actually tested it, several times, in two environments. I'd really encourage you to take a look. Here's what I've taken away so far.

• In practice, the diagnoses hold up well. RunLore doesn't hand back a vague answer: it surfaces the logs and events that matter and proposes a likely cause, backed by evidence.

• Their quality comes down to two things. The data sources you plug in (without metrics, logs or GitOps history, the agent reasons blind) and the model. A solid reasoning model (Opus, GLM, GPT…) produces far more reliable diagnoses than a small, cost-optimized model tuned for latency. On ambiguous, multi-signal incidents, where you really have to correlate, the gap becomes obvious.

• The upfront effort is real. Without a minimum of discipline, the knowledge base won't fill itself. In an already noisy production environment, count on a few hours of analysis and PR reviews before it starts paying off. It's an investment, not an install-and-forget.

• The fuller the knowledge base, the more useful the agent. Every resolved investigation you validate (your analysis, your context, a precise RCA) becomes reusable knowledge: an incident you've already seen comes back out of the KB in seconds, without re-running a full analysis, and the entries that actually resolve gain confidence while those recalled without ever resolving are dropped. The real unknown is still maintaining the base over time: how investigations map onto the right entries, what happens when the same symptom hides a different cause (RunLore groups them and asks for a human check), and whether duplicates and conflicts stay manageable. Guardrails are in place, but only weeks of real traffic will tell whether they hold.

So I'll report back after a longer run. Until then, the project is open (Apache-2.0), and right now is when your feedback matters most: try it on your platform and tell me how it holds up, open an issue for a bug, an idea or a use case, or send a PR. Everything that comes back from the field shapes what comes next.

🔖 References

• RunLore — GitHub repo · getting started guide
• Open Knowledge Format (OKF) — Knowledge Catalog
• Open Knowledge Format — Google Cloud announcement
• Andrej Karpathy's LLM-wiki pattern
• cloud-native-ref — reference platform
• k8sgpt · HolmesGPT · kagent