Get in touch
Thank you! Your submission has been received!
Close Form
Oops! Something went wrong while submitting the form.
Blog
Every team has a few engineers who just know.
They know which dashboards matter, which logs lie, which alert is a false positive, and what script actually works when everything's on fire.
They've seen every outage, every weird failure mode, every workaround that never made it into Confluence. When incidents hit, everyone turns to them, or waits until they're online.
That hard-won knowledge keeps systems running, but it's also what makes them brittle. It doesn't scale, it doesn't transfer, and when those engineers leave, reliability leaves with them.
"We have years of institutional knowledge, but it's all in people's heads or buried in Slack." — Engineering manager at a global software company, during a product review
The fix isn’t “better documentation.” It’s executing expert investigations automatically, so the know-how lives in the system, not in someone’s head.
Tribal knowledge isn't just a documentation issue; it's an operational risk. The next evolution of SRE practice isn't more dashboards or faster alerts. It's eliminating the need for tribal knowledge entirely.
Engineers don’t avoid documentation out of neglect. In the moment, incidents demand full attention. Afterward, fatigue and context-switching can get in the way of write-ups. The result is knowledge that lives in people’s heads and travels through conversations, not artifacts.
On-call stress kills documentation.
When you’re debugging a cascading failure at 3 a.m., you’re not writing a wiki page.
Tools multiply; workflows fragment.
Metrics in Grafana, traces in Datadog, logs in Splunk, configs in GitHub—the path through them lives in people’s heads.
Runbooks age instantly.
Even when they’re written, they’re often stale by the next quarter.
Social knowledge moves faster.
Slack threads, DMs, and hallway conversations contain the real lessons—but aren’t written down before the next incident hits.
"We’ve got documentation, but most of the real context is in Slack. You have to remember what to even search for." — Senior engineer at a large observability platform, during a customer interview
Documentation decay isn’t just annoying. It causes teams to reinvent the “incident investigation” wheel every time a serious alert fires.
The answer isn’t more discipline. Asking engineers to write longer postmortems or exhaustive runbooks only trades one form of toil for another. Creating a clear, reusable investigation record often takes longer than fixing the issue itself. It’s a different skill, a different mindset—and one that burns people out.
And even when it’s done well, the reward can be fleeting: systems change, dependencies shift, and that beautifully detailed runbook cluld be outdated before the next incident. It’s no wonder documenting feels futile.
Tribal knowledge feels like speed—until it doesn’t.
Hero culture. Senior engineers become bottlenecks for reliability.
Slow onboarding. New hires take months to develop “intuition.”
Repeat incidents. The same failures reappear because fixes never become institutional memory.
Lost resilience. When experts leave, the organization forgets how to recover.
"When someone goes on vacation, we literally get slower. Nobody else knows the right dashboards." — SRE leader at a mid-size SaaS company, during a design session
Every outage teaches something valuable. The problem is only a few people remember the lesson.
And for years, this has seemed unsolvable—until now.
The real solution isn’t better documentation—it’s eliminating the need to remember or recreate how an investigation was done in the first place. Make the investigation run itself, in parallel, across your systems, and capture the steps as they happen.
Let’s look at how that shift translates tribal knowledge into institutional knowledge.
PagerDuty fires an alert: API latency has crept up to 3 seconds across your checkout service.
This isn't a vision of the future. RunLLM does this in production today. When an alert fires, RunLLM:
Latency spike caused by connection-pool exhaustion in checkout-service v2.1.4, deployed 14 minutes ago.
New database query in PaymentProcessor.processOrder() is holding connections 8× longer than baseline.
Recommended rollback: checkout-service → v2.1.3.
Instead of recording what engineers did, the AI replicates their reasoning—running the same queries, making the same correlations, and reaching the same conclusions your best engineers would have.
And it gets better over time. Each investigation teaches it more about your systems and failure modes. The next time a connection-pool issue appears, the diagnosis is immediate and the runbook reflects what actually worked.
Humans can reason across telemetry, infer causality, and learn from every incident too—but only a handful can do it reliably, and never at scale. That's what creates tribal knowledge in the first place: a few experts carrying the cognitive load for everyone else.
AI changes that. It doesn't replace human reasoning; it makes expertise available to everyone. By capturing and refining the investigative patterns that work, it turns expert intuition into infrastructure the whole team can use.
This changes how reliability scales.
Before:
Incidents required expert intuition. Junior engineers escalated. Senior engineers became bottlenecks. Knowledge transfer happened slowly through mentorship and repetition.
After:
The AI conducts the investigation. Engineers review findings and approve actions. Every incident strengthens the system's diagnostic capability. New team members get the same investigative support as veterans.
"We want new engineers to handle incidents with the same confidence as the veterans. If AI can provide that context and reasoning automatically, it's transformative." — Director of engineering at a high-growth SaaS company, during evaluation
The tribal elders are still the experts—but now their expertise executes automatically, every time, for everyone.
For SREs: Faster incident resolution, fewer 3 a.m. fire drills, and investigations that run while you’re still putting on coffee. More importantly, the load stops falling on the same few senior engineers—expertise scales without burnout.
For leaders: Institutional knowledge becomes operational infrastructure. Reliability no longer depends on who’s online.
For systems: Resilience compounds. Every incident makes the next investigation faster and more accurate.
"Our runbooks are never up to date. If AI can keep them fresh based on real investigations, that’s a game-changer." — SRE leader at a large digital platform company, in a strategy call
Tribal knowledge is how teams survive in the short term—but it's how they fail in the long term.
The future of SRE isn't about capturing what experts know. It's about building systems that can execute expert reasoning automatically—and improve with every incident.
Human engineers still lead the response. But the diagnostic work—the hunting, the correlation, the causal analysis—runs itself.
That's not augmented reliability. That's reliability that doesn't need heroes.
© RunLLM, Inc. 2025
.avif)