AI Fails Like We Do

Ten minutes into our conversation, Sylvain Kalache drafted me into a role-play. “Let’s say I’m the SRE and William, you are the developer,” he said. Production is down. He runs git blame on the broken microservice and finds my name. “Hey William, how about you come sit next to me, or let’s jump in a Slack channel — maybe you can help me solve it.” That’s the oldest move in incident response: find the author, sit them next to the problem, let what’s in their head speed up the debugging. And git blame will still find the author fine. What it can’t find anymore is an author who knows the code. “But now with GenAI, you actually may not know what you wrote, because you prompted an AI assistant to do it.”

His summary: “Potentially more incidents, and also less help when you have to troubleshoot something.”

Sylvain has been on the receiving end of that pager for most of his career. He started out building private clouds in France before public cloud was really a thing (“crawling in data center floors, passing cables”), then joined SlideShare as the SRE practice was first emerging, and rode the acquisition into LinkedIn, where he spent three-plus years as a senior SRE. Along the way he co-founded Holberton School, an engineering school with no teachers and no lectures that has trained thousands of engineers across more than 25 countries. Now he runs AI Labs at Rootly, the incident-response platform, where his job is to prototype what AI can do for reliability. “It’s kind of a constant hackathon,” he told me. Sylvain hedges everything he hasn’t verified, flags his theories as theories, and, as it turns out, will happily tell you about the time his own conference talk fell apart on him.

The bummer

Last October, Sylvain gave a talk at SREcon, the conference where the reliability community compares notes. When he sat down to prepare it, the plan was to catalog all the novel ways generative AI breaks production: have some fun with AI slop, watch the LLMs fail, name the new failure modes the industry needed to worry about. The plan didn’t survive the preparation. “I was really trying hard to find new failure cases,” he admitted. “To be honest, I didn’t find them, which was a bummer, because my entire talk was about that.”

It’s not that he found nothing. If you ask a coding assistant to write a unit test for code it just wrote, “it will write a unit test replicating the already broken code.” There’s slopsquatting, an attack that at least sounds brand new: LLMs hallucinate package names, attackers register those names and fill them with malicious code. In one experiment, a researcher registered a hallucinated package name with a dummy library, and it was downloaded thousands of times in a matter of months and cited in README files across GitHub.

Then he showed the list to a friend, an SRE leader at Airbnb. The reaction he got back: “Boy, I’m pretty sure that humans do the same type of mistake.”

So Sylvain checked. Slopsquatting is blindly pulling in a package you’ve never inspected, and we already do that: it’s curl | bash, which we’ve been piping into our shells for decades and which, he pointed out, is still how Docker recommends you install it. AI agents hallucinating their way into deleting someone’s files? “We’vebeen ding rm -rf by mistake, using root.” One by one, the novel failures paired off with old habits, and what he was left with was the opposite of the talk he’d planned: “The way that LLMs are failing is stuff we’ve been doing for the longest time as humans.”

On cats and LLMs

The strangest example he brought up involved cats. Researchers gave an LLM a problem statement with one irrelevant sentence appended: cats sleep for most of their life. With the cat fact in the prompt, the model was something like 40% less likely to solve the task correctly. The machine gets distracted — and nobody needs a study to prove that humans do too. “As you know, you also get distracted,” he said. “Cats are the most important thing on the internet.”

By this point it was all starting to sound a little too tidy, so I pushed on the example that seemed most machine-shaped: the lazy unit test. A human could write a test that just ratifies their own broken code, but a human who generally likes their job mostly doesn’t. Isn’t that a class of failure that’s simply more prevalent in the machines? Sylvain didn’t defend the thesis. “I think what you described is absolutely true, but I’m not sure if the issue is the LLM or the prompt.” The chaos of LLMs, he pointed out, is that they’ll produce something no matter what you give them: “whatever prompt you give them, it’s gonna work.” And the gap between a good prompt and a bad one is enormous. His own benchmark project found that prompt optimization alone could double some models’ performance on SRE tasks. So maybe it’s a new failure class, or maybe it’s an old one: garbage instructions, confidently executed. He wasn’t sure, and he said so.

V times P

What he is sure about is the arithmetic. For the talk, he deconstructed incident rate into two numbers: V, the volume of changes to a system per unit of time, and P, the probability that any one change breaks something. AI is driving V up dramatically; that’s the whole point of it. Hold P anywhere near constant and the conclusion writes itself: “The incident rate will drastically grow.” Then he immediately fenced it in: “That’s actually the theory on paper. I’m not saying this is happening.”

The concern survives the hedge, though. Even if AI’s mistakes are our mistakes, “it’s doing it at least 10 times faster.” The failure modes are reruns; the throughput is new. The guardrails SREs have spent the last ten or fifteen years building (test coverage, deployment gates, observability, chaos testing, capacity planning) were designed to catch mistakes made by unpredictable humans, and in Sylvain’s telling that work is “not becoming obsolete, but more important than ever.”

He also thinks the same force pushing V up can push P down. Mutation testing has been around for years without much adoption; throw an LLM at it and it scales. He pointed to Meta’s published work on LLM-powered mutation testing: roughly 10,000 mutants generated across the Facebook, Instagram, and WhatsApp codebases, around 500 generated test cases, three quarters of them approved by engineers. Run the two effects against each other and “it might equalize, or we might even end up with better reliability at the end of the day.”

The dream machine

There’s a portion of his community that wants nothing to do with any of this, and he role-played them too: “I’m not touching systems that are non-deterministic. It’s absolutely against everything I’ve been doing. And now you’re telling me to use this dream machine.”

His answer to the skeptics was the closest thing the conversation had to a thesis. “Humans are unpredictable. We’re non-deterministic.” Every guardrail in the SRE canon exists because the industry already learned to run reliable systems on top of unreliable authors. The blameless postmortem, trust-but-verify, defense in depth. None of it assumed the code’s author was trustworthy. “We’ve already been doing this with humans,” he said. “It’s trust but verify. Make sure that whatever they do is fine.”

That’s why the failure mode he’s actually worried about is cultural, not technical. He told me about a consultancy that surveyed engineers at a bank about their outages. The answer that came back: “It’s not my fault. GenAI wrote that.” One survey, one company, he cautioned. But if one engineer says it out loud, others are thinking it. “There is an erosion of the ownership of the code base, which I think is totally wrong. And I think engineers need to start thinking more as SREs.” His suggestion is concrete: don’t just accept what your assistant writes. “How about using another model to judge your code?” The blameless culture survives the agents fine. What can’t survive is blameless sliding into ownerless.

SEV2, maybe SEV1

Fifteen years ago at LinkedIn, annoyed by repetitive incident work, Sylvain wrote up a concept for a system that would learn from logs and signals and heal itself. “We have all these signals, all these logs, all this data — we should be able to learn from that and correlate.” Machine learning wasn’t good enough to build it. His employer patented it anyway (“that was not really my goal, but my company was like, hey, we need to patent stuff”), and he moved on to found a school.

Now he’s at Rootly watching the industry build the thing. The tools are called AI SREs (he knows people hate the name — Rootly wrote a whole post about it), and the first thing they do isn’t even intelligence, it’s logistics: pulling context from Slack, postmortems, Git, and observability tools into one window in minutes, work that used to mean tab-hopping between Grafana dashboards mid-outage. The second thing is the intelligence: developing the intuition a tenured SRE has, the “when this service breaks, it’s usually that database” sense, and investigating on its own. For simple incidents, the SEV2s and SEV3s, “we are seeing customers where the incidents are no longer handled by humans.” Complex distributed failures, not yet. There, it “kickstarts your investigation, gets some context going, and then the human can take over.”

Nobody’s job disappears in his telling; it relocates. “We’re all becoming managers in some ways — or maybe not managers, but team leads.” Engineers move “away from the manual work of typing lines of code to directing,” closer to the business, further from the craft of typing. And the incidents themselves? He loves the thrill, he admitted, just not at night. “I think that’s definitely something we want to solve, and not say, hey, let’s keep this problem for humans.”

Sylvain went looking for proof that the machines fail in ways we’ve never seen, and found a mirror instead. That’s the reassuring part. We know how to build for unreliable authors; we’ve never had any other kind. The uncomfortable part is the other thing he found: the guardrails we built for mistakes made at human speed now have to hold at machine speed. When did you last check that yours would?

High Output is brought to you by Maestro AI. Sylvain’s prescription for AI-written code is trust but verify: don’t blindly accept what the agent produces, put a second set of eyes on it, even a second model. But verification is a habit, and it happens (or doesn’t) inside the working session between an engineer and an agent, where a PR count can’t see it. Your Anthropic bill tells you something is happening. Maestro tells you what. Maestro plugs into Claude Code and Codex and turns agent sessions into an honest record of the work: where the tokens went, what kind of work they bought, and whether anyone verified the result before it shipped.

Visit https://getmaestro.ai to see how we help engineering leaders calibrate AI sessions against shipped outcomes.

Thank you for subscribing. Share this episode.