Ship It Conversations: Meta’s Francois Richard on AI Incident Response, SLOs, and Reliability at Scale

Transcript

AI is making it easier than ever to create more

software, more code, more diffs, more experiments,

more changes moving through systems that were

already complicated before everybody got a robot

assistant in the editor. And that sounds great

until you are the team responsible for production

because production does not really care that

the code was generated faster. It still cares

about latency. It still cares about overload.

It still cares about dependencies, rollbacks,

traffic routing, region failures, weird edge

cases, and whether the people on call actually

know what to do when the system starts acting

strange. That is the part of the AI conversation

that feels under-discussed to me. Not just can

AI write code, it can. Not just can AI help debug,

sometimes yes. The better question is what happens

to reliability when the volume of change goes

up, the amount of generated code goes up and the

context behind that code gets thinner. Because

incidents are not just caused by bad code. They

are caused by systems behaving in ways people

did not expect, under conditions they did not

practice, with dependencies they forgot were

part of the critical path. That is really what

this conversation is about. Not just SLOs, not

just incident reviews, not just AI agents. More

like, what does reliability actually look like

when change is accelerating? Systems are more

connected, and recovery matters just as much

as prevention. I'm Brian Teller from Teller's

Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, where I filter the noise and

focus on what actually matters when you are the

one running infrastructure and owning reliability.

Most weeks, it's a quick news recap. In between

those, I do conversation episodes with people

who are building platforms, running infrastructure,

organizing events, and thinking through where

this industry is actually headed. Today is one

of those conversations. I'm joined by Francois

Richard, an engineering director at Meta. We're

talking about reliability at scale, how AI and

automation are changing production risk, what

teams actually learn from incidents, and why

recovery practice matters just as much as prevention.

And I like this conversation because it gets

past the neat, clean version of reliability that

fits nicely in a dashboard. Reliability is not

just a number. SLOs matter, dashboards matter,

alerts matter, guardrails matter. But none of

that means much if the team cannot use it to

make better decisions, recover from real failures,

and improve the system after something breaks.

You start with the obvious goal. Keep the system

up. Keep the users happy. Avoid incidents. Do

not page people for nonsense. Do not let every

deployment feel like a coin flip. Then reality

shows up. A region has problems. A dependency

gets slow. A service gets overloaded and cannot

restart because it is too overloaded to recover

cleanly. Traffic spikes during a major event.

A change rolls out faster than the team understands

it. Or an AI-generated pile of code technically

works until it fails in a way nobody has enough

context to explain quickly. And somewhere in

the middle of that, reliability stops being just

prevention and starts becoming practice. In this

conversation, Francois and I talk about how Meta

thinks about reliability across both reactive

and proactive sides. Incident response, incident

reviews, SLOs, guardrails, validation, disaster

recovery testing and what happens when you actually

practice taking a region out instead of just

assuming the failover plan works because the

diagram looks good. We also get into what teams

learn during incidentsss, not just from the post

postmortem afterward but inside the pressure cooker

itself where engineers have to make decisions

quickly build consensus fast understand the system

under stress and figure out what is actually

happening while the clock is running there is

also a good thread in here around AI agents in

incident response not the fantasy version where

you hand production to an agent and hope it saves

the day more the practical version where AI helps

with investigation telemetry metrics logs relationships

across services and narrowing down what might

be happening faster than a human clicking around

dashboards alone and towards the end we talk

about recovery practice known failures versus

unknown failures why teams should test the failure

modes they claim they can survive how smaller

teams can learn from Meta-scale reliability patterns

and why not every system needs six nines on day

one so if you work around DevOps, SRE, platform

engineering, infrastructure, engineering leadership,

incident response, or you are just trying to

figure out what reliability looks like when AI

is increasing the volume of change, this one

should be worth your time. All right, let's jump

in.

Today, I'm joined by Francois Richard. He is an engineering director at Meta, and we're

talking about reliability at scale, how AI and

automation are changing production risk, what

teams learn from incidents, and why recovery

practice matters just as much as prevention.

Francois, thank you for joining me. Thank you,

thank you. Happy to be here. Okay, so when you

think about reliability at Meta scale, where does

the work actually start? I think for me, reliability

is actually two things. What people kind of like

know usually it's the reactive side is how do

we manage incidentss? How do we respond to problemss?

And how do we handle things and recover and fall

back when something goes bad? But at the same

time, it's also everything that I call like,

you know, the proactive side of the house. This

is everything that has to do with, hey, how are

we going to prevent reliability problems from

happening? What sort of guardrails do we put

in place? What sort of framework? What sort of

tooling do we allow platform or do we give to

platform services and application developers

to use? And also, what is the real validation

that we are doing? There's a lot of papers available

in the public and so on around like, you know,

there's something that we call like the storm

program. And it's basically our way to test like,

you know, disaster recovery, where once in a

while we take an entire region out, out of the

like, you know, like if there was an earthquake

or there was like a giant electrical grid failure.

And we test what happens with the system. How

do they recover? How do they handle the spike

in traffic and so on? So it's both like for me,

reliability is both like, you know, the reactive

side and also the proactive side. Is that kind

of like chaos engineering, the concepts? Yes,

yes, yes, exactly. And, you know, we go as far

as, you know, disconnecting a complete region from

the map. Interesting. Okay, so speaking about

reliability, what makes an SLO useful instead

of just another dashboard number? Like, how do

you think about that? Yes, an engineer, like

a senior engineer that I work with always says

that the SLO of something, of a system, is actually

the promise that you're making to your customer.

That, you know, XYZ should work 99.99% of the

time and you should expect that amount of latency

on average at, you know, P90 or something like

this. So we use this, like, you know, most of

the platform we operate at Meta and also the

product services and so on. They will all, the

big ones, will all have some form of SLO for

the top APIs or the top services that they are

doing. We have some alerts that will trigger

on SLO, but usually this is too late. Like, you

know, we want to catch problemss. The SLO allows

us in the long run to actually figure out like,

hey, are we investing enough in reliability?

Are we keeping the bar high enough? Or do we

have wiggle room to take some risk, introduce

new features? Because you all know that when

you introduce or you do a giant migration, there's

a period of time where it takes time to stabilize

things. So the SLOs are helping us make these

decisions in the long run. And you can clearly

see when you've been tracking them for a while

that some systems are trending up and some systems

are trending down. And for the ones that are

trending down, it allows you, it gives you the

argument and the data to start investing more.

So this is kind of like, you know, our way about

deciding if we have enough investment in an area

or if, you know, we are okay and we can take

more, you know, progressive risk. So you had,

we talked a little bit about incidents before

the show, and then you had just mentioned incidents.

Across the incident lifecycle, Where do teams

actually learn the most? So in terms of learning

during incidentss, I think it happens in two phases.

First, it happens while handling the incident

during what I call like the pressure cooker.

When you actually need to handle the incident,

you actually need to make decisions quickly,

build consensus fast with a small group of people

about what's the next action to take. And then

what are the bypass, the restart process, which

system is calling what. This is actually a time

where a lot of people, like junior engineers

and senior, they learn a lot. They learn a lot

about the system, but they also learn a lot about

team dynamics and themselves during that period.

Then the next phase happens when kind of like

you build the... incident review report or we

call it the SEV report, where you basically,

you know, once the incident is handled, mitigated,

you know, a few days after, you basically sit

down and then we have like a tool to do this.

And then people sit down and they write a report

around like, you know, hey, what exactly happened?

What was the root cause of this? What was the

exact timeline, which is kind of like forcing

people to go back into logs and all of this?

Could the alerts have fired earlier? Did we have

the right instrumentation? Did we diagnostic,

like diagnose this thing, like, you know, the

proper way? And, you know, what are the follow

-up tasks, like to make sure that this doesn't

happen again? or that we have some level of automation

to be able to remediate this type of problem

automatically. So you build that report. But

then from there, you go into a series of reviews.

You review it with your team, with your organization.

And if the incident was severe enough, we actually

reviewed them at the company level. And during

that time, it's a big opportunity to get feedback

from other teams. from other senior engineers

that, yeah, we see this pattern. Here's, you

know, how we handle this type of thing. And you

actually learn a lot during that process. So

this is usually like, you know, the learnings

happen in kind of like two phases. Yeah. And

so given that learning, what should an incident

review actually produce? Like once the incident's

over? The incident review, like we have a report

type of thing. It's a template. It has like,

you know, walks you to, for you to get like two.

the root cause of it. But I think the most important

thing that it produces is also like, you know,

what are the follow-up actions? The follow-up

could be like, hey, we need to implement more

redundancy. We need to implement better diagnostics.

We need to implement better ways of handling

these exceptions or problems. We need a different

deployment process. This is usually like, you

know, the best outcome we could get because,

you know, the more you mature your system, the

more you'll be able to cover a lot of these,

like, kind of like unknown slash unexpected situationss.

So this is usually like the good outcome of these.

And what do you find is the difference between

finding the cause versus improving the system?

The difference between the cause and improving

the system. Or is there a difference? There is

a bit. Because for me, finding the cause overall

is, do I have the right tools to pinpoint the

problem very quickly. And then I'll be able to

find the cause. And then the improvements are more

are more about eliminating this exact problem

or that class of problem. And the improvement

can be that like you know, like from a code structure

that this will always happen, but you can work

around with either deployment strategy, automation

of recovery and things like this so that it's...

It's not visible to the end users with strategies

of retries and things like that. So there's a

big difference between finding the cause. Like

if you work in mobile apps, you know that there's

going to be a couple of cases where like the

cause is going to be like, you know, the mobile

networks just drop on you. And the app needs over

time to come up with ways to mask it. You'll

never eliminate the cause, but you'll have apps

that are a lot better at masking this type of

problem. So it's slightly different. Okay, so

let's switch gears to AI agents. Are you using

AI in incident response at all? Yes. In triaging?

We use it for what I would call like investigation.

So, you know, there's usually like two classeses

of production incidentss. You have one class when

it happens. You kind of know what this is. Like,

you know, it's a pattern that you already recognize.

The alarms or alerts are very clear about what's

happening. So you kind of know and you can narrow

down and then you can just go focus on, hey,

you know, how do I remediate that? There are

some of the cases like where you kind of see

like, oh, I'm starting to see transient errors.

I'm starting to see latency creeped up and where

it's kind of like, I'm kind of not sure. And

then usually like, you know, we launch what's

called like investigations and things like that.

Really, the AI agents are helping us in investigation

because they can analyze and gather large swath

of data very, very quickly and then pinpoint

relationships between datasets. Like, you know,

it's probably the same for everybody, but we

have multiple layers of software on top of each

other. They call each other and they call back.

Sometimes finding complex situation requires

us to analyze a lot of data at the same time.

Even myself personally, like, you know, we have

like centralized, like, you know, monitoring

system. And then if you aggregate a lot of data

with very small granularity, like, you know,

at the 10 second or like at the minute level,

and you have a lot of time series, not the data

point, it gets a pain to actually like get the

right query and the right graph for the problem

that you're investigating at a point in time

like so the AI agent has kind of like shifted

this thing where they help us find these things

and even like you know in some cases like what

i do is like i will dump all the metrics locally

in the SQLite database and then like do all sorts

of things because it's local and my SSD can do

it a lot faster And then you use also the AI

agents to help you craft the queries. I don't

remember on top of my head all the various flavors

of joins and all of that. So it's helping a

lot. So with investigation, it helps a lot. It's helping

a lot. It's not perfect. You still need to guide

it very well because the topology of the system,

the request... The request flow between the system

is still not something that I will call it like

the various AI agents are actually mastering.

They master code very well. But these flows are

a little bit more difficult and understanding

how they map into the application or the product

flows is even more complicated. But it does help.

That's the investigation side of it. We're doing

a lot also into what I call like having a standard

dataset across the company that kind of like

represents the state of reliability. Like we

drive a couple of initiatives globally, like

we call it like change safety. Do you have SLOs

for something? Do you have guardrails? And with

these datasets being kind of official, then it's

easy for each service team or product team to

build their own customized kind of dashboard

like right now the cost of making dashboards is

almost free and really tailoreded to their use cases

or to their product using the standard dataset

and that's another place where it's been like

oh, this has been great. And how are you, I'm assuming

you're still validating that that root cause

though right if it comes up with a you know it

uses MCP servers it's reaching out to Argo Kubernetes

whatever it's gathering all the telemetry from

Grafana and then it's putting that together and

it's saying, hey, this pod was crash loop back

off. This caused this issue. Are you still validating

that? And what is that process for validating

that? Like, is there a standard SOP that you're

following? No, I don't think we have like a standard

evaluation. The evaluation, like most of the

time, the tools that you have in the moment from

an incident is either you move away from the

problem or you roll back the problem so we will

try like you know first you know move away let's

call it like a problem is a backend problem

it's isolated to a region we'll move away from

that region to kind of like prove that theory

and then we'll probably during that time roll

back the system or the backend or whatever

it was that introduced that problem and then

you can imagine that we can then now that the

hypothesis is live then we can shift a portion

of traffic back to that problem and see like

oh is it still happening or not this is all manual

we have some of these process that are automated

but they really boxed into very very well known

like you know they had like very well established

runbook that have been like you know in place

for the longest time that we know that when this

happened do this like it's almost like if then

else But the most like the investigation result,

the validation is more to roll out the fix, you

know, driven by human with like, you know, slow

roll type of techniques. Do you think that AI

is increasing the speed of change, the volume

of change or the type of failure that teams need

to expect? But it's clear that it's increasing

the volume of change. We have data internally

that, you know, the number of diffss is through

the roof. The number of lines of code changed is,

again, through the roof. The type of failure

is like one thing that, because there's more

code and it's going faster, it's the type of

failure becomes more into the, what I would call

like pseudo-unexpected. When a product developer,

application developer, when they... roll out

a new feature, they do understand the business

logic. They do understand how it should show

to the user, what they expect the user to do,

where's a corner where a user could get stuck.

They understand that. That's not a problem. But

if you have to build on modern apps on mobile

phones or on backend, A lot of these things are

built on top of frameworks. There's a lot of boilerplate

stuff, especially to handle like, you know, async

callbacks and things like this. And since a lot

of that code gets auto-generated, people are

losing context around these things. And when

something doesn't work, we realize that it takes

us more time to understand why it doesn't work.

Because there was not that additional context

that... the developer or that team developing

that feature took a lot of time to craft it and

a lot of like internal reviews and things like

that so so it goes faster out but that context

on how it's built is actually lost along the

way and this is basically what we need to reconstruct

live and again like i said earlier a lot of our

tooling is like you know in this case we're just

like we'll roll back and you figure it out like

bring back the system to a stable state and then

you can figure it out on your own time without

being on the pressure cooker. Okay, so speaking

along the same lines, you had mentioned too that

recovery comes down to practice. So what does

good recovery practice actually look like? You

know, there's a set of standard failuress you expect

your app to have. And it starts at, you know,

if you're running, if you have data centers or

region in the United States or in Europe or something

like that, you get... Tons of things like fiber

cut in hurricanes and, you know, power grid and

things like that. You need to understand how

your system will react. Like, you know, a simple

thing, like if I'm a startup and I run into a

AWS region, but I actually never test the fact

that one of them will go down, I can guarantee

you, like, as soon as you get a little bit of

complexity in your system, something will not

work as expected in there. for us is actually

to do, to validate it, to not just test, like

we run, I would call it like data simulation

and analysis in advance to figure out like, okay,

if we lose that data center, we'll be okay. But

it is always interesting. And then after we've

done the validation and vetted that it all worked.

We actually do the test and we discover a suite

of other problems that the analysis had never

found. So exercising the failure for real at

the lower scale allows you to find a lot of these

problems. And it's the same thing like, you know,

if you have very sensitive hotspot in your application

and your product, injecting failure for real

at a very low percentage of user will... allow

you okay do i have the right detection like did

that trigger properly did it get to the right

person did this team know how to handle it like

without this it's it all becomes like improv

like you improv all the time versus when you

have a portion of your infrastructure that's

always tested in such a way first you keep on

hardening and it gets better and it gets better

and the muscle of handling it gets better also

so i guess that I was going to ask about like

how, how would teams practice known failures

versus unknown failures? But it sounds like it

comes down to just that practicing and pulling

out the regions. Okay. And there's, there's a

bunch of types of failure. Like, you know, I,

I made an allusion about worst-case failures

versus worst-case failure, which is, um, you

know, you lose a complete region. And with the

weather patterns in the U.S., you know, we have

hurricane season, we have like fire season, tornado,

like the cuts happen everywhere. So you got to

be ready. But there's also other things like,

you know, for meta, we are often, I would call

it like we got, we get these traffic spikes.

It could be because there's something in the

news related. It could be a big event like the

Super Bowl, New Year's Eve. The World Cup's coming.

So we know that every time there's going to be

a goal, it's going to spike. What we realize

over time with practice is that some system will

be overloaded. And in some cases, you'll have

to restart them. And during overload, the damn

thing cannot restart. You cannot kill it. Like

it's all completely dead. Like it's completely

blocked. So we practice a lot of these cases

too. And it's something specific. To us, we get

these spikes that are kind of unexpected. Most

systems have some form of overload protection

up to a point, but it's something we really had

to invest in, into like, in the worst load scenario,

can I restart? Do I have, like, is my process

smart enough to just accept requests later down

the road? Or do I have enough control in the

traffic routing to, like, okay, choke the traffic,

bring back up, warm up your memory, and then,

like, gradually. Like, we had to practice that

a lot. Yeah, so you're dealing with, like, thrashing

and having, like, no... Yeah. Yes, yes. Resources

being completely exhausted. Yeah, yeah. Completely

gone. And is that typically at the control plane

Completely gone. And is that typically at the control plane

level? Or... Some of our control plane is pretty good

you know, there's some backend. Like you can

imagine like, you know, an Instagram feedback

an Instagram feed backend or Facebook feed backend

same type of things. And we have a couple of

academic paper. People can look at it. It's called

like Taiji, I believe. And then when we discuss

these things on how we can reroute traffic and

then control and then remove it from an area

to allow the system to reload. Okay. So with

that in mind. I would imagine most listeners

are not operating at Meta scale, so they're not

dealing with every time there's a goal at the

World Cup, you know, their servers go down. What

ideas can actually transfer for most like SRE

DevOps that are listening? I think the example

I gave earlier, like you often see like no blame

on Amazon, but if US East go down, like, you know,

we see the world of the internet like go like

berserk. So I'm like, guys, like, you know, you

got to start testing these cases. Like you've

seen it happen, like, you know, at least like,

you know, three or four times in the last two

years. You got to start like, you know, having

your dual regions and really validating it. And

maybe it's a question about like, you know, every

Tuesday, I do not run a single thing in US East,

you know, whatever the name is these days

to make sure that it's, you know, it's performing.

And then it's kind of like tackling the problems

in a way like, you know, when you look at your

incident inside a small, a smaller company, medium

sized startup or so on, like I get a list of

incidents. There's a point where you start having

enough data that you could say, okay, most of

them are caused by, it's because we launched

that feature or because we're doing configuration

change or because we have a billing problem.

It's starting to. trend your your SEV data and

figure out hey what are the top two or three

then we can start attacking and really focusing

because like you can focus on everything but

you'll get nowhere it's really start bucketizing

like you know in having the discipline of writing

it down like it's not that hard to write like

a SEV report there's a lot of example in there

and having the discipline of going back and Now

with LLMs, do the analysis is kind of like trivial.

You need to, we used to need to allow a lot of

people to do this and now it's a lot easier.

So I think that's a big opportunity to actually

really focus on where reliability matters. Yeah,

that's fair. You had mentioned too that reliability

expectations should match the system and product

life cycle. Yes. Can you talk about that? Like,

because 100 % uptime is not always like the right

goal. Yes, yes, yes, yes. There is. there's like

you know when i say about the expectation versus

life cycle we should always expect like if something

is is has some level of complexity and some level

of feature at the beginning when you roll it

out like you know getting two nines or even three

nines of reliability it's pretty good like like like

not a simple system that's that that that's usually

like you can get it easy but like if it's early

in the life cycle of that product or that backend

at the beginning, it's going to be rocky and

it should be expected. Like I see a lot of people

start with the assumption that, Hey, everything

should be like six nines, six nines guys. Like

this is like insane. Like, and you invest a lot

and then you, you kind of like mix, uh, miss

your product market fit while you're doing that.

Like there are some like Facebook and Instagram,

there's a gazillion experiments that runs at

a given point in time. some of them are not fully

reliable like and it's okay because we're trying

to figure out if that feature will be something

that users enjoy and use or not but they are

other part of the app which like needs to be

like rock solid and this is the place where we

invest so depending on the life cycle if you

are early or not because there's a cost at investing

and then when especially like either your SRE

or production engineering and you have to also

convince your PM, your product managers that

this is the case, having that conversation from

that standpointpoint is a lot more easier than just like

being the one that says, no, I need six nines

of reliability for everything. That's not happening.

Yeah. Well, six nines too, that's like what,

31 seconds a year of downtime? I mean, the amount

of infrastructure and cost complexity to just

keep a system at scale. Depending on the system,

I mean, I assume if it's like a very important

financial system, maybe it matters. But for the

majority of systems, like do the users really

care about 31 seconds of outage per year? I mean,

are they going to? They do care. You know, we

can see from our own data that, you know, if

we have too much problems in a row, we can

see that, you know, engagement eroding. So they

do care. They do show. But it tends to stabilize

when we stabilize things. So there's a correlation

there. But sometimes the investment to reach

that extra nine is just so high. And while you're

doing that extra nine, you're also sacrificing

not just the speed and velocity, but also your

mitigation time for future unknowns. Because

when you have invested so much complexity and

then you get into an unknown situation and then

now you have to untangle all of this, your mitigation

time will be higher. So it's kind of like it's

a delicate trade-off. Like it's not a win-win

all along. Yeah, for sure. It's a reality there.

A real conversation that you have to have and

a balance that you have to have there. Okay,

so wrapping up, this conversation also connects

to At Scale systems and reliability. Why does

that program feel relevant right now? Yeah. So

we have this suite of conferences, like it's called

At Scale. And we have like four conferences a year.

And the next one that's coming is basically systems

and reliability together. And what we want to

target for this specific conference is, hey,

how are we injecting? What are the systems that

are like, you know, building reliability for

AI? And what are the systems that are under the

hood for AI? Because people tend to talk about,

hey, we talk about the models, okay? But we never

talk about like the underlying plumbing that

is required to train and serve the model and

the insane amount of data that we need to move

back and forth. And then on the other angle,

we also, we will discuss like... All the other

areas where we actually use AI to enhance reliability.

So it's both like, you know, both cases that

we want to do. This conference is going to happen

in person. It's going to be in Bellevue. And

you can find the detail. The website is at scale

conference. And we could put that at the show

notes. Yes. So if someone works in infra SRE

platform or engineering leadership. Why should

this be on their radar? And what are you watching

specifically with the conference? I'm just curious.

Personally, I'm a big fan of this conference.

I've been part of it for a while. And one thing

that I tell the speakers that are coming from

Meta, and we have speakers from NVIDIA, from

Microsoft, from Google, the goal here, I want

to have the real technical discussion. With Meta,

I'm not in the business of selling cloud services.

Like I'm not the business of selling API or things

like this. So what I want to talk is like, you

know, Microsoft, you have a Kubernetes cluster with

one million servers. How the

hell did you do that? Like, this is like, you

know, the goal is really like, I want to really

understand the story behind the system. I want

to have like the real technical conversation.

And I want to avoid the sales pitch of like,

oh. use that service, use that thing. So this

is really the focus of the conference is to really

have that technical discussion and also the story

behind the system. Like, I'm always fascinated.

I've always been fascinated by, like, okay, a

team starts something. Like, I've been involved

in ZooKeeper and all sorts of other things in

my days at Yahoo back then. You start something,

it was to solve a problem. Then you open source

and you're like, oh, that became this? Oh my

God. So I'm really interested in this and then

like the ups and downs of the team because you

always have like, you know, the hype at the beginning

and then, oh, the reality hits and you're like,

oh my God, like this will not work as expected.

And how do you overcome that? Like that, for

me, that's the most interesting part of all of

these stories and all of these presentations.

So, okay, wrapping up, what... kind of reliability

conversations does the industry need more of

i think right now the industry does understand

the the use of AI for code generation like i

think we get that i think we can get that we

can go all out i think we understand that everybody

can build a custom app like for themselves and

they will only use it for for them it's gonna

be perfect for them and then it's okay if it

goes down but i think the rest of the industry

has not kept up with that rate of change and

there's not enough investment in kind of like

defense like we're able to generate code are

we able to debug it faster are we able to understand

it faster are we able to troubleshoot it faster

like like that has not kind of followed. And

then I feel that we are catching up now. And

then the hype seems to be like mostly on, on

the model. And then there's amazing infrastructure

that had to be built underneathneath. And I think

you know, everybody needs to understand it a

little bit more. Yeah, for sure. Awesome. So

I will put links for At Scale Systems and Reliability

reliability in the show notes. I can put your

LinkedIn there as well. Is there any other links

or comments that you'd like to give to the audience

before we wrap up? No, I think that's it. Don't

give up. Awesome. Thank you so much, Francois,

for coming on. Really appreciate your time. Thank

you so much. Thank you. Bye-bye. All right.

That was my conversation with Francois Richard

from Meta. My biggest takeaway from this one

is that reliability is not just about preventing

failure. The better question is, what happens

when prevention fails? Because sometimes the

answer is a rollback. Sometimes it is moving

traffic. Sometimes it is draining a region. Sometimes

it is restarting a service. Sometimes it is realizing

that the service cannot restart cleanly because

it is already overloaded, which is the kind of

fun little production detail that does not usually

show up in architecture diagrams. That is the

part that I think is worth paying attention to.

a lot of teams talk about reliability like it

is mostly a tooling problem. Get the right dashboards.

Get the right alerting. Define the SLOs. Add

some runbooks. Maybe sprinkle in some AI and

pretend the incident lifecycle is solved. But

reliability is not just the tools. It is the

practice around the tools. It is whether the

SLO actually represents a promise to users. It

is whether the alert fires early enough to matter.

It is whether the incident review produces real

follow-up work instead of just a nicer explanation

of what broke. It is whether the team has practiced

the failure mode before production forces them

to learn it live. And honestly, that is the part

of the conversation that translates really well.

Even if you are nowhere near Meta scale. Most

of us are not dealing with World Cup traffic

spikes or massive global systems, but a lot of

us are depending on a cloud region more than

we want to admit. A lot of us say we are multi

-region, but have not actually run without the

primary region on a boring Tuesday. A lot of

us have runbooks that look reasonable until someone

has to follow them under pressure. A lot of us

have services that should recover automatically,

but only if the failure happens in the exact

way we imagined. That is where the work is. Practice

the recovery. Test the boring assumptions. Look

at your incident data. Bucket the causes. Figure

out what keeps showing up. Then go after the

top patterns instead of trying to boil the ocean.

I also liked Francois' point about AI changing

the reliability equation. AI can absolutely help

with investigation. It can look across a lot

of data quickly. It can help build queries, connect

patterns, and speed up the part where humans

are trying to figure out what changed and what

is related. But AI is also increasing the volume

of change. More diffs, more generated code, more

boilerplate, more systems moving faster, and

sometimes less human context behind the code

that just went out. That is a weird trade-off.

Because if code moves faster than understanding,

Reliability teams are going to feel that gap

during incidentsss. The system breaks, and now

someone has to reconstruct not just what changed,

but why it changed, what the generated code is

actually doing, what assumptions it made, and

how to get back to a stable state. That does

not mean AI is bad. It means the defensive side

has to catch up. Debugging has to get better.

Observability has to get better. Incident response

has to get better. recovery practice has to get

better. And humans still need to be in the loop

for judgment, especially when the system is too

important to let a guess turn into the next mitigation.

I also think the lifecycle point matters. Not

every system needs the same reliability target.

A brand new experiment probably should not get

the same investment as a core production path

that millions or billions of people depend on.

Six nines sounds impressive until you realize

what it costs, what complexity it adds, and how

much slower it can make future changes. But the

reverse is true too. If a system becomes important

and the reliability investment never catches

up, you are just borrowing risk until production

collects. So maybe the healthier conversation

is not how do we make everything maximally reliable.

It is more like what promise are we making? Who

depends on this system? What happens when it

fails? And have we practiced the recovery enough

to believe our own answer? That is probably where

a lot of reliability conversations are heading.

Not AI will fix incidents. Not SLOs solve reliability.

Not just make everything multi-region and call

it done. More like, what failures should we expect?

What failures have we practiced? And what are

we learning every time production teaches us

something? I'll have links to Francois, Meta's

At Scale systems and reliability event, and

anything else we mentioned in the show notes.

If you enjoyed this conversation, follow or subscribe

to Ship It Weekly wherever you listen to podcasts.

It helps the show and it makes sure you get both

these conversation episodes and the weekly DevOps,

SRE, platform, cloud, and security news recaps.

You can also find the show notes and links over

at shipitweekly.fm. Thanks for listening, and

I'll see you later this week.

Ship It Conversations: Meta’s Francois Richard on AI Incident Response, SLOs, and Reliability at Scale

Watch this episode here

Chapters

Transcript

Catch This Episode

Host Commentary

Show Notes

👤 Guest

More from Ship It Weekly

EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

Amazon Q CVEs, Hijacked npm and Go Packages, AWS WAF HTTP/2 Issues, Lambda MicroVMs, and Why Execution Is the Boundary Now

containerd CRI Vulnerabilities, Datadog PostgreSQL HA on Kubernetes, AWS DevOps Agent with Datadog MCP Server, EKS Control Plane Egress, and Why Users Feel the Wait

CISA’s GitHub Leak, AI Root Cause Analysis, Copilot Agents, Claude Code in CI/CD, and Kubernetes Seccomp Risk

Get the next episode in your inbox