Ship It Weekly Transcripts

Mobile app security is one of those areas where cloud and backend people can get a little too comfortable because in the backend, you control a lot. You control the runtime, you control the servers, you control the network, you control the policies. You can wrap the thing in IAM, logging, monitoring, deployment controls, runtime controls, and a whole pile of infrastructure that at least gives you a fighting chance.

Mobile is different. With mobile, you ship the app and now it is out there. It is on someone else's device, in someone else's environment, maybe rooted, maybe jailbroken, maybe running next to malware, maybe sitting in the hands of someone trying to reverse engineer it, hook values, bypass checks. Scrape data, or figure out how your business logic talks to your backend.

And that changes the trust model because scanning the app before release matters, but it does not answer the bigger question. What happens after the app leaves your build pipeline? That is really what this conversation is about. Not just mobile scanning, not just App Store approval, not just we don't store secrets in the app, so we're fine. More like, what does mobile app security actually mean?

When the thing you shipped is now a public artifact that attackers can pull apart, instrument, modify, and abuse. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick news recap.

In between those, I do conversation episodes with people who are building platforms, running infrastructure, securing applications, and thinking through where this industry is actually headed. Today is one of those conversations. I'm joined by Joel DeStefano from Guardsquare, where he is a product manager.

We're talking about mobile app security, what it means beyond basic scanning, why mobile has a different trust model than backend or cloud systems, and how teams can improve posture without turning CI/CD into a nightmare. And I like this conversation because mobile security can be easy for infrastructure teams to underestimate.

A lot of us are used to thinking in terms of backend controls, cloud permissions, network boundaries, deployment pipelines, runtime monitoring, secrets management, service to service auth, the usual stuff. But a mobile app is not just another backend service. Once it ships, you do not control the runtime in the same way. You do not control the device. You do not fully control what else is running on that device.

And you definitely cannot assume that the OS the App Store or a pre-release scan is going to cover everything for you. You start with the obvious goal. Build the app. Scan the app. Avoid obvious vulnerabilities. Check dependencies. Fix bad TLS configuration. Do not ship secrets. Get through the App Store process. Then reality shows up. Attackers can decompile the app. They can inspect the business logic.

They can look for sensitive workflows. They can run debuggers and instrumentation frameworks. They can hook values, bypass checks, abuse accessibility services, overlay screens, or try to convince your backend that a compromised app is still genuine. And somewhere in the middle of that, mobile security stops being just a pre-release checklist and starts becoming a layered strategy.

In this conversation, Joel and I talk about why mobile is different from backend and cloud, what teams assume the OS is protecting them from, why scanning is necessary but not enough, and why there is no sensitive data in the app can be a dangerous oversimplification. We also get into the main layers of mobile app security, testing, hardening, runtime protection, and monitoring.

That includes code hardening, obfuscation, runtime checks, detecting rooted or compromised environments watching for hooking frameworks and monitoring what is actually happening after the app is in the wild there is also a good thread in here around cicd because security that destroys the release process usually does not last so we talk about build overhead release gates what should block a release what should become

Tracked risk and how teams can add mobile protections without making every app release feel like a hostage negotiation.

And toward the end, we talk about AI accelerating attackers, why fundamentals still matter, what a sane first step looks like for teams trying to improve mobile security, and why App Store approval should not be treated like a security strategy.

So if you work around DevOps, SRE, platform engineering, AppSec, mobile engineering, CI/CD, or you support teams shipping mobile apps that connect to important backend systems, this one should be worth your time. All right, let's jump in. Today, I'm joined by Joel DeStefano from Guardsquare, where he's a product manager.

We're talking about what mobile app security actually means beyond basic scanning, why it's different from backend and cloud security, and how teams can improve posture. Without turning CI/CD into a nightmare. Joel, thank you for joining me. Hey, Brian, thank you. I appreciate being invited onto the show. So I'm curious, why is mobile different from backend and cloud? What changes in the threat model?

Fundamentally, it's a shift in the trust model because... On the backend and in cloud, you have relatively good controls over the runtime environment, the servers, the infrastructure, the policies that you put in place. If you look at a mobile application, you ship it. It's a public artifact at that point. You completely lose control over what's happening to that and how is it being used.

So once it's out the door, if you don't have reasonable levels of protection into that application, it can be used to read. Attackers can reverse engineer that. You might be installed into an environment on a phone that has malware that is going to scrape that user's information. So there's risks when you ship a mobile application to go. I wouldn't say it goes beyond the backend.

It basically essentially opens up a different paradigm of risk and a different trust model. So if you're like a DevOps or SRE person, what's the simplest way to think about it? So there's, I would. Think about it in terms of integrating security into your everyday life, into your build processes. And there's a couple of different facets to that. There's the scanning aspect, which you had mentioned.

Scanning is something you would do before you release the application. That's sort of like your background to make sure that. I don't have any vulnerabilities. I understand my third party dependencies. My coding practices, like literally how I code something is not insecure and or improper. So I could maybe have TLS improperly implemented. Well, that's going out the door, right?

So that's sort of the, we do this in -house, we do this before release. But then once you release those type of things, those are more point in time type of protections. Now you've done your due diligence on. Prior to release. When it goes out, you need to still be able to trust that the applications and the users that are connecting to your backend are using a genuine application. It hasn't been compromised.

And the integrity of that app and the entire runtime execution is good. That's the only way you could actually sort of protect yourself. You need to be able to trust it. And that's sort of the gap and the challenge with a mobile app opposed to a backend or a cloud service is that. That visibility is lost. It's basically a public artifact at that point. And you really don't know. You don't control that environment.

You don't know if it's going to a malware environment. You don't know if it's going to the hands of attackers. You don't know what they're doing with it. That's the real fundamental difference. I would call it a shift in trust. And it's essentially centered around the control. Is there common misconceptions that people have about mobile app security?

Does it just kind of encompass what we just talked about or is there? There's a handful of misconceptions. One might be the actual scanning, like that scanning is enough. If I, you know, use proper coding practices and I check all my third-party libraries and there's no vulnerabilities there and I do a lot of dependency tracking, that you're done.

Or if you do that in combination with an app store approval process that, okay, we're all set. We really shouldn't need to consider security at that point. That is a...

A misconception because of what I just explained um oh yeah I wanted to bring up okay yeah and there's another misconception about what if there's any sensitive information in the actual application I have come across a lot of people who say well we don't have anything sensitive in our app we do all our protections on the backend with our APIs. And that may be true. They may not have a lot of sensitive information.

And if they code it right, they don't have a lot of sensitive information. What they don't understand generally is that an attacker doesn't necessarily need to find the same things that you might be worried about or think are a concern. I would say almost any app that has business -critical workflows transactions, logins, dealing with money. I don't really care where the sensitive information is stored.

Something can be learned from decompiling that application. And again, this goes back to that trust and that lack of visibility. You don't know what they're figuring out from that application. That's why you proactively harden that app with code hardening.

And we call it obfuscation to basically scramble up names and things so that you can't reverse this and really kind of quickly figure out the business logic or what's going on with this application. In addition to that, there is a requirement to do runtime checks. And now those are basically checks that can be injected into that application that goes beyond just reverse engineering it.

Into detecting whether or not something about that environment or that runtime execution at that exact moment is not right. There are certain techniques, frameworks, and utilities that attackers will use to maybe change a value, maybe bypass something in the application, and those need to be picked up because those are not necessarily protected by code hardening.

They help prevent that from happening because it's harder to figure out where I might want to go look, but... Those things can still be attached to the application. You could run it in a debugger, you could run instrumentation.

And then once you start doing these things to say maybe hook a value, and what a hook a value basically means is that in simplest terms, if I'm depending on a particular variable to come back to allow myself access to do something, I could hook that variable and basically change the value. So if I can't get in with a false value, then maybe I'll hook it and I'll change it to true.

So there are a lot of threats and risks that are in that mobile application beyond we have no sensitive keys in there or something like that. It's not just the PII from customers. It's not. It's not. I mean, there are probably apps out there that aren't worth protecting, but that's fine. They'll probably know. But I have heard that narrative, which is a big misconception from people.

That work at places that I know for fact are dealing with logins transactions money so that is a dangerous misconception is there a different con security concern between android android ios windows and like what are those if there are I wouldn't say there are different concerns. The landscape, the ecosystem is a little bit different. The same type of attacks can be carried out on both platforms.

The same risks are there. The ecosystem in Android is a little bit different because it's a little less controlled as far as the number of devices that are floating around in the world, the number of operating systems that are still in support. And it is an open source model, but it... That is a misconception as well.

There's a misconception out there that because iOS is closed source and Android is open source that all the problems happen there. There's probably a degree of people, their impression of that being skewed by the fact once you get outside of like the United States and Europe, almost everybody's using Android phones and they're using older ones.

So worldwide, the distribution, not everybody is walking around with $1,000. IPhone in some parts of the world. So essentially, same attacks. You can attack iOS in the same ways that you attack Android. The difference is that that ecosystem is a little bit different and there's many more devices out there that are Android. So what do you think teams assume the OS is protecting them from that maybe it isn't?

Well, it's definitely not protecting them from somebody reverse engineering the application. Or trying to figure out how they're going to bypass their certificate in that application, or it's not going to tell you that another app was downloaded that is malicious and it's going to have a screen overlay, or it's going to abuse accessibility services on your application. The operating system...

Has implicit security, but it's a different focus. This is a different paradigm. It's not, it's concerned. It has some security measures for the actual device and the operating system itself, but it's not specifically concerned with the application or what else is running on there. It would be incorrect. Assumption that the operating system is going to keep me safe. It's like anything in security.

You're making an incorrect assumption if you think anybody else is going to take care of your problems, essentially. It's not the App Stores in their approval process that's going to save you. It's not the operating system that's going to save you. It's not your scans that you run ahead of time to make sure that your dependencies have no vulnerabilities and you're coding well.

The only thing that's going to really save you is to... Have a reasonable level of trust that that application that's connecting to your APIs is genuine and it's not compromised, meaning it hasn't been modified. It's not a fake app. That's the only way to have any level of comfort that you've done what you need to do from that angle. And it makes a lot of sense if you think about it.

In this day and age, you would not ignore some part of your system. And these are, let's face it, those are mobile apps are not apps. They're payment platforms. They're banking platforms, they're medical device managers, they're retail, streaming, entertainment platforms, heavily monetized by these companies. I know it has outpaced the web as far as where people spend their time and handle workflows and transactions.

So if you look at it from the business perspective, these are critical to these organizations, large and small, and those are critical workflows. And there is something to be learned from that application, even if it is not sensitive to data floating in it. Yeah, for sure. And so we kind of talked a little bit about scanning. Teams love to say that we scanned it or we do scan it. Why is that not enough for mobile?

I think so. The way I kind of explained it a little bit earlier is that scanning is going to tell you something about how you might be coding that application or the way you're implementing certain inherent security measures like secure communications and TLS. If you scan the application, it might say, well, it's probably not a great idea to have this secret. Floating around in your application.

So, okay, I'll take the secret out. Or my TLS is improperly configured, so it's actually not doing what it's supposed to do. Scanning can check for your vulnerabilities in third-party libraries. So you're doing your due diligence to, it's like kind of taking care of the basics, to be honest with you.

And those things do not actively or explicitly protect that application from what an attacker is going to do to that application as far as decompile it. Start picking away at it. Figure out exactly where the sensitive parts, the sensitive workflows of that application engage with your backend, essentially. You have to do both, essentially. And that's where it comes back to, like, it's not a narrow scope.

It's not as narrow a scope as just running a scanner. That's too narrow of a scope. And it's ignoring all the other implications and inherent behavior within an application. And it's not protecting you specifically. From what the attacker is going to do. The attackers now are much more organized and much more financially motivated than we saw even like 10 years ago.

And just like everything else, they have at their disposal a large number of pretty advanced frameworks and tooling to actually help them do this faster. It's a bit of an arms race in that regard because we can't control.

The things that are out there that they're going to be able to use, but all those things that are out there help them carry off these specific attacks and also speeds up their time so that they can sort of industrialize the entire thing.

Whereas maybe 10 years ago, it would have been somebody taking a look at this manually, decompiling it and playing around and maybe spending a week, a week and a half trying to figure out what's going on. Now these advanced frameworks and utilities are out there that speed that up very quickly. I'm guessing AI is probably also impacted. Like the time for CVEs and impacts and security vulnerabilities?

Have you noticed like an acceleration over the last like year or so? We, so there's undeniably a bit of a paradigm shift there in the sense that attackers are going to be looking and using AI assistance. And what that will do is essentially speed up the time it takes for them to figure out. The logic and figure out what they want to go after.

We have not seen an increase in our customer base being attacked as a result because our fundamental principles that we have in place are actually the right ones from the beginning prior to the explosion in AI here. And it's going to be true really. In many different paradigms, but especially in the attacker and the defender space, the AI assistance is going to speed up the attacker.

If the defender was not in the right place or not using the right approach to begin with, then they're going to be in a little bit more trouble than someone who has the right fundamental principles in place and multiple layers of...

Reverse engineering static reverse engineering very resilient runtime checks and even a service to basically validate that your application that's connecting to your apis is a genuine non-compromised uncompromised app so yeah it changes the game a little bit but it does not fundamentally change it it gives them some tools it gives everybody some it gives everybody some tools so I was reading on Guardsquare.

Can you like explain the layers to me? Kind of touched on a little bit. There's testing, hardening, runtime protection, and monitoring. How does that work as like one cohesive strategy? Yeah, so the testing is like the scanning aspect, which would look for the vulnerabilities, insecure practices, insecure coding practices, improper implementations.

The hardening is our core product set, which is the decompiler-based or a compiler -based approach to adding protections. Into that application. So we'll take the output IPA or APK, depending upon the platform, decompile that, add the code hardening and obfuscation to basically make that decompiled code look like gibberish to somebody, and also inject what I was referring to as runtime checks.

So, for example, is this device rooted or is someone attaching a hooking framework? Has somebody gotten around the actual certificate? Or is there malware on the device that's trying to abuse accessibility services or using overlay or a screen reader to scrape information? So that's the hardening aspect.

And then the monitoring aspect is another important part of this broader picture, which is, I spoke about this earlier, where that whole trust model, that lack of visibility. Even if you are protected, you still want to monitor what's going on because things shift. You might see all of a sudden evidence of certain behavior in a certain part of the world.

So maybe you want to pause for a second and be like, hmm, do we want to change our configuration, release a new application, maybe improve some of the protections here? Like what's going on? So it's not just, you can't just add your protections and just, I'm done. I'm good. You should probably. You should be monitoring that and essentially allow yourself the flexibility to change your posture as needed, essentially.

So, OK, keeping the like CI/CD framing, we talked about the runtime protection, the obfuscation. Does that create any like or what is the build overhead from that? And does that slow builds down? How do you keep security checks from like killing the velocity of like normal app development? That's a good question. I mean, that's always a concern in any paradigm of security, no matter what it is, because.

It doesn't come for free. Like you, if you're going to do it correctly, but there's always a line between protection and usability. You don't want to, you know, put a motor on your house and a drawbridge because you're going to end up leaving the drawbridge down because you really don't want to wait for all that time for it to go up.

So yeah, there's always a trade -off there where you have to monitor and take care of the constraints. So you want to protect as much as you can without infringing upon your constraints. People might have constraints about the size of their application. They might have specific performance constraints.

And these things will always sort of shine brighter than the protection because the fundamental job of all these teams is to release this application with features that drive business and bring in revenue and they can monetize their services. So there is a balancing act. It's a unavoidable balancing act.

But if your vendor has the right tooling and options and features, it shouldn't be a pain that's not worth, you know, accepting or a penalty that's not worth accepting. Yeah, it's a fundamental part of adding protections. But if you go about those protections in the right way and use the right tooling and understand what you're trying to protect, it should be a minimal impact to your delivery process.

So, okay, with that in mind, what would like a good or a sane gating strategy look like? Like what would block a release versus what becomes tracked risk for down the road? As far as testing the application is concerned? Yeah, or vulnerability tracking or whatever. Like how does that impact the CI/CD work?

Depending upon the severity of the vulnerability or the insecure coding practices, it could be a high risk thing that should be mitigated immediately. But a lot of scanning tools will also surface maybe medium or low risk things that you can just kind of track and be like, okay, I'm going to put that on my wishlist.

And next time around, you know, we're not going to block the release over this because it's not severe. It's not a high risk situation. And then the lower ones, you can basically make a list. That's what people do. And then sort of when they have time, pluck away at a few of those because those are not critical, critical risks.

From the testing standpoint of a protected application, going back to The usability, it cannot impact the usability and whatever those constraints might be. So it has to be protected and then verified by the QA team that, okay, we're still within the constraints of acceptability. And then if it's not, maybe there's a change to the configuration or maybe you're being too aggressive.

On certain aspects of the protection, in which case you can change your posture there and dial it down. If you don't have evidence of past attacks, you're not going into a high risk, a really high risk known malicious situation, maybe you could gamble a little bit and take a step back on doubling down on those protections for the sake of that usability. But there is a sweet spot, really.

I mean, essentially, you know, you don't want to, I probably didn't give the best advice there. You really don't want to look at it that way. You want the sweet spot, which is. I'm reasonably well protected against all of these things with minimal to no impact in the usability. Certain things are acceptable as far as impact. Certain things are not.

It's not acceptable to cause a stability issue, but it might be acceptable to lose a couple milliseconds on a particular transaction or some sort of a method or something for the sake of the assurance that it's now secured, right? So it's a bit of a trade-off there. What's important to you from a protection standpoint, and also what is important or critical on the constraint side.

But those are essentially the only things that would really just like stop the release. Like I have high severity, critical things that need to be fixed. And that comes down to the policies, the internal policies, compliance policies, and governance of that organization, that they make a rule that, and they should make a rule that there's a high severity finding and scanning. It's a stop the press thing.

And usually those things shouldn't be too difficult to fix. And then as far as protections, you know, it's just like any release cycle. You add it, even if it's not a protection, you still have to test your application. So when you integrate the protection into your CI/CD, it's not like you're going down a different path.

You're essentially just running it through your system tests and your QA team with the protections. So if you're an engineering team and you're trying to improve mobile app security posture today, is there like a top three or top five things that you would start with? Yeah, I would say not to get overwhelmed. I would take a step back and start asking some really basic questions. What matters to us?

How is this app interacting with our backend? How could this, what could possibly hurt us? Like start brainstorming around that and then consider a structured programmatic approach to layering in your security essentially so that it becomes part of your routine. Can't be an after the fact type of thing. You should be thinking about it up front, during, and obviously at release and after release.

So it's a bit of a mindset. If you treat it as a checklist type of thing at the very tail end, that's where you usually get yourself maybe into some trouble because you either didn't protect it well enough or you did it in a way that... Might risk some of your constraints. So it's a mindset that security is top of mind across all facets all the time.

And if it's part of your daily routine, just like exercising or something, if you make it a habit, you're going to do it all the time. If you're considering security as being important, as you should, it just becomes, it's an adoption and understanding that this is part of life. And it's the right way to go about it because you'd rather not be.

In the news because someone made an example of your application or some of your users were victims of fraud or even your organization was part of some sort of criminal misuse situation, which does happen. Yeah. So kind of thinking of the inverse in that, is there anything that you would kind of throw out as security posture that maybe companies are doing that maybe they shouldn't be?

It's kind of not effective, not really providing value from a security perspective. I mean, the worst thing they could possibly do is the scenario we were talking about earlier, which was basically just assuming that some level of scanning is enough and App Store acceptance is sufficient.

As far as other things, I think people probably are not generally aware enough of the ways attackers go about reverse engineering applications. So it's hard for them. Maybe it's probably more a situation of... Misunderstanding or like not being knowledgeable enough.

If you don't know about something, you have no idea the importance of code hardening or changing names, then you're probably not going to value it at all. So I don't know if that answers your question or not, but I would certainly, and there's plenty of information and knowledge out there. You can go to our website. There's organizations like OWASP that have lots of guidance around.

What is needed to properly secure a mobile application. And I would recommend that highly because it's really like anything else. If you don't really know what you're doing or you don't really understand the fundamentals, like you're probably not going to be doing this or you're probably not going to pick the right vendor. You're probably not going to use that vendor the right way. Yeah, it's an education thing.

Awesome. So where can people find more about you, about Guardsquare? So www.guardsquare .com, we have a wealth of knowledge on there, like resources and blogs. And they're not just, you know, for lack of a better word, marketing fluff. They're written by security research analysts that work for us.

And they'll give you a broad and, if you want, a deep level understanding of the protections that are needed and the techniques that are attacks attackers are essentially using. We also have a security research center on the website. So there's a lot of research.

There's a lot of resources and papers to get a basic knowledge and a good understanding of what you might need to do to properly secure your mobile application. Cool. Awesome. I'll put links up for Guardsquare's blog, OWASP, and some of the other things that we talked about in the show notes. Sure. Joel, thank you so much for coming on. Really appreciate it. Appreciate it, Brian. You're welcome. Thank you. All right.

That was my conversation with Joel DeStefano from Guardsquare. My biggest takeaway from this one is that mobile security is really about the trust boundary moving. In backend and cloud systems, we spend a lot of time trying to control the environment. We harden infrastructure. We manage IAM. We monitor runtime behavior. We control deployments. We log everything we can.

We build policies and guardrails and assume that if something weird happens, we at least have some visibility into the system. With mobile, that changes. You ship the app, and now the app is out in the world. And once it is out there, attackers can study it on their own time. They can decompile it. They can instrument it. They can run it in weird environments. They can try to bypass checks, hook values.

Scrape information, or learn enough about the workflow to abuse the backend in ways the backend team may not have expected. That is the part I think is worth paying attention to. A lot of teams treat mobile security like a scanning problem. Run the scanner. Fix the dependencies. Fix the obvious insecure coding patterns. Make sure there are no secrets sitting in the app. Pass the App Store review. Ship it.

And to be clear, that work matters. But it is not the whole story. Scanning tells you something useful before release. It helps you catch problems in code, dependencies, configuration, and implementation. But it does not protect the app once someone is actively trying to reverse engineer it or tamper with it in the wild. That is where the layered approach matters. Testing matters. Hardening matters.

Runtime protection matters. Monitoring matters. And honestly, the monitoring part is easy to underappreciate. Because even if you do the right things before release, the threat landscape changes after release. Attackers change tactics. Activity spikes in certain regions. New tools get popular. Something starts showing up in the field that you did not plan for. If you are not watching that, you are basically guessing.

I also liked Joel's point that we do not have sensitive data in the app is not the same as the app does not need protection. That is a trap. A mobile app does not need to contain a database full of customer PII to matter.

If it handles login, payments, banking, medical workflows, retail transactions, subscriptions, account recovery, loyalty points, streaming access, or any business-critical path, then there is something useful for an attacker to learn. The sensitive thing may not be a secret string. It might be the workflow. It might be the trust relationship. It might be how the app talks to the backend.

It might be the condition an attacker needs to bypass to make the backend believe something it should not believe. That is a different way of thinking about the app. Not just what data is stored here, more like what can someone learn from this app and how could they use that knowledge against the system? The CI/CD part matters too, because security teams can say protect everything all day long.

But if the protection adds too much pain, slows builds too much, creates unstable releases or turns every app update into a fight, engineering teams will find a way around it. So the goal has to be sane integration. High severity findings should block. Critical issues should stop the release. But lower severity findings may become tracked risk. Runtime protections need to be tested like any other release change.

Performance and stability constraints still matter. There is always a trade-off between protection and usability. That does not mean avoid security. It means design it like it has to live inside a real delivery process. Because it does. The other thing that stood out is the AI angle. AI may make attackers faster. It may help them understand logic faster. It may help them reverse engineer or reason about code faster.

But Joel's point was that the fundamentals do not disappear. If you already have the right layers in place, AI does not completely rewrite the model. It increases speed. It gives attackers better tools. It raises the pressure. But the principles still matter. Know what matters in the app. Protect the app. Make reverse engineering harder. Add runtime checks. Validate that the app connecting to your backend is genuine.

Monitor what is happening in the wild. And do not assume that the OS, the App Store, or one scan in the pipeline is going to save you. That is probably where I would land with this whole conversation. Mobile security is not magic. It is also not just scan it and ship it. It is a different trust model and teams need to treat it that way.

I'll have links to Joel, Guardsquare, OWASP mobile resources, and anything else we mentioned in the show notes. If you enjoyed this conversation, follow or subscribe to Ship It Weekly wherever you listen to podcasts. It helps the show and it makes sure that you get both these conversation episodes and the weekly DevOps. SRE, Platform, Cloud, and Security News Recaps.

You can also find the show notes and links over at shipitweekly.fm. Thanks for listening, and I'll see you later this week.

PeopleSoft gets hit by a zero-day. Npm changes what code is allowed to run during install. GitHub removes long-lived personal access tokens from agentic workflows. And Anthropic reminds everyone that your AI dependency can disappear for reasons that have nothing to do with uptime. The theme this week is simple. Default trust is getting punished. Old enterprise systems were trusted. Because they were internal.

Package installs were trusted because that is just how the ecosystem worked. CI/CD tokens were trusted because they were convenient. AI models were trusted because the API worked yesterday. But production does not care why you trusted something. It only cares what that trust was allowed to do. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, the show where we look at the DevOps, SRE, cloud, platform, and security stories that actually matter when you are the person who eventually has to keep the thing running. This week is about default trust. We're starting with Oracle PeopleSoft and a zero-day that was reportedly exploited before Oracle released an emergency patch. Then we'll talk npm v12.

Changing install-script defaults which is great for supply-chain security and maybe a little annoying for some CI pipelines after that we'll get into github agentic workflows no longer needing personal access tokens which is the kind of boring credential change that actually matters then we'll talk about Anthropic disabling fable 5 and mythos 5 after a us export control directive because AI model availability is now also a policy-risk dependency.

In the lightning round, we'll hit Tekton CloudEvents moving to a dedicated events controller, NVIDIA Triton 26.04 changing inference server defaults, AWS Nitro Isolation Engine using formal verification, and Homebrew 6.0 requiring explicit trust for third-party taps. The human closer this week is about trust defaults, because a lot of incidents start with one quiet assumption. The system is internal.

The package is fine. This token is safe. This model will be there tomorrow. And sometimes that assumption is the real production dependency. So let's get into it. Oracle issued an emergency advisory for CVE-2026-35273 in PeopleSoft PeopleTools. And yes, I know, PeopleSoft does not sound like the kind of shiny cloud-native story that makes people sprint to the podcast app. But that is exactly why it matters.

A lot of modern companies still depend on older enterprise systems for HR, finance, payroll, student records, supply-chain. And all the boring data that becomes extremely exciting when attackers get it. Oracle says that the vulnerability is remotely exploitable without authentication and can result in remote code execution.

Public reporting says exploitation was observed before Oracle's emergency advisory, with ShinyHunters linked activity targeting mostly U.S. Organizations, especially higher education. So the shape of the story is pretty clear. Old enterprise application, sensitive data, remote unauthenticated exploit, active exploitation, emergency patch. That is a bad combination.

And the lesson for DevOps, SRE, platform, and cloud teams is not just patch PeopleSoft. Although, yes, obviously, patch PeopleSoft. The bigger lesson... Is that the enterprise app tier is still part of your perimeter, even if everyone emotionally moved on to Kubernetes. A lot of teams have a weird relationship with older enterprise platforms.

They are important enough that nobody can turn them off, but old enough that nobody really wants to own them. They live in a special network zone. They have a special upgrade window. They have special vendor requirements. They have special firewall rules. From 2017 that everyone is afraid to touch. And somehow, because they are enterprise apps, people stop thinking of them like internet-exposed production systems.

Attackers do not care what generation of architecture the system belongs to. They care whether it has data, access, and a path in. PeopleSoft may not be the cool part of your stack, but if it holds HR data, finance data, student data, employee data, or business workflow data, it is very much part of your attack surface. This is also a good reminder that legacy does not mean low-priority.

Sometimes legacy means high-value, hard-to-patch, badly isolated, and full of data. That is not low-priority. That is where an attacker brings a shovel. The practical takeaway is simple. Know where these systems are. Know whether they are internet exposed. Know who owns patching. Know who owns logs. Know what normal access looks like.

Know whether unusual agents, web shells, or scheduled tasks would actually be noticed. And if you are a platform or cloud team that thinks this is someone else's problem, be careful. Because during the incident call, someone else's problem has a funny way of becoming why did nobody know this existed. Your ERP systems, HR systems, finance systems, and admin portals are production systems. Treat them that way.

Second story. Npm v12 is changing some defaults around install scripts. And this one matters to developers, platform teams, CI maintainers, security teams, and anyone who has ever watched an npm install do 12 things they did not explicitly ask it to do. GitHub says npm v12 is expected in July 2026. And one of the big breaking changes is that allowScripts will default to off.

That means npm install will no longer execute pre -install, install, or postinstall-scripts from dependencies unless they are explicitly allowed. Prepare scripts from git, file, and link dependencies are blocked the same way. These settings are already available in npm 11.16.0 for migration testing. This is a big ecosystem shift. For years, npm install has been more powerful than people like to admit.

You run the package install. The package runs code. That code might build native bindings, download binaries, generate files. Patch something, bootstrap something, do the magic that makes everything work. But that same install-time execution path is also a supply chain attack surface. A malicious package does not need your app to import it at runtime if it can run code during install.

It only needs to land in the dependency tree. Then the CI runner, developer laptop, or build machine does the rest. This is why this change matters. It moves the ecosystem from implicit trust towards explicit trust. And yes, that is good. But some builds are going to break. Because a lot of pipelines have been relying on install-time magic without documenting it. Native modules. Post install downloads.

Git dependencies. File dependencies. Link dependencies. Internal packages. Packages that compile. Packages that generate code. Packages that do one tiny little install step that nobody has looked at since the person who wrote it left the company. Those assumptions are about to become visible. And visibility is great. Right after it ruins your Tuesday. The practical takeaway is not panic about npm v12.

The takeaway is to test early. Run builds with the new behavior before it becomes the default. Find which dependencies need scripts. Decide which scripts you actually trust. Document why. And do not just flip everything back on because one pipeline failed. That defeats the point. This is a chance to clean up dependency trust, especially in CI. Ask what install-scripts run today.

Ask whether CI runners have secrets available during install. Ask whether build steps can reach internal services. Ask whether developers are running installs on machines with cloud credentials, SSH keys, or package publishing rights. Because the blast radius of an install-script is not just the package. It is whatever the install environment can access. I like this change.

It is going to annoy people, but sometimes the annoying default is the correct default. Npm install is finally getting less magical. And that means that some teams are about to discover how much magic they were depending on. Third story. GitHub says agentic workflows no longer need a personal access token. And yes, that sounds boring. But boring credential changes are often the ones that save you later.

Previously, agentic workflows needed personal access tokens in certain setups. Now GitHub says, These workflows can use the built-in GITHUB_TOKEN and the copilot request write permission instead. The point is to avoid creating and storing long-lived personal access tokens for agentic workflows. This matters because agentic CI/CD is already risky enough.

These workflows can read context, generate changes, open pull requests, review code, call tools, interact with repo state, and eventually, in some environments, they may become part of release, remediation, triage, or automation paths. So the credentials they use matter a lot. A personal access token is often tied to a person. It may be long-lived. It may have broad permissions. It may sit in secrets.

It may get copied between repos. It may survive longer than anyone intended. It may still exist after the workflow changed, the person changed teams, or everyone forgot. Why the token was created. That is not a great pattern for agentic automation. A scoped workflow token is not magic. You still have to configure it correctly. You still have to review permissions.

You still have to think about which events can trigger the workflow. You still have to worry about untrusted input. But it is a better default than handing an agent a long-lived personal credential and hoping everyone remembers where it lives. The platform takeaway is simple. If you are experimenting with agentic workflows in GitHub, review the whole trust model. What repos can run it? What events can trigger them?

What permissions do they get? Can pull request content influence the agent? Can issue comments influence the agent? Can the agent write code? Can it approve changes? Can it call external tools? Can it see secrets? Can it publish artifacts? Can it affect production? And are you still using personal access tokens where you do not need them? Because agents are not just assistants once they enter CI/CD.

They become automation services. And automation surfaces need boring, strict, reviewable credentials. So yes, use the better GitHub auth model. But do not stop there. If it can change code, trigger work, or spend money, it belongs in your production automation threat model. Fourth story. Anthropic published a statement saying that the U.S.

Government issued an export-control directive requiring Anthropic to suspend access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees. Anthropic said that the net effect was that it had to abruptly disable Fable 5 and Mythos 5 for all customers. To ensure compliance. Access to other Anthropic models was not affected.

Now, this is not a normal SRE outage. There was not a regional failure. There was not a bad deploy. There was not a database incident. There was not a capacity problem. But for customers who depend on those models, the effect is still operational. The dependency disappeared. And it disappeared for policy reasons. That is the lesson.

If AI APIs are part of your production path, dependency risk is no longer just uptime, latency, quota, and cost. It is also model availability, geography, export-controls, regulatory shifts, vendor policy, safety policy, commercial access, model deprecation, compliance interpretation, and whatever new category we invent next quarter.

This is uncomfortable because AI dependencies can be harder to substitute than people pretend. You can usually swap one HTTP endpoint for another at a code level. But can you swap the behavior, the context window, the latency, the tool calling behavior, the moderation behavior, the cost profile, the prompt format, the eval results, the customer experience, the internal workflow?

The fallback may not be use a different model. The fallback may be a degraded mode, a smaller model, a cached response, a human review queue, a simpler workflow, a feature flag, a regional restriction, or a message that says this capability is temporarily unavailable. Boring fallback behavior is better than waking up to discover your product path depends on a model that is no longer available to you.

The practical takeaway is to add AI models to your dependency register like any other production dependency. Track which features use which model. Track whether the model is in a critical path. Track where your users are. Track fallback options. Track whether you can degrade gracefully. Track whether a vendor-side access change would be an incident. And test the fallback. Do not just write...

Fall back to another model in a doc and call it done. Actually try it. Because the weird thing about AI infrastructure is that even when the API shape looks portable, the behavior is not. Your AI dependency can disappear for reasons that have nothing to do with uptime. That means your reliability plan needs to cover more than outages. Now let's do a quick lightning round.

First, Tekton pipelines changed how cloud events are sent. Cloud events for PipelineRuns and TaskRuns now move through a dedicated Tekton events controller instead of the core pipeline run and task run controllers. That means operators need to make sure the tekton-events-controller deployment is actually running. Small looking platform changes like this can have real operational impact.

If CI/CD observability, event routing, notifications, or automation depends on Tekton events, make sure the new controller is deployed, monitored, and part of the upgrade checklist. Because the pipeline ran, but the event never showed up, is exactly the kind of thing that sends teams into Slack archaeology. Second, NVIDIA Triton Inference Server 26.04 has changes operators should read before upgrading.

Client shared memory is now disabled by default and must be enabled explicitly if you rely on the old behavior. Triton also enforces max inflight requests as a shared limit across ensemble requests, which is meant to prevent unbounded queue growth in ensemble pipelines. The short version is this. AI inference servers are now normal platform infrastructure. Defaults change.

Queue behavior changes, memory behavior changes, model control changes. So upgrade them like production systems, not sidecar experiments. Third, AWS published a deep dive on the Nitro Isolation Engine. AWS says the Nitro Isolation Engine is generally available on Graviton5-based instances and uses formal verification to demonstrate isolation properties.

That is interesting because cloud isolation is usually something customers trust, but cannot directly inspect. Formal verification does not mean bugs are impossible, but it does show where cloud providers are heading. Smaller trusted components, tighter isolation boundaries, and mathematically checked properties for the parts that protect customer workloads. Fourth, Homebrew 6.0 introduced tap trust.

Homebrew now requires explicit trust. For third-party taps before evaluating or running their Ruby code, while official taps remain trusted by default. This is another supply-chain default, changing from sure, run it, to are you sure you trust this? And that is good. A third-party tap can contain arbitrary Ruby.

If your developer laptops, CI runners, or bootstrap scripts use Homebrew, This change is worth knowing about. Some installs may prompt or fail until trust is explicit. That might annoy people. But again, that is the point. Developer tooling is part of the supply-chain. And fewer things should get to run code just because they were convenient. The human closer this week is about default trust.

PeopleSoft was trusted because it was an enterprise system that had been around forever. Npm install was trusted because package managers have always done some magic. Agentic workflows were trusted with personal tokens because that was the easy path. AI models were trusted because the API was available yesterday. Tekton events were trusted to keep flowing. Triton defaults were trusted until they changed.

Cloud isolation was trusted because that is the cloud model. Homebrew taps were trusted because developer tooling has historically been pretty casual about running code. Different stories, same pattern. Production risk often hides inside the trust defaults nobody has revisited. The system is internal. The install-script is normal. The token is convenient. The model will be available.

The event will fire the tap is probably fine those assumptions may be true until they are not so the practical question this week is simple what are you trusting by default which enterprise apps are still exposed which package installs run code in ci which agents still use long-lived tokens which ai models sit in production paths which developer tools can execute code before anyone reviews it that is platform work

Now not just building the shiny new thing but revisiting the old trust assumptions before attackers vendors regulators or outages revisit them for you find the defaults then decide whether they still deserve your trust That's it for this week of Ship It Weekly.

We covered the Oracle PeopleSoft zero-day, npm v12 install-script changes, GitHub agentic workflows moving away from personal access tokens, Anthropic's Fable and Mythos model suspension, and a lightning round on Tekton, NVIDIA Triton, AWS Nitro Isolation Engine, and Homebrew 6.0 tap trust. If this episode was useful, follow or subscribe wherever you are watching or listening. If you are on YouTube, hit subscribe.

If you're in a podcast app, follow the show there. And if you know someone dealing with enterprise app security, CI/CD supply-chain risk, agentic workflows, AI dependency planning, or developer tool trust, send this one to them. It helps the show grow, and it helps me keep making this kind of content for people who actually live with these systems.

You can find the weekly brief at OnCallBrief.com, and more episodes and show notes at shipitweekly.fm. I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, production does not care why you trusted the default. It only cares what that default was allowed to do.

AI is making it easier than ever to create more software, more code, more diffs, more experiments, more changes moving through systems that were already complicated before everybody got a robot assistant in the editor. And that sounds great until you are the team responsible for production because production does not really care that the code was generated faster. It still cares about latency.

It still cares about overload. It still cares about dependencies, rollbacks, traffic routing, region failures, weird edge cases, and whether the people on call actually know what to do when the system starts acting strange. That is the part of the AI conversation that feels under-discussed to me. Not just can AI write code, it can. Not just can AI help debug, sometimes yes.

The better question is what happens to reliability when the volume of change goes up, the amount of generated code goes up and the context behind that code gets thinner. Because incidents are not just caused by bad code. They are caused by systems behaving in ways people did not expect, under conditions they did not practice, with dependencies they forgot were part of the critical path.

That is really what this conversation is about. Not just SLOs, not just incident reviews, not just AI agents. More like, what does reliability actually look like when change is accelerating? Systems are more connected, and recovery matters just as much as prevention. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick news recap. In between those, I do conversation episodes with people who are building platforms, running infrastructure, organizing events, and thinking through where this industry is actually headed.

Today is one of those conversations. I'm joined by Francois Richard, an engineering director at Meta. We're talking about reliability at scale, how AI and automation are changing production risk, what teams actually learn from incidents, and why recovery practice matters just as much as prevention.

And I like this conversation because it gets past the neat, clean version of reliability that fits nicely in a dashboard. Reliability is not just a number. SLOs matter, dashboards matter, alerts matter, guardrails matter. But none of that means much if the team cannot use it to make better decisions, recover from real failures, and improve the system after something breaks. You start with the obvious goal.

Keep the system up. Keep the users happy. Avoid incidents. Do not page people for nonsense. Do not let every deployment feel like a coin flip. Then reality shows up. A region has problems. A dependency gets slow. A service gets overloaded and cannot restart because it is too overloaded to recover cleanly. Traffic spikes during a major event. A change rolls out faster than the team understands it.

Or an AI-generated pile of code technically works until it fails in a way nobody has enough context to explain quickly. And somewhere in the middle of that, reliability stops being just prevention and starts becoming practice. In this conversation, Francois and I talk about how Meta thinks about reliability across both reactive and proactive sides.

Incident response, incident reviews, SLOs, guardrails, validation, disaster recovery testing and what happens when you actually practice taking a region out instead of just assuming the failover plan works because the diagram looks good.

We also get into what teams learn during incidentsss, not just from the post postmortem afterward but inside the pressure cooker itself where engineers have to make decisions quickly build consensus fast understand the system under stress and figure out what is actually happening while the clock is running there is also a good thread in here around AI agents in incident response not the fantasy version where you hand

Production to an agent and hope it saves the day more the practical version where AI helps with investigation telemetry metrics logs relationships across services and narrowing down what might be happening faster than a human clicking around dashboards alone and towards the end we talk about recovery practice known failures versus unknown failures why teams should test the failure modes they claim they can survive

How smaller teams can learn from Meta-scale reliability patterns and why not every system needs six nines on day one so if you work around DevOps, SRE, platform engineering, infrastructure, engineering leadership, incident response, or you are just trying to figure out what reliability looks like when AI is increasing the volume of change, this one should be worth your time.

All right, let's jump in. Today, I'm joined by Francois Richard. He is an engineering director at Meta, and we're talking about reliability at scale, how AI and automation are changing production risk, what teams learn from incidents, and why recovery practice matters just as much as prevention. Francois, thank you for joining me. Thank you, thank you. Happy to be here.

Okay, so when you think about reliability at Meta scale, where does the work actually start? I think for me, reliability is actually two things. What people kind of like know usually it's the reactive side is how do we manage incidentss? How do we respond to problemss? And how do we handle things and recover and fall back when something goes bad?

But at the same time, it's also everything that I call like, you know, the proactive side of the house. This is everything that has to do with, hey, how are we going to prevent reliability problems from happening? What sort of guardrails do we put in place? What sort of framework? What sort of tooling do we allow platform or do we give to platform services and application developers to use?

And also, what is the real validation that we are doing? There's a lot of papers available in the public and so on around like, you know, there's something that we call like the storm program.

And it's basically our way to test like, you know, disaster recovery, where once in a while we take an entire region out, out of the like, you know, like if there was an earthquake or there was like a giant electrical grid failure. And we test what happens with the system. How do they recover? How do they handle the spike in traffic and so on?

So it's both like for me, reliability is both like, you know, the reactive side and also the proactive side. Is that kind of like chaos engineering, the concepts? Yes, yes, yes, exactly. And, you know, we go as far as, you know, disconnecting a complete region from the map. Interesting. Okay, so speaking about reliability, what makes an SLO useful instead of just another dashboard number?

Like, how do you think about that? Yes, an engineer, like a senior engineer that I work with always says that the SLO of something, of a system, is actually the promise that you're making to your customer. That, you know, XYZ should work 99.99% of the time and you should expect that amount of latency on average at, you know, P90 or something like this.

So we use this, like, you know, most of the platform we operate at Meta and also the product services and so on. They will all, the big ones, will all have some form of SLO for the top APIs or the top services that they are doing. We have some alerts that will trigger on SLO, but usually this is too late. Like, you know, we want to catch problemss.

The SLO allows us in the long run to actually figure out like, hey, are we investing enough in reliability? Are we keeping the bar high enough? Or do we have wiggle room to take some risk, introduce new features? Because you all know that when you introduce or you do a giant migration, there's a period of time where it takes time to stabilize things. So the SLOs are helping us make these decisions in the long run.

And you can clearly see when you've been tracking them for a while that some systems are trending up and some systems are trending down. And for the ones that are trending down, it allows you, it gives you the argument and the data to start investing more.

So this is kind of like, you know, our way about deciding if we have enough investment in an area or if, you know, we are okay and we can take more, you know, progressive risk. So you had, we talked a little bit about incidents before the show, and then you had just mentioned incidents. Across the incident lifecycle, Where do teams actually learn the most?

So in terms of learning during incidentss, I think it happens in two phases. First, it happens while handling the incident during what I call like the pressure cooker. When you actually need to handle the incident, you actually need to make decisions quickly, build consensus fast with a small group of people about what's the next action to take.

And then what are the bypass, the restart process, which system is calling what. This is actually a time where a lot of people, like junior engineers and senior, they learn a lot. They learn a lot about the system, but they also learn a lot about team dynamics and themselves during that period. Then the next phase happens when kind of like you build the...

Incident review report or we call it the SEV report, where you basically, you know, once the incident is handled, mitigated, you know, a few days after, you basically sit down and then we have like a tool to do this. And then people sit down and they write a report around like, you know, hey, what exactly happened? What was the root cause of this?

What was the exact timeline, which is kind of like forcing people to go back into logs and all of this? Could the alerts have fired earlier? Did we have the right instrumentation? Did we diagnostic, like diagnose this thing, like, you know, the proper way? And, you know, what are the follow -up tasks, like to make sure that this doesn't happen again?

Or that we have some level of automation to be able to remediate this type of problem automatically. So you build that report. But then from there, you go into a series of reviews. You review it with your team, with your organization. And if the incident was severe enough, we actually reviewed them at the company level. And during that time, it's a big opportunity to get feedback from other teams.

From other senior engineers that, yeah, we see this pattern. Here's, you know, how we handle this type of thing. And you actually learn a lot during that process. So this is usually like, you know, the learnings happen in kind of like two phases. Yeah. And so given that learning, what should an incident review actually produce? Like once the incident's over? The incident review, like we have a report type of thing.

It's a template. It has like, you know, walks you to, for you to get like two. The root cause of it. But I think the most important thing that it produces is also like, you know, what are the follow-up actions? The follow-up could be like, hey, we need to implement more redundancy. We need to implement better diagnostics. We need to implement better ways of handling these exceptions or problems.

We need a different deployment process. This is usually like, you know, the best outcome we could get because, you know, the more you mature your system, the more you'll be able to cover a lot of these, like, kind of like unknown slash unexpected situationss. So this is usually like the good outcome of these. And what do you find is the difference between finding the cause versus improving the system?

The difference between the cause and improving the system. Or is there a difference? There is a bit. Because for me, finding the cause overall is, do I have the right tools to pinpoint the problem very quickly. And then I'll be able to find the cause. And then the improvements are more are more about eliminating this exact problem or that class of problem.

And the improvement can be that like you know, like from a code structure that this will always happen, but you can work around with either deployment strategy, automation of recovery and things like this so that it's... It's not visible to the end users with strategies of retries and things like that. So there's a big difference between finding the cause.

Like if you work in mobile apps, you know that there's going to be a couple of cases where like the cause is going to be like, you know, the mobile networks just drop on you. And the app needs over time to come up with ways to mask it. You'll never eliminate the cause, but you'll have apps that are a lot better at masking this type of problem. So it's slightly different. Okay, so let's switch gears to AI agents.

Are you using AI in incident response at all? Yes. In triaging? We use it for what I would call like investigation. So, you know, there's usually like two classeses of production incidentss. You have one class when it happens. You kind of know what this is. Like, you know, it's a pattern that you already recognize. The alarms or alerts are very clear about what's happening.

So you kind of know and you can narrow down and then you can just go focus on, hey, you know, how do I remediate that? There are some of the cases like where you kind of see like, oh, I'm starting to see transient errors. I'm starting to see latency creeped up and where it's kind of like, I'm kind of not sure. And then usually like, you know, we launch what's called like investigations and things like that.

Really, the AI agents are helping us in investigation because they can analyze and gather large swath of data very, very quickly and then pinpoint relationships between datasets. Like, you know, it's probably the same for everybody, but we have multiple layers of software on top of each other. They call each other and they call back.

Sometimes finding complex situation requires us to analyze a lot of data at the same time. Even myself personally, like, you know, we have like centralized, like, you know, monitoring system.

And then if you aggregate a lot of data with very small granularity, like, you know, at the 10 second or like at the minute level, and you have a lot of time series, not the data point, it gets a pain to actually like get the right query and the right graph for the problem that you're investigating at a point in time like so the AI agent has kind of like shifted this thing where they help us find these things and

Even like you know in some cases like what I do is like I will dump all the metrics locally in the SQLite database and then like do all sorts of things because it's local and my SSD can do it a lot faster And then you use also the AI agents to help you craft the queries.

I don't remember on top of my head all the various flavors of joins and all of that. So it's helping a lot. So with investigation, it helps a lot. It's helping a lot. It's not perfect. You still need to guide it very well because the topology of the system, the request... The request flow between the system is still not something that I will call it like the various AI agents are actually mastering.

They master code very well. But these flows are a little bit more difficult and understanding how they map into the application or the product flows is even more complicated. But it does help. That's the investigation side of it. We're doing a lot also into what I call like having a standard dataset across the company that kind of like represents the state of reliability.

Like we drive a couple of initiatives globally, like we call it like change safety. Do you have SLOs for something? Do you have guardrails?

And with these datasets being kind of official, then it's easy for each service team or product team to build their own customized kind of dashboard like right now the cost of making dashboards is almost free and really tailoreded to their use cases or to their product using the standard dataset and that's another place where it's been like oh, this has been great.

And how are you, I'm assuming you're still validating that that root cause though right if it comes up with a you know it uses MCP servers it's reaching out to Argo Kubernetes whatever it's gathering all the telemetry from Grafana and then it's putting that together and it's saying, hey, this pod was crash loop back off. This caused this issue. Are you still validating that?

And what is that process for validating that? Like, is there a standard SOP that you're following? No, I don't think we have like a standard evaluation.

The evaluation, like most of the time, the tools that you have in the moment from an incident is either you move away from the problem or you roll back the problem so we will try like you know first you know move away let's call it like a problem is a backend problem it's isolated to a region we'll move away from that region to kind of like prove that theory and then we'll probably during that time roll back the

System or the backend or whatever it was that introduced that problem and then you can imagine that we can then now that the hypothesis is live then we can shift a portion of traffic back to that problem and see like oh is it still happening or not this is all manual we have some of these process that are automated but they really boxed into very very well known like you know they had like very well established

Runbook that have been like you know in place for the longest time that we know that when this happened do this like it's almost like if then else But the most like the investigation result, the validation is more to roll out the fix, you know, driven by human with like, you know, slow roll type of techniques.

Do you think that AI is increasing the speed of change, the volume of change or the type of failure that teams need to expect? But it's clear that it's increasing the volume of change. We have data internally that, you know, the number of diffss is through the roof. The number of lines of code changed is, again, through the roof.

The type of failure is like one thing that, because there's more code and it's going faster, it's the type of failure becomes more into the, what I would call like pseudo-unexpected. When a product developer, application developer, when they... Roll out a new feature, they do understand the business logic.

They do understand how it should show to the user, what they expect the user to do, where's a corner where a user could get stuck. They understand that. That's not a problem. But if you have to build on modern apps on mobile phones or on backend, A lot of these things are built on top of frameworks. There's a lot of boilerplate stuff, especially to handle like, you know, async callbacks and things like this.

And since a lot of that code gets auto-generated, people are losing context around these things. And when something doesn't work, we realize that it takes us more time to understand why it doesn't work. Because there was not that additional context that...

The developer or that team developing that feature took a lot of time to craft it and a lot of like internal reviews and things like that so so it goes faster out but that context on how it's built is actually lost along the way and this is basically what we need to reconstruct live and again like I said earlier a lot of our tooling is like you know in this case we're just like we'll roll back and you figure it out

Like bring back the system to a stable state and then you can figure it out on your own time without being on the pressure cooker.

Okay, so speaking along the same lines, you had mentioned too that recovery comes down to practice. So what does good recovery practice actually look like? You know, there's a set of standard failuress you expect your app to have. And it starts at, you know, if you're running, if you have data centers or region in the United States or in Europe or something like that, you get...

Tons of things like fiber cut in hurricanes and, you know, power grid and things like that. You need to understand how your system will react.

Like, you know, a simple thing, like if I'm a startup and I run into a AWS region, but I actually never test the fact that one of them will go down, I can guarantee you, like, as soon as you get a little bit of complexity in your system, something will not work as expected in there.

For us is actually to do, to validate it, to not just test, like we run, I would call it like data simulation and analysis in advance to figure out like, okay, if we lose that data center, we'll be okay. But it is always interesting. And then after we've done the validation and vetted that it all worked. We actually do the test and we discover a suite of other problems that the analysis had never found.

So exercising the failure for real at the lower scale allows you to find a lot of these problems. And it's the same thing like, you know, if you have very sensitive hotspot in your application and your product, injecting failure for real at a very low percentage of user will...

Allow you okay do I have the right detection like did that trigger properly did it get to the right person did this team know how to handle it like without this it's it all becomes like improv like you improv all the time versus when you have a portion of your infrastructure that's always tested in such a way first you keep on hardening and it gets better and it gets better and the muscle of handling it gets better

Also so I guess that I was going to ask about like how, how would teams practice known failures versus unknown failures?

But it sounds like it comes down to just that practicing and pulling out the regions. Okay. And there's, there's a bunch of types of failure. Like, you know, I, I made an allusion about worst-case failures versus worst-case failure, which is, um, you know, you lose a complete region.

And with the weather patterns in the U.S., you know, we have hurricane season, we have like fire season, tornado, like the cuts happen everywhere. So you got to be ready. But there's also other things like, you know, for meta, we are often, I would call it like we got, we get these traffic spikes. It could be because there's something in the news related. It could be a big event like the Super Bowl, New Year's Eve.

The World Cup's coming. So we know that every time there's going to be a goal, it's going to spike. What we realize over time with practice is that some system will be overloaded. And in some cases, you'll have to restart them. And during overload, the damn thing cannot restart. You cannot kill it. Like it's all completely dead. Like it's completely blocked. So we practice a lot of these cases too.

And it's something specific. To us, we get these spikes that are kind of unexpected. Most systems have some form of overload protection up to a point, but it's something we really had to invest in, into like, in the worst load scenario, can I restart? Do I have, like, is my process smart enough to just accept requests later down the road?

Or do I have enough control in the traffic routing to, like, okay, choke the traffic, bring back up, warm up your memory, and then, like, gradually. Like, we had to practice that a lot. Yeah, so you're dealing with, like, thrashing and having, like, no... Yeah. Yes, yes. Resources being completely exhausted. Yeah, yeah. Completely gone. And is that typically at the control plane Completely gone.

And is that typically at the control plane level? Or... Some of our control plane is pretty good you know, there's some backend. Like you can imagine like, you know, an Instagram feedback an Instagram feed backend or Facebook feed backend same type of things. And we have a couple of academic paper. People can look at it. It's called like Taiji, I believe.

And then when we discuss these things on how we can reroute traffic and then control and then remove it from an area to allow the system to reload. Okay. So with that in mind. I would imagine most listeners are not operating at Meta scale, so they're not dealing with every time there's a goal at the World Cup, you know, their servers go down.

What ideas can actually transfer for most like SRE DevOps that are listening? I think the example I gave earlier, like you often see like no blame on Amazon, but if US East go down, like, you know, we see the world of the internet like go like berserk. So I'm like, guys, like, you know, you got to start testing these cases.

Like you've seen it happen, like, you know, at least like, you know, three or four times in the last two years. You got to start like, you know, having your dual regions and really validating it. And maybe it's a question about like, you know, every Tuesday, I do not run a single thing in US East, you know, whatever the name is these days to make sure that it's, you know, it's performing.

And then it's kind of like tackling the problems in a way like, you know, when you look at your incident inside a small, a smaller company, medium sized startup or so on, like I get a list of incidents.

There's a point where you start having enough data that you could say, okay, most of them are caused by, it's because we launched that feature or because we're doing configuration change or because we have a billing problem. It's starting to.

Trend your your SEV data and figure out hey what are the top two or three then we can start attacking and really focusing because like you can focus on everything but you'll get nowhere it's really start bucketizing like you know in having the discipline of writing it down like it's not that hard to write like a SEV report there's a lot of example in there and having the discipline of going back and Now with LLMs, do the analysis is kind of like trivial.

You need to, we used to need to allow a lot of people to do this and now it's a lot easier. So I think that's a big opportunity to actually really focus on where reliability matters. Yeah, that's fair. You had mentioned too that reliability expectations should match the system and product life cycle. Yes. Can you talk about that? Like, because 100 % uptime is not always like the right goal. Yes, yes, yes, yes. There is.

There's like you know when I say about the expectation versus life cycle we should always expect like if something is is has some level of complexity and some level of feature at the beginning when you roll it out like you know getting two nines or even three nines of reliability it's pretty good like like like not a simple system that's that that that's usually like you can get it easy but like if it's early in the

Life cycle of that product or that backend at the beginning, it's going to be rocky and it should be expected.

Like I see a lot of people start with the assumption that, Hey, everything should be like six nines, six nines guys. Like this is like insane. Like, and you invest a lot and then you, you kind of like mix, uh, miss your product market fit while you're doing that. Like there are some like Facebook and Instagram, there's a gazillion experiments that runs at a given point in time.

Some of them are not fully reliable like and it's okay because we're trying to figure out if that feature will be something that users enjoy and use or not but they are other part of the app which like needs to be like rock solid and this is the place where we invest so depending on the life cycle if you are early or not because there's a cost at investing and then when especially like either your SRE or production

Engineering and you have to also convince your PM, your product managers that this is the case, having that conversation from that standpointpoint is a lot more easier than just like being the one that says, no, I need six nines of reliability for everything.

That's not happening. Yeah. Well, six nines too, that's like what, 31 seconds a year of downtime? I mean, the amount of infrastructure and cost complexity to just keep a system at scale. Depending on the system, I mean, I assume if it's like a very important financial system, maybe it matters. But for the majority of systems, like do the users really care about 31 seconds of outage per year?

I mean, are they going to? They do care. You know, we can see from our own data that, you know, if we have too much problems in a row, we can see that, you know, engagement eroding. So they do care. They do show. But it tends to stabilize when we stabilize things. So there's a correlation there. But sometimes the investment to reach that extra nine is just so high.

And while you're doing that extra nine, you're also sacrificing not just the speed and velocity, but also your mitigation time for future unknowns. Because when you have invested so much complexity and then you get into an unknown situation and then now you have to untangle all of this, your mitigation time will be higher. So it's kind of like it's a delicate trade-off. Like it's not a win-win all along.

Yeah, for sure. It's a reality there. A real conversation that you have to have and a balance that you have to have there. Okay, so wrapping up, this conversation also connects to At Scale systems and reliability. Why does that program feel relevant right now? Yeah. So we have this suite of conferences, like it's called At Scale. And we have like four conferences a year.

And the next one that's coming is basically systems and reliability together. And what we want to target for this specific conference is, hey, how are we injecting? What are the systems that are like, you know, building reliability for AI? And what are the systems that are under the hood for AI? Because people tend to talk about, hey, we talk about the models, okay?

But we never talk about like the underlying plumbing that is required to train and serve the model and the insane amount of data that we need to move back and forth. And then on the other angle, we also, we will discuss like... All the other areas where we actually use AI to enhance reliability. So it's both like, you know, both cases that we want to do. This conference is going to happen in person.

It's going to be in Bellevue. And you can find the detail. The website is at scale conference. And we could put that at the show notes. Yes. So if someone works in infra SRE platform or engineering leadership. Why should this be on their radar? And what are you watching specifically with the conference? I'm just curious. Personally, I'm a big fan of this conference. I've been part of it for a while.

And one thing that I tell the speakers that are coming from Meta, and we have speakers from NVIDIA, from Microsoft, from Google, the goal here, I want to have the real technical discussion. With Meta, I'm not in the business of selling cloud services. Like I'm not the business of selling API or things like this.

So what I want to talk is like, you know, Microsoft, you have a Kubernetes cluster with one million servers. How the hell did you do that? Like, this is like, you know, the goal is really like, I want to really understand the story behind the system. I want to have like the real technical conversation. And I want to avoid the sales pitch of like, oh. Use that service, use that thing.

So this is really the focus of the conference is to really have that technical discussion and also the story behind the system. Like, I'm always fascinated. I've always been fascinated by, like, okay, a team starts something. Like, I've been involved in ZooKeeper and all sorts of other things in my days at Yahoo back then. You start something, it was to solve a problem.

Then you open source and you're like, oh, that became this? Oh my God. So I'm really interested in this and then like the ups and downs of the team because you always have like, you know, the hype at the beginning and then, oh, the reality hits and you're like, oh my God, like this will not work as expected. And how do you overcome that?

Like that, for me, that's the most interesting part of all of these stories and all of these presentations. So, okay, wrapping up, what...

Kind of reliability conversations does the industry need more of I think right now the industry does understand the the use of AI for code generation like I think we get that I think we can get that we can go all out I think we understand that everybody can build a custom app like for themselves and they will only use it for for them it's gonna be perfect for them and then it's okay if it goes down but I think the

Rest of the industry has not kept up with that rate of change and there's not enough investment in kind of like defense like we're able to generate code are we able to debug it faster are we able to understand it faster are we able to troubleshoot it faster like like that has not kind of followed.

And then I feel that we are catching up now. And then the hype seems to be like mostly on, on the model. And then there's amazing infrastructure that had to be built underneathneath. And I think you know, everybody needs to understand it a little bit more. Yeah, for sure. Awesome. So I will put links for At Scale Systems and Reliability reliability in the show notes. I can put your LinkedIn there as well.

Is there any other links or comments that you'd like to give to the audience before we wrap up? No, I think that's it. Don't give up. Awesome. Thank you so much, Francois, for coming on. Really appreciate your time. Thank you so much. Thank you. Bye-bye. All right. That was my conversation with Francois Richard from Meta. My biggest takeaway from this one is that reliability is not just about preventing failure.

The better question is, what happens when prevention fails? Because sometimes the answer is a rollback. Sometimes it is moving traffic. Sometimes it is draining a region. Sometimes it is restarting a service. Sometimes it is realizing that the service cannot restart cleanly because it is already overloaded, which is the kind of fun little production detail that does not usually show up in architecture diagrams.

That is the part that I think is worth paying attention to. A lot of teams talk about reliability like it is mostly a tooling problem. Get the right dashboards. Get the right alerting. Define the SLOs. Add some runbooks. Maybe sprinkle in some AI and pretend the incident lifecycle is solved. But reliability is not just the tools. It is the practice around the tools.

It is whether the SLO actually represents a promise to users. It is whether the alert fires early enough to matter. It is whether the incident review produces real follow-up work instead of just a nicer explanation of what broke. It is whether the team has practiced the failure mode before production forces them to learn it live. And honestly, that is the part of the conversation that translates really well.

Even if you are nowhere near Meta scale. Most of us are not dealing with World Cup traffic spikes or massive global systems, but a lot of us are depending on a cloud region more than we want to admit. A lot of us say we are multi -region, but have not actually run without the primary region on a boring Tuesday. A lot of us have runbooks that look reasonable until someone has to follow them under pressure.

A lot of us have services that should recover automatically, but only if the failure happens in the exact way we imagined. That is where the work is. Practice the recovery. Test the boring assumptions. Look at your incident data. Bucket the causes. Figure out what keeps showing up. Then go after the top patterns instead of trying to boil the ocean.

I also liked Francois' point about AI changing the reliability equation. AI can absolutely help with investigation. It can look across a lot of data quickly. It can help build queries, connect patterns, and speed up the part where humans are trying to figure out what changed and what is related. But AI is also increasing the volume of change.

More diffs, more generated code, more boilerplate, more systems moving faster, and sometimes less human context behind the code that just went out. That is a weird trade-off. Because if code moves faster than understanding, Reliability teams are going to feel that gap during incidentsss.

The system breaks, and now someone has to reconstruct not just what changed, but why it changed, what the generated code is actually doing, what assumptions it made, and how to get back to a stable state. That does not mean AI is bad. It means the defensive side has to catch up. Debugging has to get better. Observability has to get better. Incident response has to get better. Recovery practice has to get better.

And humans still need to be in the loop for judgment, especially when the system is too important to let a guess turn into the next mitigation. I also think the lifecycle point matters. Not every system needs the same reliability target. A brand new experiment probably should not get the same investment as a core production path that millions or billions of people depend on.

Six nines sounds impressive until you realize what it costs, what complexity it adds, and how much slower it can make future changes. But the reverse is true too. If a system becomes important and the reliability investment never catches up, you are just borrowing risk until production collects. So maybe the healthier conversation is not how do we make everything maximally reliable.

It is more like what promise are we making? Who depends on this system? What happens when it fails? And have we practiced the recovery enough to believe our own answer? That is probably where a lot of reliability conversations are heading. Not AI will fix incidents. Not SLOs solve reliability. Not just make everything multi-region and call it done. More like, what failures should we expect?

What failures have we practiced? And what are we learning every time production teaches us something? I'll have links to Francois, Meta's At Scale systems and reliability event, and anything else we mentioned in the show notes. If you enjoyed this conversation, follow or subscribe to Ship It Weekly wherever you listen to podcasts.

It helps the show and it makes sure you get both these conversation episodes and the weekly DevOps, SRE, platform, cloud, and security news recaps. You can also find the show notes and links over at shipitweekly.fm. Thanks for listening, and I'll see you later this week.

An AWS data hall overheats. A single availability zone goes dark. And Coinbase spends hours recovering systems that were not as failure-independent as everyone hoped. Meta's AI support tooling becomes part of an Instagram account recovery path. And suddenly the chatbot is not just helping customers. It is touching identity. AWS has another agent tooling bug, this time in AgentCore CLI.

Where imported agent metadata could become generated Python code. And Google Cloud disclosed an Apigee cross-tenant issue, which is exactly the kind of phrase that makes cloud engineers sit up a little straighter. The theme this week is simple. The risky part of modern infrastructure is not always the big obvious production system. Sometimes it is the glue, the recovery path, the support workflow, the generated file.

The tenant boundary. The thing everybody depends on, but nobody quite treats like production until it breaks. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, the show where we look at the DevOps, SRE, cloud, platform, and security stories that actually matter when you are the person who eventually has to keep the thing running.

This week, we're starting with Coinbase's May 7th outage postmortem. Then we'll talk about Meta's AI support incident, where attackers reportedly hijacked Instagram accounts through an AI-assisted recovery flow. After that, we'll get into AWS AgentCore CLI CVE-2026-11393, where collaborator metadata could break out into generated Python code.

Then we'll talk about Google Cloud Apigee and a cross -tenant vulnerability, because tenant isolation is one of those cloud promises you only think about when it fails. In the lightning round, we'll hit Cloudflare threat intelligence in WAF rules, AWS Lambda tenant isolation with event source mappings, the next generation of OpenSearch Serverless, and GitHub Enterprise Managed Users IP allow list coverage.

So let's get into it. First up, Coinbase published a postmortem for its May 7th outage. And this one is worth reading if you work anywhere near SRE, platform, cloud, infrastructure, Kafka, financial systems, or any system where people say we're multi-AZ and then everyone quietly hopes that means what they think it means.

Coinbase says that trading, deposits, withdrawals, and most customer -facing services were unavailable or degraded for roughly eight hours. Full recovery took even longer. The initiating event was physical. Multiple chiller units failed in a single AWS us-east-1 data hall, specifically availability zone use1-az4.

That cooling issue triggered a thermal safety shutdown of affected racks, taking EC2 instances and EBS volumes in that building offline. That is the part that looks like a classic cloud outage story. Something physical breaks inside a data center and cloud resources disappear. But the more useful lesson is what happened next. Because most of us already know that an availability zone can fail. At least in theory.

We have all seen the diagram. Three boxes, three AZs, nice clean arrows, everybody nods. Then a real failure happens, and the system has to prove whether those boxes were architecture or just decoration. For Coinbase, one major issue was that the matching engine was tied closely to the failed zone. And this is where the nuance matters. Low latency systems often make tradeoffs that normal apps do not.

If you need extremely tight latency, deterministic behavior, and careful state handling, you may choose a design that favors performance over easy failover. And that is not automatically irresponsible. But it does mean the recovery path has to be real, not theoretical. Not we documented it once. Real. Tested. Owned. Practiced. Known by more than one person. The other big lesson is state.

Coinbase called out delays around managed Kafka recovery. And that is extremely relatable. During an outage, stateless compute is usually the easy part. You restart it. You move it. You scale it. You reroute around it. Then you hit the system with state, ordering, offsets, replication, partitions, or money-adjacent semantics, and everything slows down. That is where incidents get complicated.

The app might be multi-AZ, but what about the broker? The database? The cache? The topic, the volume, the thing that keeps the system honest. That is the real test. The operator takeaway here is simple. Do not let multi-AZ become a comfort phrase.

Find the parts of your system that are still effectively tied to one zone, one stateful dependency, or one recovery path, especially the low-latency pieces, the Kafka pieces, the databases, the matching engines, the services with ordering, offsets, or money adjacent semantics. Then ask the annoying question, what happens if that AZ disappears? What depends on recovering in place? What can actually fail over?

How long does it take? And when was the last time we proved it? Because resilience is not where the diagram says the boxes are. It is where the system can actually recover when one of those boxes disappears. Second story, Meta confirmed an incident involving its AI-assisted Instagram support flow. Where more than 20 ,000 accounts were likely impacted.

The reported issue was that attackers could use Meta's AI support tooling to trigger password reset links to an email address that was not actually associated with the victim's account. Meta described it as a bug where the system did not properly verify that the provided email matched the email on the Instagram account. So attackers could receive reset links for accounts they did not own.

Now, I know what you might be thinking. Instagram accounts do not sound like a normal DevOps or SRE story. But the real story is not social media accounts got hacked. The real story is this. AI support automation became part of an identity recovery control plane. And that is absolutely our lane. Account recovery is privileged infrastructure.

It is the break-glass path for identity it can reset passwords it can change emails it can bypass parts of the normal login path when the user needs help that means it has power and if ai can influence that workflow then the ai is not just answering support questions anymore it is participating in an identity decision that changes everything a chatbot that answers how do I update my profile photo is one thing A chatbot that helps move an account recovery flow forward is very different.

That system needs real authorization logic, real verification, rate limits, abuse detection, audit logs, escalation paths, blast radius controls, and a very clear line between what AI can suggest and what the system is allowed to do. Because attackers do not care whether your org chart says this belongs to support. Identity, security, product, or infrastructure. They care that the workflow can reset accounts.

So they attack the workflow. This is also a reminder that AI risk is not always a coding agent running shell commands. Sometimes it is much more boring. A support flow trusts the wrong field. A reset link goes to the wrong place. And suddenly, the AI part gets the headline. But the real failure? Is the control boundary around the action. The takeaway here is not AI support is bad.

The takeaway is that support workflows can be privileged infrastructure. Password resets, MFA resets, email changes, account recovery, support impersonation, admin ownership changes. Those are not just customer service features. Those are identity control paths. So when AI gets added to that flow, The question is not just, is the answer helpful? The question is, what can this workflow actually do? Can it suggest?

Can it route? Can it trigger actions? Can it change trust? And if it can touch identity, it needs the same seriousness as any other production control plane. Because the scariest AI system in production might not be your coding agent. It might be the support flow that can reset passwords. Third story. AWS published a security bulletin for CVE-2026-11393 in AgentCore CLI.

And if last week's Kiro bug was stdin answered the approval prompt, this one is more like agent metadata became Python code, which is not the kind of sentence you want in a security bulletin. The issue involves AgentCore CLI when importing a Bedrock supervisor agent with multi-agent collaboration enabled.

The advisory says the CLI fetched collaborator metadata from the Bedrock API and inserted the collaborationInstruction field into a triple quoted Python string in the generated main.py file. The problem was improper escaping of triple quotes. So a crafted instruction could break out of that intended string and inject Python code into the generated file.

Then, if the file was run locally or deployed into AgentCore Runtime, the injected code could execute with the credentials available to that environment.

AWS recommends upgrading the CLI, removing affected imported agents rerunning the import with the patched cli and redeploying if you cannot upgrade right away they recommend manually inspecting generated main.py files for suspicious triple quote sequences in collaborator instructions now this is preview tooling and the attack path is specific but the pattern matters generated code is code that sounds obvious but teams do not always treat generated code with the same suspicion as handwritten code.

Generated files feel like outputs, artifacts, something the tool made, something official, something you probably do not need to read too closely. But if untrusted or semi-trusted metadata gets inserted into source code, your generator becomes a compiler for attacker-controlled input. And we have seen this movie before. SQL injection. Template injection. Shell injection. CI config injection.

YAML templating weirdness. Now it is agent metadata turning into Python. Different costume, same villain. The AI angle makes it feel new, but the secure engineering lesson is old. Do not turn text into executable content without a hard boundary. Do not assume metadata. Is safe because it came from an API. Do not assume generated code is safe because it came from an official tool.

And definitely do not assume instructions are harmless. In agent systems, instructions are operational inputs. They can shape behavior. They can get passed between systems. They can become config. They can become prompts. And sometimes they can become code. The takeaway here is short. Generated code is still code.

If a tool turns metadata, instructions, templates, or agent definitions into executable files, that path needs review. Keep the CLI updated. Inspect generated files before running them in trusted environments. Run agent tooling with limited permissions. And pay attention to what credentials are available if something goes wrong. Because the blast radius is not just the bug.

The blast radius is what the generated code can access when it runs. Fourth story. Google Cloud disclosed a vulnerability in Apigee, CVE-2025-13292. Google's release notes say the issue could have allowed a malicious actor with administrative or developer -level permissions in their own Apigee environment to elevate privileges and access cross-tenant data.

NVD describes it as unauthorized read and write access to Apigee analytics data and access logs belonging to other customer organizations. Google says the issue was patched and no user action is required. That is good. But the phrase that matters is cross-tenant. Tenant isolation is one of the deepest assumptions in cloud computing. You have your environment. Other customers have their environments.

The provider keeps those boundaries in place. That is the model. Most of the time, you do not have to think about it. And you cannot think about it all day. If cloud customers had to constantly wonder whether another tenant could see their logs, analytics, objects, volumes, or traffic, nobody would get anything done. So when a managed service has a cross-tenant issue, even a patched one, it is worth paying attention.

And Apigee is not some random side tool. API management sits in a sensitive part of the stack. It can see traffic patterns, access logs, analytics, client behavior. API paths, timing, errors, and sometimes more than people realize. Access logs are not always harmless. They can reveal service names, query parameters, client identifiers, internal routing, usage patterns.

And because logging systems are logging systems, sometimes they capture fields everyone later wishes had been redacted. So cross-tenant access to analytics and logs is not just a privacy issue. It can be a reconnaissance issue, a compliance issue, a customer trust issue. The takeaway is not never use managed services. That would be ridiculous.

The takeaway here is that tenant isolation is not just a cloud provider promise. It is a design requirement. For every platform team building multi-tenant systems, logs, analytics, dashboards, exports, admin tools, support consoles, background jobs, search indexes, all of those can become tenant boundary problems. So do not only test the obvious request path, test the internal paths too.

Can tenant A see tenant B's logs? Can a support view cross the wrong boundary? Can an analytics query forget tenant context? Can an async job mix records that should never touch? Cross tenant bugs usually do not start in the happy path. They start in the seams. Now let's do a quick lightning round. First, Cloudflare is turning threat indicators into real-time WAF rules.

Cloudflare says Cloudforce One threat intelligence can now be used directly inside the WAF engine. That means teams can write rules based on things like known attacker names, targeted industries, source countries, target countries, and attack context. I like the direction. It moves security from, we know this traffic is bad, towards we can actually enforce on that knowledge. But the operator warning is obvious.

Threat intel in the blocking path is still production change management. Start in visibility mode. Watch false positives. Stage the rollout. Make sure that the logs explain why something was blocked. Automation is great until it blocks your biggest customer because a field matched a little too creatively. Second, AWS published guidance. For Lambda tenant isolation mode with event source mappings.

The interesting part is that async systems do not naturally carry tenant context the same way synchronous API requests do. With an API request, you may have headers, claims, and request context. With SQS, EventBridge, or other event sources, tenant identity may be inside the payload or may need to be extracted and passed along carefully. That matters because multi-tenant bugs love async systems.

A message loses context. A worker assumes the default tenant. A retry uses the wrong metadata. A batch mixes records that should never be mixed. Tenant isolation is not just about the runtime. It is about context propagation. Especially when queues and events are involved. Third, the next generation of Amazon OpenSearch Serverless is generally available.

AWS says it provisions faster, scales faster, supports scale to zero, and can reduce costs compared to provisioning for peak load. That is a FinOps story, but it is also an operations story. If teams start treating search and vector infrastructure as more elastic and disposable, they still need to understand cold starts, latency, indexing behavior, cost patterns, and traffic spikes.

Serverless does not mean operationally invisible. It means the operational questions moved. Fourth, GitHub Enterprise Managed Users IP Allow List coverage is generally available. Enterprises using EMUs can now enforce GitHub's native IP Allow List configuration across user namespaces. That matters. Because source control is production infrastructure. Repo access is part of your security boundary.

And more of that boundary is moving into identity, network policy, device posture, and enterprise governance. IP allow lists do not solve everything. But repo access is still one of the fastest ways to turn a credential problem into a production problem. The human closer this week is about the glue. Coinbase was not just an AZ failure. It was state, latency, Kafka, and recovery paths.

Meta was not just an AI support bug. It was account recovery becoming identity infrastructure. AgentCore was not just a Python escaping issue. It was metadata turning into executable code. Apigee was not just a patched cloud vulnerability. It was tenant isolation showing up in logs and analytics, not just the main product path. Different stories, same pattern.

Modern reliability and security problems keep showing up in the seams. The support flow. The generated file. The recovery path. The admin console. The queue. The logs. The things between two systems that quietly decides what is allowed to happen next. That is where a lot of real platform work lives now. Not just making the app scale. Not just adding another region. Not just buying a managed service.

But asking where the authority actually sits. Who can recover an account? Who can generate code? Who can access another tenant's data? Who owns the failover path? Who tested the recovery plan? And who gets paged when the glue fails? So the takeaway this week is simple. Do not just review the big production systems. Review the seams. That is usually where the incident is waiting.

That's it for this week of Ship It Weekly. We covered Coinbase's May 7th outage postmortem, Meta's AI support and Instagram account recovery issue, AWS AgentCore CLI CVE-2026-11393, Google Cloud Apigee's cross-tenant vulnerability, and a lightning round on Cloudflare WAF threat intel, AWS Lambda tenant isolation, OpenSearch Serverless, and GitHub Enterprise Managed Users IP allow lists.

If this episode was useful, follow or subscribe wherever you are watching or listening. If you are on YouTube, hit subscribe. If you are in a podcast app, follow the show there. And if you know someone dealing with cloud resilience, AI support workflows, agent tooling, tenant isolation, or repo governance, send this one to them.

It helps the show grow, and it helps me keep making this kind of content for people who actually live with these systems. You can find the weekly brief at OnCallBrief.com and more episodes and this week's show notes at ShipItWeekly.fm. I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, the system that breaks production is not always the big obvious service. Sometimes it is the glue everybody forgot was holding the whole thing together.

Your AI coding tool asks for permission. Then stdin answers for you. That is not a sentence I expected to say this week, but here we are. This week, we've got Kiro CLI with an approval bypass through piped input. Amazon Braket with a very normal Python pickle problem inside a very expensive-sounding quantum SDK. AWS Organizations finally logging account membership changes to CloudTrail.

And KEDA releases that remind us autoscaling upgrades are not just version bumps. Then in the lightning round, we'll hit Kubernetes dashboard being archived, Google Cloud's managed MCP server for AlloyDB, Kafka 4.3 .0, and Atlassian's AI productivity claims around more pull requests and saved developer time. The theme this week is pretty simple.

Automation keeps getting more authority, but the boring boundaries still decide what happens. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly. The show where we look at the DevOps, SRE, cloud, platform, and security stories that actually matter when you are the person who eventually has to keep the thing running.

This week is a nice mix of AI tooling risk, cloud governance, Kubernetes operations, and the ongoing question of what happens when automation gets more power than people realize. We're starting with Kiro CLI and a bug where crafted piped input could be treated like user approval. Then we'll talk about Amazon Braket SDK and an insecure deserialization issue in quantum job results.

After that, we'll get into AWS Organizations finally emitting CloudTrail events when accounts join or leave an organization. Then we'll talk KEDA 2.20 and 2.19. Because autoscaling upgrades can move assumptions around events, permissions, scalers, and monitoring.

In the lightning round, we'll hit Kubernetes Dashboard being archived with Headlamp as the path forward, Google Cloud Remote MCP Server for AlloyDB, Apache Kafka 4 4.3.0, and Atlassian's AI-native SDLC productivity claims. And the human closer this week is about hidden boundaries, because the scary part of automation is not always what it can do. Sometimes it's what we forgot it was allowed to do.

So let's get into it. First up, Kiro CLI had a pretty interesting security issue. AWS published a bulletin for CVE-2026-9255 affecting Kiro CLI versions before 1.28.0. The short version is that crafted content piped into Kiro CLI through stdin could be consumed as confirmation input, and that could allow tools, including shell commands to execute without the user actually approving them.

So yes, the AI tool asked for permission. And stdin answered for you, which is funny in the way that makes you stop smiling after about half a second. Because this is not just a random command-line parsing bug. This is an approval boundary bug. A lot of AI coding tools are built around the idea that the agent can suggest actions, but the user approves the risky parts.

That approval step is the thing that makes people feel safer. The tool says, hey. I want to run this command. The human looks at it and says yes or no. That is the mental model. But if crafted input can accidentally or intentionally flow into the confirmation path, now that boundary is not really where people thought it was. And that matters because these tools are not just answering trivia questions.

They can read files. They can inspect repos. They can call tools. They can run commands. They can modify code. They can sometimes interact with cloud CLIs, Kubernetes configs, package managers, and all the usual developer machine chaos. So when approval gets weird, the blast radius gets weird too. The practical lesson here is bigger than Kiro.

If you are building or adopting AI tools, Approval needs to be treated like security controls, not UX prompts. Security controls. That means you should think about where approval input can come from. Can it come from standard in? Can it come from a file? Can it come from pasted content? Can it come from generated content? Can it come from previous agent output?

Can it come from something the model read and then accidentally turned into an action? Because when an AI tool has access to execution, The approval path is part of your trust boundary. And trust boundaries are where boring bugs become very interesting. For operators and platform teams, I would treat this as another reminder to be careful with agent tooling in high trust environments. Keep tools updated.

Avoid running agentic CLIs with broad permissions unless you really need to. Be careful piping untrusted content into tools that can execute actions. And do not assume that a confirm before running prompt automatically means the human is the only thing that can confirm. The button is not the boundary. The input path is. Second story. Amazon Braket SDK had an insecure deserialization issue.

And I have to admit, there is something funny about quantum computing getting a very normal Python pickle bug. Like, yes. We are doing quantum workloads and advanced simulation and cloud-managed quantum jobs. But also, somewhere in the middle of that, Python Pickle is still standing there with a folding chair. AWS published a bulletin for CVE-2026-9291 affecting Amazon Braket SDK versions before 1.117.0.

The issue involved job results. The SDK trusted a data format field in job result JSON. And under the wrong conditions, it could call pickle.loads() on attacker-controlled data. That could lead to remote code execution for a remote authenticated user with S3 write access to the job output bucket. This is a very specific vulnerability, but the pattern is extremely familiar. You have an SDK. You have job output.

You have S3. You have metadata that says what format the data is in. And somewhere in that chain, the client trusts the metadata too much. Then the parser becomes an execution path. That is the lesson. Deserialization bugs are not old news just because the service sounds modern. Pickle is dangerous when used on untrusted input. That sentence has been true for a very long time.

And apparently, it is still true even if the words around it include quantum computing. For DevOps and platform teams, I think the takeaway is about the trust boundaries around job output and storage buckets. A lot of systems treat output buckets as boring storage. The job writes results. The client reads results. Everybody goes home.

But if the client treats those results as trusted instructions for how to deserialize data, then whoever can write to that bucket may be influencing code execution on the client side. That changes the security posture. So yes, patch the SDK. But also, think about the surrounding pattern. Who can write to job output buckets? Are those buckets shared? Are they cross-account?

Can temporary jobs write results that other systems later consume? Are clients running in developer environments, CI, notebooks, or production workflows? And are any of those clients deserializing data in a way that assumes storage means trusted? Because storage is not trust. Storage is just where the problem waits politely until something reads it.

Before we get to the next story, a quick word from this week's sponsor, @Scale: Systems & Reliability. @Scale: Systems & Reliability is happening June 25th at the Meydenbauer Center in Bellevue, Washington. The event is focused on the infrastructure behind AI-scale systems, reliability, high-performance compute, storage, networking, and the operational side of AI-driven systems.

So if you are thinking about how reliability changes when AI workloads, agents, and automation enter the picture, this is worth a look. Register today at atscaleconference.com/events/systems-reliability-2026 or use the link in the show notes. All right, back to the show. Third story, AWS organizations now emit CloudTrail events when accounts join or leave an organization.

And this is one of those updates that sounds small until you remember what AWS Organizations actually represents. An AWS Organization is not just a billing container. It is a security boundary. It is where SCPs apply. It is where centralized logging expectations live. It is where guardrails live. It is where account ownership and governance gets enforced.

So when an account joins or leaves an organization, that is not just an administrative event. That is a security and governance event. AWS says organizations now emit account joined organization and account departed organization events to the management account. Those events include things. Like the join or departure method and timestamp.

You can feed them into CloudWatch alarms, EventBridge roles, or whatever your alerting and governance process looks like. And honestly, this feels like one of those things that should have already existed, which is not a compliment exactly, more like a very cloud engineering version of relief. Because account movement is the kind of thing you want to know about immediately.

If an account leaves your organization, what happened to the SCPs? What happened to centralized CloudTrail? What happened to Config rules? What happened to Security Hub aggregation? What happened to access policies that assume the account is still under org control? And if an account joins your organization, is that expected? Is it coming from an acquisition? A sandbox? A delegated admin flow?

A weird test account someone created with a personal email in 2021 and is now trying to bring home like a stray cat. The practical takeaway is pretty simple. Do something with these events. Do not just mentally file this under nice AWS improvements. Create EventBridge rules. Alert on account join and departures. Send them somewhere your cloud platform or security team actually watches. Attach context if you can.

Was this account expected? Who owns it? Which OU did it join? What guardrails apply? Was the departure planned? Do we need to investigate? Because the worst time to discover an account left your organization is later, when you are trying to explain why the normal guardrails did not apply. Account membership is not just billing metadata. It is control plane state. And control plane state deserves alerts. Fourth story.

KEDA has had a couple of releases. Worth operators actually reading. And I know, a KEDA release note is not exactly the kind of thing that makes people cancel dinner plans. But autoscaling changes are production changes. KEDA sits in a very sensitive part of the system. It watches external signals. It decides when workloads should scale.

It interacts with Kubernetes events, RBAC, metrics, scalers, and workload behavior. So when KEDA changes event recording, scalers, permissions, or assumptions, that is not just a version bump. That is part of how your workload responds to pressure.

The on-call brief this week flags KEDA version 2.20 and version 2.19 changes, including event recording moving to the events.k8s.io API group, plus scaler changes and additions around things like OpenSearch, Elastic Forecast, and Kubernetes resources. The event recording change is the one I would pay attention to because events sound like observability garnish until something breaks.

Then suddenly, everybody wants to know what scaled, when it scaled, why it scaled, what failed, and which controller said what. If events move API groups, that may affect RBAC. It may affect tools that scrape or watch events. It may affect alerting. It may affect the way your team reconstructs scaling behavior during an incident. And that is the hidden risk with autoscaling systems.

When they work, nobody thinks about them. When they fail, everybody discovers they are part of the incident timeline. The practical takeaway is not avoid KEDA upgrades. KEDA is useful. The takeaway is to treat autoscaler upgrades like production behavior changes. Read the release notes. Check RBAC. Check event collection. Check dashboards. Check scaler-specific changes for the triggers you actually use.

Test the upgrade in a non-prod cluster with real-looking metrics if you can. And make sure your team still knows how to answer the basic questions. Why did this workload scale? Why did it not scale? What signal was it watching? What did KEDA think was happening? And where would we see that during an incident? Autoscaling is one of those areas where it just works is great right up until it very much does not.

Now let's do a quick lightning round. First, Kubernetes Dashboard has been archived and Headlamp is the path forward. We have mentioned Headlamp before, so I'll keep this short. The new angle is that dashboard being archived means platform teams should decide what the supported cluster UI actually is.

If developers, support engineers, or internal teams rely on a UI to inspect workloads, logs, events, or namespaces, that UI is part of your platform surface. So do not let the migration happen by accident. Decide what you support. Decide who gets access. Decide how auth works. Decide what actions are allowed. And please, do not replace one risky dashboard with another risky dashboard and call it modernization.

Second. Google Cloud Remote MCP Server for AlloyDB is generally available. I know we have covered MCP a lot, so I'm not going to make this the whole show. But the trend matters. MCP is moving from local developer demos into managed cloud data access. That means the questions get more serious. Who can connect? What data can the agent reach? What identity is used? What audit trail exists? And can the agent only read?

Or can it take action? Managed MCP is not just convenience. It is another control plane for agent access. Third, Apache Kafka 4.3 .0 is out. For most people, a Kafka release is not something you rush into production during lunch. At least I hope not. But it is worth tracking because Kafka sits under a lot of real-time systems, event-driven platforms, data pipelines, and very nervous on-call rotations.

The high-level takeaway is simple. Read the release notes. Watch client compatibility. Check broker upgrade paths. Check the features that affect your actual deployment model. And remember that Kafka upgrade is rarely just a Kafka upgrade. It usually involves consumers, producers, schemas, connectors, monitoring, retention, partition behavior, and some service nobody remembered still uses the old client.

Fourth, Atlassian published AI-native SDLC productivity claims around more pull requests and saved developer time. The numbers sound good. More PRs. More time saved. More development throughput. And I am not against that. But more PRs is not automatically more throughput. If planning, review, testing, security, release, and operations cannot absorb the extra output, then you may not have improved flow.

You may have just moved the bottleneck. AI can absolutely help teams produce more work. The leadership question is whether the system can safely absorb it. Because more code is not the same as more value. And it is definitely not the same as more reliable production. The human closer this week is about hidden boundaries. That is the thread running through all of this. Kiro CLI had an approval boundary.

Amazon Braket had a deserialization boundary. AWS Organizations has an account membership boundary. KEDA had an autoscaling and event visibility boundary. Kubernetes Dashboard and Headlamp sit on the platform access boundary. MCP sits on the agent data access boundary. Kafka sits on the event pipeline boundary. And AI productivity claims run straight into the boundary between output and actual throughput.

The scary part of automation is not always what it can do. Sometimes it is what we forgot it was allowed to do. That is the part teams need to keep looking at. What counts as approval? What counts as trusted data? What counts as an account boundary? What counts as a safe scaling signal? What counts as supported platform access? What counts as developer productivity? Those are not abstract questions.

Those are production questions. Because automation does not remove boundaries. It moves them. And if you do not know where the boundary moved, you may not know what you are trusting anymore. And that is where staff and principal engineering work often lives. Not just picking the tool. Not just approving the upgrade. Not just reading the headline. But asking where the authority actually sits now. Who can approve?

Who can write? Who can scale? Who can access? Who can deserialize? Who can query? Who can ship? That sounds boring. But boring is usually where the real risk is hiding. So the takeaway this week is simple. When automation gets more powerful, look for the boundary. Then make it visible. That's it for this week of Ship It Weekly.

We covered the Kiro CLI approval bypass, Amazon Braket SDK insecure deserialization, AWS Organizations CloudTrail account membership events, KEDA autoscaling changes, and a lightning round on Kubernetes Dashboard and Headlamp, Google Cloud Remote MCP Server for AlloyDB, Kafka 4 4.3.0, and Atlassian's AI-native SDLC claims. If this episode was useful, follow or subscribe wherever you are watching or listening.

If you're on YouTube, hit subscribe. If you are in a podcast app, follow the show there. And if you know someone on a DevOps, SRE, platform, security or engineering leadership team who is dealing with AI tools, AWS governance, Kubernetes autoscaling, or platform access, send this one to them. It helps the show grow and it helps me keep making this kind of content for people who actually live with these systems.

You can find the weekly brief at OnCallBrief.com and more episodes and this week's show notes at shipitweekly.fm. I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, If your AI tool asks for permission, your SDK trusts a pickle, your account leaves the org, or your autoscaler changes behavior, the question is not just what happened. The question is where the boundary was supposed to be.