Ship It Weekly Transcripts

Trusted tools are having a rough week. A popular VS Code extension reportedly helped expose thousands of GitHub internal repositories. Megalodon hit thousands of public repos with poisoned commits and GitHub Actions workflow abuse. Railway had a platform-wide outage after Google Cloud incorrectly suspended its production account. Discord dropped 17% of active sessions during a Kubernetes migration.

And AWS is changing SDK retry behavior, which sounds boring until you remember retries are how your app behaves when the world is already on fire. The theme this week is simple. The tools that we trust most are becoming some of our riskiest production dependencies. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, the show where we look at DevOps, SRE, cloud, platform, and security stories that matter when you are the person who eventually has to keep the thing running.

This week, we're starting with GitHub supply chain risk from two directions: a compromised VS Code extension tied to a GitHub internal repo breach and the Megalodon campaign abusing public repos and CI/CD workflows then we'll talk about railway's gcp account suspension outage discord's voice outage postmortem AWS changing SDK retry defaults, and a RabbitMQ AWS plugin issue that accidentally shipped debug code into production builds.

In the lightning round, we'll hit OpenTelemetry Graduation, Claude Code RCE, GitLab Secrets Manager, Google Cloud AI Spend Caps, and a Redshift Python driver RCE. And the human closer is about trusted tools because the systems we trust most are often the ones we forget to threat model. So let's get into it. First up, GitHub had a rough supply chain week. The developer toolchain got hit from multiple directions.

The first story is the Nx Console VS Code extension compromise. Reporting says a malicious version of the Nx Console extension was published and that the extension was later tied to a breach involving thousands of GitHub internal repositories. StepSecurity says the compromised version was Nx Console 18.95.0 and that the root cause involved a contributor's GitHub token being scraped in a prior supply chain attack.

The Hacker News reported that roughly 3,800 GitHub internal repositories were exposed after a GitHub employee device was compromised through the malicious extension.

Now, I'm not saying every VS Code extension is evil, but every VS Code extension is still code running next to the repos you clone, the terminal you use, and sometimes the tokens you forgot were still hanging around, which is not exactly a low-trust environment. A lot of developers install extensions like browser tabs. That looks useful. That has a nice icon. That has a lot of installs. Sure, why not?

And normally that feels fine until an extension update becomes an initial access path. That is the real lesson. The developer workstation is part of the production attack surface now, not because it serves customer traffic, because it touches the things that eventually do. Repos, secrets, CI/CD, cloud credentials, deployment tooling, package publishing, SSH keys.

Kubeconfigs all the fun little artifacts we pretend are carefully managed until somebody runs `ls ~/.aws` and the room gets quiet. Then there's Megalodon. Security researchers reported a campaign hitting more than 5,500 public repositories with malware -laden commits, stealing CI/CD secrets like AWS and Google Cloud keys. SSH private keys, and Kubernetes configs. That makes this more than a GitHub story.

It's a reminder that CI/CD is not just where code gets tested. It is where trust gets converted into artifacts. If a workflow has cloud credentials, package publishing tokens, signing keys, or deployment authority, then a compromised workflow is not just a dev problem. It is a release system compromise. The takeaway is not ban all extensions or never use GitHub Actions. That's not serious.

The takeaway is to treat developer tooling as production-adjacent infrastructure. Review extensions with broad file, terminal, or workspace access. Use short -lived credentials where you can. Keep cloud keys out of CI when OIDC works. Lock down GitHub Actions permissions. Require review on workflow changes. And please.

Do not let every workflow run with full write access because that was easier during the first setup. A trusted extension, a trusted repo, and a trusted workflow can all become part of the same attack path. That's the part that matters. Next up, Railway published an incident report about a platform-wide outage caused by Google Cloud. Incorrectly suspending Railway's production account.

That sentence alone is enough to make most cloud engineers sit up a little straighter. Railway says the outage started on May 19th when Google Cloud incorrectly placed their production account into a suspended status. That took Railway's API, dashboard, control plane, databases, and GCP-hosted compute infrastructure offline. And then it got more interesting.

Railway also runs workloads on Railway Metal and AWS burst-cloud environments. And those workloads initially stayed up. But Railway's edge proxies relied on a GCP-hosted control plane API to populate routing tables. So when route caches expired, the outage cascaded beyond GCP.

Workloads that were still technically running became unreachable because the network control plane could no longer resolve routes to active instances. At peak impact, Railway says all workloads across all regions were unreachable. And that is the story. Not just Google Cloud suspended an account. The real story is that multi-cloud did not save the system because the control plane dependency was still in the hot path.

That is where architecture diagrams get too optimistic. You can draw AWS over here and GCP over there. Metal in another box. Some arrows, maybe a nice little mesh diagram. And suddenly, everybody feels resilient. But resilience is not about how many providers appear on the diagram. It is about what has to work during a failure.

If your data plane in AWS still needs a control plane API in GCP to route traffic, then GCP is still in the hot path. If your failover region needs the primary region's identity system to approve failover, then the primary region is still in the hot path. If your emergency deploy process depends on the same CI/CD platform that is currently broken, then congratulations.

You have invented a circular dependency with branding. The strongest thing in Railway's writeup is that they owned it. They said that they take responsibility. For the architectural decisions that allowed one upstream provider action to cascade into a platform-wide outage. That's the right posture.

Customers do not care whether the thing that broke was technically Google, Railway, GitHub, Stripe, AWS, or a squirrel with a networking certification. They see your product. So when you say multi-cloud, ask what dependency is still centralized? What service discovers routes? What service holds identity? What API does the edge need? What happens when cached state expires?

And what dependency do you only discover when the provider account disappears and everyone suddenly becomes very interested in architecture diagrams? Multicloud is not magic. Sometimes it is just single cloud with extra invoices. Third story. Discord published a really good postmortem. On its March 25th voice outage. The title is perfect. You've got too much mail.

Because this outage wasn't just Kubernetes killed some pods. It was a chain reaction where a routine infrastructure change hit a stateful system, dropped a large number of sessions, and downstream systems got overwhelmed by the recovery behavior. Discord says voice and video suffered major degradation. For a little over three hours. Users were mostly unable to start or join calls and saw an awaiting endpoint message.

The trigger came during a Kubernetes migration for Discord's Elixir services. They were tuning session management service resources and pod counts. As Kubernetes applied the change, it terminated 50 % of the pods in one zone. Since sessions were balanced across three zones, about 17% of active sessions were ungracefully stopped. That alone is not great. But the cascading part is the real lesson.

Discord’s systems use Elixir GenServer processes, and those processes have mailboxes. When all those sessions vanished, other processes received a flood of messages saying sessions were down. That caused reconnect behavior, rate limit pressure, memory spikes. Gateway problems, and eventually voice and video routing issues.

This is the kind of postmortem that I love because it shows how real outages are usually not one failure. They are interaction failures. Kubernetes did what it was told. The session service had handoff logic. The rate limit existed. The downstream services were designed for normal load, but the shape of the change produced a workload the system was not tuned for.

That is the part people miss when they say, why didn't they just autoscale? Because auto scaling is not a magic undo button for we just invalidated 17% of active sessions and created a reconnect storm. Sometimes the bottleneck is not CPU. Sometimes it's mailbox length, back pressure, downstream fanout. Or one angry queue quietly becoming the main character. The practical takeaway is migration safety.

When you move stateful systems into Kubernetes, think beyond pod termination. What does the rest of the system do when this pod terminates? Who gets notified? Who retries? Who reconnects? Who queues messages? Who gets overloaded trying to help? Graceful shutdown is not just a pod lifecycle feature. It is a system behavior. If you are doing Kubernetes migrations for stateful services, test the ugly cases.

Kill more than one pod. Drain a zone. Watch downstream queues. Look at reconnect behavior. Because production does not ask if the change was routine. It asks if the system was ready. Fourth story, AWS is changing retry behavior across AWS SDKs and tools. And I know that that sounds like the kind of story that you would normally skip because it has the emotional energy of a configuration footnote.

But retry defaults are invisible infrastructure. They affect latency, error rates, load during outages, and how your app behaves. When AWS services are already struggling. AWS says the updated retry behavior is available now behind opt-in and will become the default in November, 2026. The updated behavior changes how standard and adaptive retry modes handle failures.

AWS is making standard mode the default for SDKs that previously defaulted to legacy mode, adding retry quotas where they didn't exist. Changing backoff timing and treating transient errors differently from throttling errors. One big change is that transient error retries cost more retry quota than before.

The idea is that during sustained outages, the SDK fails faster instead of endlessly retrying and adding pressure to a service that is already unhealthy. That is good, but it can still surprise you. Retries are one of those things most teams do not think about until an incident. Your code says call S3. The SDK says it'll handle some retries. Your app says, great, I'll pretend that that was one request.

Then the service starts throwing errors and suddenly request latency, thread usage, connection pools, client CPU, and downstream load all depend on retry behavior you may never have explicitly configured. Retries can save you from transient failures. Retries can also turn a partial outage into a client -side traffic storm wearing a helpful little hat. So the takeaway is simple.

Do not wait until November 2026 to discover how your app behaves. Pick a non -production workload. Opt in with the new environment flag. Look at latency. Look at error surfaces. Look at max attempts. Look at throttling behavior.

Look at long-polling clients like SQS consumers and figure out whether your app depends on old retry behavior without anyone realizing it because nothing says fun on call rotation like finding out your retry strategy was inherited from 2018 and load-tested by hope. Fifth story, let's talk about RabbitMQ, debug code, secrets, and cloud cost blast radius.

AWS published a security bulletin for CVE -2026 -9133 in the rabbitmq-aws plugin. The plugin resolves AWS ARNs in RabbitMQ's broker configuration at startup and can fetch things like TLS certificates, private keys, passwords, and other secrets from AWS services. The issue is that debug code in the plugin's ARN resolver was accidentally shipped in production builds.

A debug ARN scheme accepted by a validation endpoint could allow a remote authenticated user to read arbitrary files accessible to the RabbitMQ process. That is not good. AWS recommends upgrading to rabbitmq-aws 0 .2 .1, patching forked code, and rotating secrets stored in files. The RabbitMQ process could read. This is a specific bug, but the pattern is broad.

Debug code in production should make everyone briefly stop blinking because debug paths often bypass the normal shape of the system. They inspect the thing directly. They validate a thing in a way production code usually does not.

And then somehow that path makes it into a build where a real user or attacker can reach it this also pairs with a separate aws bedrock cost story from Reddit where a user described attackers using exposed access keys from an ec2 instance to run about $14,000 of Claude calls in 24 hours That second story is a Reddit report, so I would not treat it like a formal incident report, but as a pattern it is extremely believable cloud credentials plus AI services can become a very fast money fire.

Security and FinOps are blending together. Compromised cloud keys used to mostly mean crypto mining, data access, or infrastructure abuse. Now they can also mean someone burning through model inference or AI API calls at a rate that makes finance start typing in all caps. The takeaway is not complicated. Scope credentials. Use roles instead of long -lived access keys where possible. Watch unusual service usage.

Put budgets and anomaly detection around AI services. Rotate secrets when file-read issues appear. And remember that authenticated user does not mean safe user. Small bugs get expensive when the process can read secrets and the secrets can spend money. Now let's do a quick lightning round. First, OpenTelemetry graduated from the CNCF. This is a huge milestone.

OpenTelemetry is now basically the de facto standard for vendor -neutral telemetry across traces, metrics, and logs. But the operator warning is still the same. The collector is production plumbing. Graduation does not mean that every collector upgrade is safe, every processor config is harmless, or every telemetry pipeline is suddenly boring. Standardization helps, but you still need rollout strategy.

Config validation, load testing, and a plan for what happens when the thing that reports on production becomes the thing breaking production. Second, Claude Code had a security issue. I'm keeping this short because we have covered Claude code and agent security a lot lately. But the pattern matters. AI coding tools are not just editors. They can have filesystem access, repo context, terminals, deeplinks.

Commands, and workflow integration. So when those tools have parsing bugs, deeplink bugs, or command execution paths, the risk is not theoretical. The developer environment is becoming another agent runtime, and agent runtimes need threat models. Third, GitLab 19.0 introduced GitLab Secrets Manager in public beta. This is a good direction.

Secrets closer to the pipeline, scoped to jobs, governed through the same platform people already use for CI/CD, that does not solve every secret manager problem, but it does acknowledge reality. A lot of secrets risk lives in CI/CD because CI/CD is where systems need credentials to do work. Treating pipeline secrets as first -class objects is better than pretending a masked variable named prod token is a strategy.

Fourth, Google Cloud is rolling out hard spend caps for AI services. This is a FinOps story, but it is also a reliability story. If a budget cap pauses API traffic when spend hits a limit, that can protect you from a surprise bill. It can also become an availability event if your product depends on that API. So hard caps are useful, but they need operational design. Who gets alerted before the cap?

What degrades gracefully? What is customer facing? And what do you want more? A hard outage or a hard invoice? Sometimes the answer depends on the day. Fifth, Amazon Redshift Python driver had an RCE issue. AWS reported CVE -2026 -8838 in the Redshift Python driver, where a rogue server could execute commands on a user's data warehouse client.

That is a good reminder that database clients are part of your execution boundary too. Not every RCE starts on the server. Sometimes the client connects to the wrong thing, trusts the wrong response, and becomes the thing that gets owned. So patch the driver, watch connection targets, and remember that it is just a client library is usually how the story starts, not how it ends.

The human closer this week is about trusted tools the riskiest systems are not always the mysterious ones sometimes they are the familiar ones the extension everyone installs the workflow nobody reviews the retry behavior nobody configured the plugin that shipped with debug code the control plane api that seemed fine because the cache bought you an hour trusted tools become dangerous when trust turns into invisibility That does not mean that every tool is bad.

It means that trust should have an expiration date. Every so often you need to ask, what does this tool have access to? What can it change? What happens if it is compromised? What happens if it disappears? What happens if it retries differently? What happens if the cache expires? That is not paranoia. That is being the person who has to answer the incident channel when everyone else is asking, how could this happen?

The staff and principal engineer job is often about seeing the shape of the dependency. Noticing when a developer tool is actually a production path. When a retry default is actually outage behavior. When a multi -cloud architecture still has one hot dependency. When a plugin can read secrets. When the thing that everyone trusts has become the thing nobody questions. The takeaway is not to stop trusting tools.

You cannot run modern systems that way. The takeaway is to make trust visible. Map the permissions. Review the workflows. Scope the credentials. Test the failure path. Patch the clients. Constrain the plugins. And look at your boring dependencies like they might be production infrastructure. Because they probably are. That's it for this week of Ship It Weekly.

We covered the GitHub supply chain week with Nx Console and Megalodon. Railway's GCP account suspension outage. Discord's voice outage postmortem, AWS SDK retry behavior changes, the RabbitMQ AWS plugin file-read issue, and a lightning round on OpenTelemetry, Claude Code, GitLab Secrets Manager, Google AI Spend Caps, and Redshift Python Driver RCE.

If this episode was useful, follow or subscribe wherever you are watching or listening. If you're on YouTube, hit subscribe. If you are in a podcast app, Follow the show there. And if you know someone on a DevOps, SRE, platform security or engineering leadership team who is dealing with supply chain risk, cloud dependencies, retries or trusted tooling, send this one to them.

It helps the show grow and it helps me keep making this kind of content for people who actually live with these systems. You can find the weekly brief at OnCallBrief.com and more episodes and this week's show notes on ShipItWeekly .fm. I'm Brian Teller from Teller's Tech. Thanks for listening.

And remember, if your trusted tool can install code, trigger CI, route traffic, retry requests, read secrets, or burn cloud money, it is not just a tool anymore. It is part of production. So maybe treat it like it.

For the last decade, the default infrastructure answer has basically been cloud first, Kubernetes probably, and then good luck to whoever has to operate the thing after the architecture diagram gets turned into real life. And to be fair, the cloud solved a lot of real problems.

Nobody wants to go back to waiting on hardware, filing tickets for VLANs, or discovering that the one person who understood the SAN is on vacation. But there is another side of this now. Cloud bills are getting harder to explain. Kubernetes platforms are getting harder to maintain. Data sovereignty is becoming a real business requirement.

And a lot of teams are quietly looking at all of the layers that they have built and asking a pretty reasonable question. Is this actually simpler?

Because sometimes the thing that was supposed to reduce operational burden just moves the complexity somewhere else into Terraform into Helm charts into networking into managed service glue into a platform team that is now responsible for making 15 different abstractions feel like one coherent system and that is really what this conversation is about not cloud is bad not bare metal is back everybody grab a

Screwdriver more like What happens when teams want more control, more predictable cost, better performance, and less platform sprawl without going back to the old-school pain of managing infrastructure by hand?

I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick news recap.

In between those, I do conversation episodes with people who are building platforms, running infrastructure, organizing events, and thinking through where this industry is actually headed. Today is one of those conversations. I'm joined by Jake Warner, founder and CEO of Cycle.io.

Cycle is an infrastructure platform that lets teams run containers and virtual machines across bare metal, cloud, private cloud, and hybrid environments without trying to turn every company into a full-time platform engineering shop. And I like this conversation because it gets into a topic that a lot of people are thinking about right now, even if they are not always saying it out loud.

A lot of teams are tired, not lazy, not anti-cloud, not anti-Kubernetes, just tired of the amount of complexity that has piled up around modern infrastructure. You start with a simple goal. Run the application. Scale it. Deploy safely. Keep it reliable. Don't spend a fortune.

Then suddenly, you have Kubernetes clusters, node groups, disruption budgets, autoscalers, Terraform modules, managed services, IAM policies, GitOps controllers, observability agents, service meshes, secrets systems, and a Slack channel where somebody asks why the platform is blocking delivery. And somewhere in the middle of that, people start wondering whether there is another way to think about infrastructure.

In this conversation, Jake and I talk about why some teams are moving back toward private cloud and bare metal, but not in the nostalgic racking and stacking servers was awesome kind of way. More because of cost, performance, data sovereignty. And wanting more ownership over the stack.

We also get into what people still misunderstand about bare metal, why some teams want VMs and containers living together, where Kubernetes is still the right answer, and where an opinionated platform might be a better fit than giving every team every possible knob to turn.

There's also a good thread in here around failover versus active-active systems, stateful workloads, why application -level replication often beats platform-level magic, and what it really means to make raw infrastructure feel like a cloud-like resource.

And towards the end, we talk a bit about AI workloads, GPUs, hype cycles, and why the most bleeding-edge teams are not always the same teams that want an opinionated platform.

So if you work around DevOps, SRE, platform engineering, cloud infrastructure, Kubernetes, private cloud, or you are just starting to wonder whether your modern platform has become a very expensive junk drawer, this one should be worth your time. All right, let's jump in. Today, I'm joined by Jake Warner. He's CEO and founder of Cycle.io.

We're going to be talking about private cloud, bare metal, and why a lot of teams are quietly exhausted by platform complexity. Jake, thanks for joining me. Thanks for having me. Give me your thesis. Why are teams pulling back toward private cloud and bare metal again? So, you know, for most of the people watching this podcast, we all knew that bare metal was really sexy a decade ago, right? 15 years ago.

The cloud made it super easy to move away from bare metal. It solved the big complexity that came into, you know, I mean. We all know how hard it was to get a bare metal server online, get it configured, do it at scale. In the early days of Cycle, I'd always say, you know, anyone can deploy one or two servers. But when you need to start automating 100 servers, that's when the problems get really complex, right?

I'm guessing most people listening to this podcast are completely aware of that. With what we built with Cycle, our goal was to simplify that process so that way you could have companies that were able to own their bare metal, et cetera, and still provision it like a cloud-like resource. And the reason why we kind of did that and why we see companies coming back to bare metal is, number one, cost.

Hyperscalers ended up, you know, they made it really easy. But we've all seen hyperscalers just continue to increase costs. And once you get locked into that ecosystem, your costs kind of only go up. So that's number one. Number two is about data sovereignty concerns.

I think especially with a lot of the geopolitical issues that we're seeing in the world right now, most of the companies that we've been working with are companies that are saying whether it's compliance reasons or they have customer demands, maybe is the right term for it. We have companies that are coming to us and saying. We cannot be on a US-owned hyperscaler.

Even if geographically that infrastructure is sitting over in Europe, we need to, because of the US Cloud Act, we have companies that are saying we cannot be on US -owned infrastructure. So I think that in terms of private cloud, in terms of bare metal, it all kind of comes back to number one, cost, number two, data sovereignty and compliance. And then the third item there would be performance.

And for a long time, I think people kind of... Gave up the performance that bare metal would give because of the ease of the cloud outweighed the performance you got from bare metal. But fast forward to today where you have platforms like what we've built that can make bare metal a cloud -like experience on top of bare metal.

It allows those organizations to go back toward performance, lower their costs and own more of their stack in the process. So along those same lines, is there like a common misconception you see that people have regarding? Bare metal or on-prem in general? Oh, absolutely. So, you know, as recently as, I mean, it happens all the time, but we were doing a demo back in November.

And this was with a large company over in Denmark. And they were one of the early companies to adopt AWS. And this company, they were looking at getting away from Kubernetes. They were using EKS. And the main reason they were coming to us was data sovereignty issues. They had customers that were saying that they needed to not be on AWS to be able to continue to be customers of that company.

And during that demo, one of the requirements, because I kind of posed the question, like, are you interested in just getting off AWS? Are you also interested in adopting bare metal as part of this. And the company was like, no, no, no, we don't want to talk bare metal. Like we don't, we don't want to be responsible for maintaining hardware.

We don't want to be responsible for setting up all the networks and all of those things. And you could tell, again, very technical team, but they had spent so much of the last decade plus in the cloud that they were just kind of used to that. And we did the demo and three quarters of the way through, I was like, you know, hey, we're talking about all these DevOps terms.

We're talking about containers, infrastructure, et cetera. What if I do the rest of this demo and do it on bare metal? But not tell them until afterwards. And it was kind of like a game to me. Like I had someone kind of challenge my thoughts on, you know, the ease of bare metal.

And so we got to the end of the demo and I showed my cards and I was like, oh, by the way, like the infrastructure we just deployed and the containers we just deployed, that was all bare metal. And the company immediately did a 180. And they're like, okay. You've sold us. Let's talk about bare metal now.

Because they realized it was one of those things where, you know, kind of to the three points that I just made about cost, data sovereignty and performance. The main item for them was data sovereignty. But when they realized that now they could have that cost and performance conversation at the same time, and it wasn't necessarily exclusive, it was kind of eye opening.

So yes, back to the question that you asked, which is what is the big misconception? There's a lot of people that have spent much of their careers in the cloud. I mean, again, it's been the main way we've been deploying for a decade now. That because of that, they've kind of, what's the whole cliche?

Going through the forest or the mountains or you're missing, I don't remember the cliche, but the point is that when you're so focused on something, you kind of miss some of the advancements that are happening in the process. Yeah, for sure. I mean, so I am one of those people where I was early days, I worked at a colo facility, racking and stacking, you know, servers.

Very, very different world now than it was back in the late 90s, early 2000s. Yeah, very different as far as management of those colo facilities. Completely different now. With those colo facilities, it's really interesting because so many of the people that I encounter that are bare metal friendly and like ready to return to it are the people that were racking and stacking back at that time.

And, you know, the early cPanel days and they're like, I want to get back to that because they kind of missed that. Like it means something to them. But then you have this whole generation kind of in the middle where cloud was the beginning of their career. So as you talk about racking and stacking, like that is dear to me as well. Yeah.

So when you say turn any infra into a private cloud, what does that mean for a DevOps team day two? So with Cycle, the way we kind of approach things, from a philosophical standpoint, I'm kind of anti-infrastructure as code. Like in Cycle, we call it environments as code. The idea is that you have a thin but defined line between infrastructure and everything that's going to run on top of that.

And so in Cycle, we have what we call environments as code, which is... Called stacks. And that's where you define your load balancers, containers, everything that you are going to be deploying over and over and over. But then infrastructure, we treat as just a pool of resources.

So I guess maybe that's kind of a throwback to the racking and stacking days that we're just talking about where you're taking cPanel and trying to pack these hosts with as many things as you can and so with Cycle the idea is that you know you bring bare metal you bring vms whether it's in the cloud or we have we have some companies that are running Cycle on top of vmware the idea is you bring raw compute a cycle

Will automatically join it into a mesh and then that becomes kind of your private cloud so it can be hybrid infrastructure multi-cloud fully on-prem or a mixture of all of the above And then applications that you're building, your networks, et cetera, just overlay on top of that.

So again, you have this thin but very defined line between infrastructure on one side and your applications on the other. So one of the things that I believe I read on, I believe it was on your website, you say no DevOps army required. What does that mean in practice? So I've been writing code for, I don't know, probably 22 years now.

I guess there's the common saying that the best DevOps engineer will automate themselves out of a job, whether that's true or not. Up for continuous debate, but I've been writing code for a long time and it seemed like any time it came to actually deploying that, whether it was back in the cPanel days or with Docker or Kubernetes, et cetera, that was always the thing that slowed me down the most.

And while I always loved working with infrastructure, I didn't really like maintaining it. That was always a thing like, I mean, sure, getting it up and running is fun, right? I mean, it's why I think most of us get into it. The first days of setting up infrastructure and, you know, placing an order to buy a new bare metal machine. That's exciting, right?

But, you know, three months later, no one wants to maintain it anymore. And so the goal cycle was how can we how can we build how can we build a platform that allows developers to do the things that typically you would need a DevOps engineer to do? And I'm not saying that, you know, to fully replace DevOps engineers, but in many cases, companies that are coming to us are companies where it's very engineering heavy.

So, you know, like 10 developers to every DevOps engineer or a greater ratio. We have some companies that are 25 to one, et cetera. And those seem to be the places where we do best. And one of my favorite components of that is when we have developers that are deploying things on top of the platform that aren't really DevOps engineers at all. And they can't tell you how they did something, but it works.

And, you know, that's one of the things that is nice about that. So no DevOps army is required is just that philosophy. Like, how do we empower developers to do what they need to do without having to become DevOps experts as part of that? So talking about mixed workloads, can you give me like a real world workload mix? Like, why do teams want VMs and containers living together? Why would they want that?

There's probably... Hundreds of different potential reasons. One company that I was helping earlier this morning are running all of their different microservices in containers. That's just how they built their platform. But then due to a whole bunch of legacy, their company that's been around for. I think, 18 years.

They have a number of applications that are built on top of .NET and require Microsoft SQL and things like that. For them, all of their newer microservices are running inside of containers. But for them being able to have those legacy applications running inside of Windows VMs sitting on the same infrastructure, on the same network where to the containers, they don't know they're talking to a VM and vice versa.

It's just fully abstracted. So everything just sees network endpoints, but it allows companies not have to change everything for adoption. So that's one. And I guess another use case is we have some companies that are in regulated industries and things like that where.

For certain applications, they need, you know, true virtualization for isolation as opposed to, you know, just cgroups and, you know, the isolation that comes with containers. Yeah. I feel like as an industry now too, everybody wants to containerize every service and makes sense in a lot of cases.

But I've also found that there are cases where putting it inside of Kubernetes or some sort of container orchestration layer doesn't always make sense. We're just doing it because that's the phase right now. It just seems like every trade show I go to, too, it's pushing this idea of containerization, which, again, microservice layers make sense.

There's certain applications, though, where I don't think putting bridges and stuff inside of a container makes sense, especially when you're dealing with services that can't be disrupted. It's an issue that I've come across a lot in the last year or two. Yeah, I mean, I would say that I'm probably on the opposite side of that. I containerize everything that I possibly can. But I mean, I also put the platform for it.

So, you know, kind of aligns with my belief there. But yeah, I containerize everything to the point that on Cycle, every virtual machine is also actually a container. And so that way we have container layer, we have a containerized hypervisor, and then you have your VM sitting inside of that.

And so the nice thing about that is that as we roll out new versions of the hypervisor, you know, given that it's containerized from that perspective, we're able to kind of roll out different versions of the hypervisor as users opt into it. So, you know, kind of little benefits from that. So that's interesting. So let's talk, I guess, a little bit more about the containerization aspect.

In Kubernetes, I'm dealing with PDBs, pod disruption budgets. I'm dealing with maybe Karpenter if I'm on AWS. So I'm having this scale up and down nodes as needed. And then if I'm dealing with spot instances, I could have disruptions to...

Workloads how does cycle handle that orchestration of services so there's two ways that cycle approaches that number one is that in general I believe that the idea of failover is a terrible idea and I've seen so many times in my career where you know you set up all these processes for failover and then things fail to fail over when things go wrong right and so um with that you know the idea that failover is you know kind of high risk.

It's also along the same kind of philosophy that, you know, less moving parts is better. So inside of cycle, when we talk about like stateful workloads and things like that, in general, the approach that we have is to run everything in active active 100 % of the time. So that way, if one side goes down, you're perfectly fine. Like, you know, when that side comes back up, you know, you'll recover.

Now, if you need to evacuate, you know, to another host, you can do that too. But there's a reason why, Mongo and so many of these modern database technologies have their own replication built into it.

And I guess maybe that's a little bit of a tangent is it bothers me so much when platforms try to automate storage replication, as opposed to letting the application that knows how to replicate it properly, replicate it. And so in Cycle's world, we try to, everything is active, active. We try to move things as little as possible.

And then if you are deploying a container and you mark it as stateful, the platform will treat that container separately, almost like a VM. It'll almost treat it like a VM. So that way its data will always move with it. Like we will like, instead of like creating like a volume claim in Kubernetes, we will create an attached LVM, like a raw LVM, and then migrate that as needed.

And if you need to scale up, we'll scale up, but we're going to rely on the underlying application to know how to replicate that data. Because again, if we're talking about like Mongo as an example, being able to have collection-level locking as you do migrations and things like that, as you have elections happening.

Great why should a platform try to do that in general what would be like a best bare metal use case right now given the trajectory of the industry I mean I think that For companies that are trying to reclaim ownership of their stack, bare metal is a great way of doing that. At that point, you're just consuming raw compute. The different services that you're kind of locked into are significantly less.

I mean, I guess it depends on what provider you're going with and things like that. But for the most part, your pricing is going to be significantly more stable. For example, we just signed a partnership today to be announced soon that includes 100 terabytes of bandwidth per service. Server on a 10 gig link.

And, you know, I mean, you know, when you start talking about 100 terabytes of bandwidth of bandwidth, per, you know, VM or, or whatever AWS, you're talking about a huge bandwidth bill.

Uh, so, uh, the fact that some of these, these bare metal providers are including a 100 terabytes, um, per server just out of the box with normal pricing is wild um and maybe I should take my earlier response back and instead talk about performance um and performance density right because like we as a company we've been buying some more bare metal um recently for we're getting ready to launch a european control plane

So we're buying more infrastructure for that control plane and the fact that we were able to buy physical machines with 24 physical cores for $280 a month with dual 10-gig links and 100 terabytes of bandwidth included and 192 gigabytes of memory each and RAID 1 with terabyte NVMe drives.

Like, I mean, those same specs would be, I mean, I don't know the exact number, but I'm guessing probably three to four times over at AWS. And then you'd still have the virtualization overhead on top of that. What do you think is the reason for that? Do you think that these colo or server providers are trying to compete with that market? Or what's the reason for the discrepancy in pricing, you think?

Is it just because we're paying so much for the control plane at AWS? That's my theory. I don't actually know. It's wild that you can buy that level of performance for that price. And yeah, like, yes, there's... One of the cons that you typically get from most of these, these kind of bare metal providers today, one of the cons is that from a network perspective, you have way fewer PoPs, right?

So your latency from a network standpoint can be meaningfully higher where if you're using AWS and GCP, I mean, you'll have great latency to almost anywhere in the world where some of these bare metal providers, you might get one or two pops. Now, granted, there are some like Equinix and Megaport that that's what they do and they can still give you, you know, a really solid network. So there's...

You know, uh, pros and cons, but I don't know if having 10-plus PoPs, you know, at GCP is also worth a three X price increase on compute. I don't know if that's the reason why, but I think the network is like the network resiliency is probably the one con to bare metal today. Yeah. I guess it matters what your workloads are if you, if you're latency sensitive, but yeah.

Like if you're doing high traffic bidding applications where you need to bid for, for.

Creative or something I could see you needing to have that low latency for that but I don't think that most people if especially if you're serving websites or serving a web app I don't think for the most part it would matter but also I mean Level 3 is in northern virginia in the same area as you know you could probably buy colo space again I'm sure northern virginia area is is expensive just because dc metro area but You could probably buy Colo servers in that area competitively against AWS's rates as well.

Well, it's kind of funny that you mentioned that. Because of AWS's presence there, we have so many companies that when they're switching to bare metal and using Cycle as part of that process, we have companies that specifically need bare metal in that region. And it's like, hey, you can deploy bare metal anywhere in the world. And they're saying, no, no, no, no. I need to connect to Supabase or...

Some other service running on top of AWS. So even though they're trying to get off of AWS as a cloud provider, they still need to be in proximity to it because so many of the services they're communicating with are still sitting right on top of that infrastructure.

So we have companies that did initial test deployments and they were deployed to somewhere around New York City or somewhere to Atlanta and they would find out that extra. 12, 15 millisecond latency was too much for them. And so they had, so like, it's weird how there seems to be such a, and it's only on US East 1. We don't have any other companies that are like, oh, US East 2 or US West 1 is where I need to be.

But for whatever reason, US East 1, we just have an extreme number of companies that are like, I need to be close to that. And it's kind of coincidental because that's the one that always has an outage or at least it seems. And it's the control plane for the rest of the services.

So what I have found is when US East 1 goes down, if you have services that are in US West 2, maybe like an auto-scaling group that needs to scale up, you know, new instances, it's not able to because it's not able to reach out to the control plane, which lives in US East 1. Yeah. There's still marriage there as much as they maybe try to say that they're isolated. They're not.

Yeah, that is something I've absolutely noticed as well. When US East 1 goes down, you just have to assume that everything else is also impacted. Okay, so speaking about Cycle versus Kubernetes, what do you think that Kubernetes gets right? And where does it become self-inflicted pain?

So with Kubernetes versus Cycle and when is kind of a better fit for one versus the other, it really kind of comes down to the needs of individual companies from a customization perspective. For companies that are really, as I mentioned earlier, kind of engineering heavy, that are like, hey, we're mainly developers. We're very microservice heavy.

We might need object storage or like the requirements that they have are pretty, let's say, you know, commoditized, right? They need some disk. They need some, you know, object storage. They need kind of the primitives. That's where Cycle really shines. But at the same time, if you have companies that are like, hey, we need people to run on.

Very specific hardware with very specific kernel drivers and, you know, things like that. That's where Kubernetes is, you know, a better fit from that perspective because with Cycle, we ship a standardized OS to everyone. We don't provide SSH access. That OS is made to be as dumb and as tiny as possible. You know, it's 40 megabytes in size.

So that means that naturally it's going to be kind of limited in terms of what it can do. Like our goal is to target 80%, right? But if you have like really specific infrastructure you need to run or, you know, Supercomputing applications or some of these big AI models that people are buying a whole bunch of really sophisticated infrastructure for these days. Cycle today is not built for bring anything.

We are built for more of bring your typical x86 server you know with a few drives in it we'll get it up and running we'll get you what you need so it all comes back to whether do companies need just basic primitives and they don't want to be devops or do they have a very specific list of requirements um that you know extend from network to hardware to um maybe the oa the host os that those nodes need to run people

Who are coming to cycle are looking for an opinionated answer it's why most of the companies that are on Cycle today are companies that left Kubernetes for Cycle like you know they went that route they tested it um many of them were on Kubernetes for years before they decided like I think that it's natural for for all of us techies to want to play with new technologies but at some point people are like you know what

I don't I don't want to play with all the bells and whistles anymore I just wanted to work yeah and it's something that you know I kind of talked about often is you know most people when they get their first smartphone or at least you know for me and most of my friends as we were growing up your first smartphone is an android right um and I know I'm probably about to piss off a whole bunch of people but you're for

You know typically your first phone is an android phone because you want to you want to customize it you want to you know play with it you want to make it yours yeah yeah exactly um but eventually people are like I don't really care about that anymore I just want a phone that works I don't want to you know like let me change my background and I don't care and then they switch to an iphone um and so that's kind of

What I've kind of always said about cycle like Companies, you know, they're going to go play with Kubernetes.

They're going to go play with Rancher. They're going to go kind of test out, you know, the latest and greatest. They want something where they can change every variable. But at some point, it no longer becomes about changing variables and playing with the latest and greatest of everything. They just want to get back to what they want to build.

And so that's where we built Cycle for companies that are like, yeah, just give me a standard opinionated platform and I'll just work with that. And so, yeah. Yeah, it makes sense. So, okay, wrapping up, what's one thing you'd do first if you had to modernize on-prem or hybrid without blowing up the team? This is going to go back to a conversation we had, you know, 10 minutes ago, but containerize, right?

Like, you know, I'm a big fan of containerizing everything. It makes applications typically way more portable. And if your goal is to chase portability, so that way you can kind of move to whether it's, you know, VMs or bare metal or on-prem, I mean, I guess that's bare metal too. Or mixture. Being able to standardize containers is, you know, I think why I'm in such favor of them.

Yeah, I think that would be my first step. I think that would absolutely be my first step from that standpoint. Now, if the question is, what if I'm already containerized and now I'm trying to like go towards, you know, bare metal? I think the next step there is, you know, look at your dependencies. If you're in a hyperscaler. And you're using things like Lambda and S3 and things like that.

Your next step is kind of to decide what services I'm going to try to replicate on top of bare metal. What am I going to be okay having still in the cloud? And do the primitives that I need allow me to be on bare metal? Because again, if you are tightly integrated into Lambda and some of these other things, bare metal might not be the best fit for you.

But if you're just running a whole bunch of containers, then you're kind of in a good spot. So assuming you're containerized, I think then the next part of that is just... Evaluating what third-party services you need and whether you can bring those in-house or whether you're okay with those staying in the cloud.

Have you noticed an increase in AI workloads and people building containers around AI-specific workloads using Cycle? So it's kind of interesting for how much, you know, you log into LinkedIn and Reddit and things and everything's about AI. So many of the companies that are on Cycle today, like, yes, you know, a number of them do have some AI component. In what they're doing.

But I think we only have one or two clients that are running like true models with GPUs. Yeah. I didn't mean like API calls to OpenAI, but yeah, like actual modeling locally. Yeah. Yeah, so Cycle out of the box supports all NVIDIA data center class GPUs. And so that's one of the drivers that we keep up to date. So we support that.

But we were hoping that we'd have more AI, you know, focused applications on top of the platform today. But I think that it's one of these kind of interesting things where it's kind of like with IoT and then crypto and now AI. Like, yes, you know, there are hype waves. And I'm not saying that AI is, you know, you know.

Equivalent to IoT or crypto or things like that but as we see these hype waves happen we always see people who kind of love bleeding edge technology chase them right like so many of my friends that were like really gung-ho on on crypto are now the same people that are really deep into you know ai and then the next thing that comes out they're going to chase that as well like it's what they do and those people

Typically are the same people that like changing all the, you know, the dials and they like playing and tweaking.

And so that's where they kind of, I think those people, they don't want an opinionated platform like Cycle. They want something where they can customize everything. It's kind of like the book of, you know, Crossing the Chasm, right? Cycle is there for the majority in the middle. They're like, hey, no, I just want to run stuff. I want to build stuff.

But then, you know, those super early adopters that are always chasing new technology, they want to be, they want more customization than Cycle will give them. And I guess to an earlier point, that's where Kubernetes probably makes more sense for them. But when applications kind of become more standardized and they just want to run them, then that's where Cycle starts to win. Makes sense. What is Cycle?

Why would someone choose Cycle? What would be a reason for choosing Cycle? Yeah, so for an organization that is really engineering heavy, a company that wants to spend more time focused on building versus maintaining. And when I say building versus maintaining, I'm talking about building the actual applications, the platforms, the services, etc.

Less on maintaining the host OS, the host kernel, the underlying infrastructure. That's where Cycle really shines for these organizations, especially teams that where they have. Really talented developers, but these developers really don't have interest in becoming DevOps engineers as part of that process is kind of where Cycle shines. Makes sense.

Okay, so wrapping up, where can people read more about you, find out about you, read more about Cycle? Where should they go? Yeah, so to learn more about Cycle, you can visit our website, which is Cycle.io. We also have a Slack community that we have a lot of developers and DevOps engineers that hang out in. That's slack.cycle.io.

And then for people who want to maybe learn more about me, I guess linkedin.com/in/jakewarner. Like it's kind of weird to hand out my or to use my LinkedIn as that as the primary source. But I think I keep it more up to date than anything else I do these days. That makes sense. I'll also put links for all of that in the show notes as well. So make it easier. Sounds good. Awesome.

Well, thank you, Jake, so much for coming on and telling me more about Cycle and containerization. Really appreciate it. Yeah, really, really appreciate you having me on, Brian. Really enjoyed the conversation. Always fun to be able to, you know, as you mentioned earlier about racking and stacking, meet people of like mind in terms of how we got to where we are today. Awesome. Thanks. All right.

That was my conversation with Jake Warner from Cycle .io. My biggest takeaway from this one. Is that the cloud versus bare metal debate is kind of the least interesting version of the conversation. The better question is, what are you actually trying to optimize for? Because sometimes the answer is cloud. Sometimes it is Kubernetes.

Sometimes it is managed services all the way down because your team does not have the time, people, or business reason to own more of the stack. But sometimes the answer is different. Sometimes the problem is cost. Sometimes it is performance density. Sometimes it is data sovereignty. Sometimes it is compliance. Sometimes it is latency.

Sometimes it is the fact that your developers just want to ship software and your platform team is slowly drowning in a pile of abstractions that were supposed to make everything easier. That is the part I think is worth paying attention to. A lot of teams do not necessarily want to go backwards. They want to go back to owning the parts that matter without reintroducing all of the old pain.

And that is a much more useful framing than pretending there is one correct infrastructure model for everybody. I also like Jake's point about opinionated platforms. Because as engineers, we love flexibility. We love knobs. We love knowing that technically, if we really wanted to, we could customize every piece of the stack. But there is a cost to that. Every knob becomes a decision.

Every decision becomes something to document. Every exception becomes something to support. And eventually, the platform that was supposed to help teams move faster becomes another system that needs its own platform team just to keep it sane. That does not always mean opinionated platforms are always better.

If you need very specific kernel drivers, specialized hardware, deep customization, or you are doing weird edge case infrastructure work, then Kubernetes or a more flexible platform may absolutely be the better fit.

But for a lot of teams, especially teams that mostly need containers, VMs, networking, storage, load balancers, and a sane way to deploy applications, there is a real argument for fewer choices and better defaults. And honestly, that is probably where a lot of infrastructure conversations are heading. Not everything should be cloud. Not everything should be Kubernetes. Not everyone should move back to bare metal.

More like what complexity is actually helping us? And what complexity are we just carrying? Because the industry told us this is what modern infrastructure is supposed to look like. I'll have links to Jake, Cycle.io, and their Slack community in the show notes. If you enjoyed this conversation, follow or subscribe to Ship It Weekly wherever you listen to podcasts.

It helps the show, and it makes sure you get both these conversation episodes and the weekly DevOps, SRE, platform, cloud, and security news recaps. You can also find the show notes and links over on shipitweekly.fm. Thanks for listening, and I'll see you later this week.

A repo named Private-CISA was reportedly public on GitHub, which is a pretty painful reminder that naming something private is not access control. This week, we've got leaked AWS keys, internal deployment docs, Terraform, Kubernetes manifests, Argo CD files, AI root cause analysis, agents clicking through legacy apps. Claude Code showing up in CI/CD.

And another Kubernetes security reminder hiding in a boring default. The theme is pretty simple. Automation is getting more powerful. But we are still leaking secrets, missing defaults, and writing postmortem action items that quietly go to die. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly.

Welcome back to Ship It Weekly, the show where we look at the DevOps, SRE, cloud, platform, and security stories that actually matter when you are the person who eventually has to keep the thing running. This week, we're starting with the CISA contractor GitHub leak. Then we'll get into AWS DevOps agent and automated root cause analysis.

Microsoft Copilot Studio computer using agents, Atlassian adding cloud code to Bitbucket Pipelines, and CVE-2026-46333 with the Kubernetes seccomp angle. Then in the lightning round, we'll hit GitHub expanding OIDC support for Dependabot and code scanning. Java pods getting OOMKilled even when heap looks fine. And why LLM-generated SQL can be wrong in ways that still run.

And for the human closer, we'll talk about why postmortem action items die. So let's get into it. First up, the big one. A contractor for CISA reportedly had a public GitHub repository named Private-CISA. Again, Private-CISA. That is like naming an S3 bucket definitely not customer data and hoping AWS respects the vibe.

GitGuardian says it found the public repo on May 14th containing around 844 megabytes of exposed data. Reporting included plain text passwords, AWS tokens, Entra ID SAML certificates, and internal material tied to CISA systems. And this is the part that matters for us. This was not just one forgotten .env file.

The reporting describes CI/CD logs, Kubernetes manifests, Argo CD files, Terraform code, GitHub Actions workflows, deployment details, and internal docs. This is not only a secret leak. This is context leakage. Credentials are bad. AWS tokens are bad. Plain text passwords are bad.

But when you also leak deployment docs, infrastructure code, manifests, CI/CD logs, and internal workflows, you are giving someone more than a key. You are giving them the floor plan. Now, we should be careful. From the outside, we do not know the full operational impact. Public reporting says CISA was notified and the repo was taken down. But the lesson is already clear.

Secrets scanning matters, but it is not enough. The repo should not have been public. The archive should not have been there. The credentials should not have been valid. And once operational context leaks, the response cannot just be rotate the key and move on. You need to ask what the leaked context teaches an attacker. Most teams hear a story like this and think, wow, how does this happen?

And then somewhere in their own org, there is a repo called old prod migration backup final last touched by a contractor in 2022. Containing a zip file nobody wants to open because it feels cursed. Most of us have some version of this risk. Old repos, personal forks, contractor projects, build logs, Terraform state, kube configs, migration scripts, weird backup folders.

So the takeaway is not just don't commit secrets. That is true, but it is table stakes. The better takeaway is to inventory the weird stuff. Look for old repos, archived repos, contractor -owned repos, personal forks, internal examples, demo projects, and places where somebody may have dumped operational context while trying to move fast.

And when a leak happens, treat the context as compromise too, because sometimes the key is the headline, but the deployment map is the real prize. Next up, AWS published a post on using AWS DevOps Agent to automate root cause analysis across Datadog and Elasticsearch with CloudTrail and EKS involved too. This one sits right at the intersection of useful and slightly unsettling.

The AWS example starts with a Datadog alert. AWS DevOps Agent gets access to EKS. So it can describe Kubernetes objects, pull pod logs, and look at cluster events. It also correlates Elasticsearch logs, Datadog metrics, and CloudTrail deployment events to figure out what changed. And honestly, that sounds a lot like incident response. You start with one symptom. Then you try to rebuild the timeline. What changed?

What deployed? Which pod restarted? What metric moved first? Did someone push a config change? Did a dependency decide today was a great day to build character? Half of incident response is not fix the thing. Half of it is reconstructing the story fast enough that you fix the right thing. So if an agent can gather context and build a decent first-pass timeline, that is useful. Logs over here. Metrics over there.

CloudTrail in another tab. Kubernetes events in a terminal. Datadog on one monitor and Slack on another. And someone asking an update every 90 seconds, which is understandable and still emotionally damaging. But automated RCA can become very convincing very quickly.

A tool that says probable root cause with a clean summary can easily become the thing everybody believes, especially when the incident channel is noisy and everybody is tired. So I like the direction, but I would treat AI RCA like a fast incident scribe or junior investigator. Great at pulling threads. Useful for assembling context. Helpful for reducing time to understanding. But not the final authority on causality.

Humans still need to ask, is this correlation or cause? Did this happen before or after customer impact? What else changed? What evidence would prove this wrong? Because if the incident review action item is that AI said it was Elasticsearch, so we restarted Elasticsearch, that is not RCA. That is vibes with a dashboard. Before we get to the next story, a quick note from this week's sponsor, Guardsquare.

If you are building mobile apps, good enough security is usually a problem waiting to happen. Guardsquare focuses on actually protecting your code in addition to scanning it. That means code hardening, runtime protection, testing, and visibility into what's happening once your app is out in the wild.

So if you are responsible for shipping and securing mobile apps, Android or iOS, Definitely worth taking a look at guardsquare.com. Alright, back to the show. Third story. Microsoft says computer using agents in Copilot Studio are now generally available. This one is interesting because it changes the automation boundary. A lot of automation assumes there is an API. You call an endpoint. You get a response.

You wire it into a workflow. Everyone pretends the internal CRM is not held together by three workflows, a CSV export, and one person named Linda. Computer using agents are different. Microsoft describes these agents as interacting with graphical user interfaces, websites, desktop apps, screens, buttons, forms.

So instead of saying there is no API, so we cannot automate this, the pitch becomes the agent can use the UI like a human. And look. I get why this is appealing. Every company has legacy systems. Every company has some vendor portal that looks like it was designed during the emotional low point of enterprise software.

Every company has workflows where someone copies data from one screen into another and calls it a process. If an agent can take some of that away, great. Nobody needs a fulfilling career in manually clicking invoice screens. But this is also where the risk gets weird. API automation usually gives you structure. Scopes, endpoints, logs, schemas. UI automation is messier.

The agent is looking at screens, reading labels, clicking buttons, entering text, and deciding what to do next. Which means your automation path may now include a model looking at a webpage and deciding which button seems right. That sounds funny until the button says submit payment, delete record, approve request, or yes, I understand this is permanent. So the governance questions matter.

What apps can the agent access? What account does it use? Can it reach production admin screens? Are actions auditable? Can it pause before destructive steps? What happens if the UI changes slightly? And who owns the outcome when it clicks the wrong thing? Computer using agents may become useful because plenty of enterprise systems will never get good APIs.

But when the agent operates through a UI, treat that UI like an automation interface. Use restricted accounts. Use test environments. Use approval gates. Log actions. Limit destructive workflows. And be very suspicious of anything involving bulk update. A computer -using agent is not just a better macro. It is automation with eyeballs.

And like most things with eyeballs in enterprise software, it probably needs supervision. Fourth story. Atlassian says agentic pipelines now support Claude Code as a provider in bitbucket pipelines. This sounds like a feature announcement until you think about where it lives. It lives in CI/CD.

Atlassian's examples include README updates, security report triage, feature flag cleanup, PR descriptions, and other repetitive engineering chores. And this keeps coming back to a point I've made before. Developer tooling is production now. CI/CD is not just the stuff around production. It is the path code takes to become production.

So when you put agents inside pipeline workflows, you are not just making developers faster. You are changing the delivery path. And yes, some of these tasks are genuinely annoying. Security report triage, feature flag cleanup, PR descriptions, documentation updates. Please take them. Nobody is sitting around hoping for more stale flags. And slightly wrong README files.

But once an AI agent is part of a pipeline, the boring questions matter. What repository context does it get? What logs does it see? What secrets are visible to that step? Can it modify files? Can it open PRs? Can it change tests? Can it generate security triage notes that humans treat as authoritative? Can it make a flaky test look fixed by weakening the assertion? That last one is not me being dramatic.

If the task is fix the failing pipeline, a bad agentic workflow might make the pipeline pass without making the system better. And yes, humans do this too. We just call it temporary and let it survive three reorgs. Atlassian also points users to guidance around third-party agent providers and data handling.

That matters because if you bring Claude Code into a pipeline, you need to understand what code, prompts, logs, and generated context may leave your environment. That does not mean do not use it. It means do not discover your data flow by reading the invoice. Treat agentic pipeline steps like any other privileged CI step. Start with low-risk tasks. Avoid secrets. Avoid production deploy authority.

Make outputs reviewable. Require human approval when the agent changes code. And document what data goes to the provider. The agent does not need to be terrifying, but it also should not be a surprise guest in your delivery path. Fifth story. Let's talk about CVE-2026-46333 and Kubernetes seccomp defaults. This one is more technical, but it's a good grounding story after all of the AI agent stuff.

Because sometimes the most important security decision is not a new tool or a new model. Sometimes it is whether your pods are running with the syscall profile that you thought they were running with. NVD tracks CVE-2026-46333 as a Linux kernel issue in the ptrace path. Qualys describes it as a local privilege escalation and credential disclosure issue.

The Kubernetes angle is that unset or unconfined seccomp profiles can leave pods exposed to the tested path, while runtime default block the tested pidfd_getfd path. PSS restricted added more protection as well. The operator version is simple. Your Kubernetes security defaults matter. And they matter most when nobody is thinking about them.

Seccomp is easy to mentally file under container security stuff we should revisit someday. Which is engineering speak for future incident seasoning. Kubernetes docs say that if seccompDefault is enabled, pods use the RuntimeDefault seccomp profile when no other profile is specified. Otherwise, the default is unconfined. That difference matters. Because unset does not always mean safe.

Sometimes unset means congratulations. You are raw -dogging syscalls in production. Probably do not put that in your architecture diagram. So the takeaway is straightforward. Check whether RuntimeDefaults is actually being applied in your clusters. Review your pod security standards posture. Know where unconfined seccomp is allowed. Know where privileged pods exist. Know which namespaces have exceptions.

And if you use managed Kubernetes, do not assume the provider magically made your pod security posture sane because the control plane has a nice logo. Containers share the host kernel. That is the deal. So when there is a local kernel bug and your workload security posture allows the relevant path, the cluster configuration suddenly matters a lot. The boring defaults are not boring. They are latent decisions.

And every once in a while, a CVE shows up and asks what you decided. Now let's do a quick lightning round. First, GitHub expanded OIDC support for Dependabot and code scanning. GitHub says that Dependabot and code scanning now support OpenID Connect authentication for organization-level private registries for Cloudsmith and Google Artifact Registry.

The short version is fewer long-lived registry credentials sitting around as secrets. And that is good. OIDC -based access is not magic. But short-lived identity-based auth is usually healthier than, here's a token, Please don't leak it. Best of luck to everyone involved. Second, Java pods getting OOMKilled in Kubernetes even when the heap looks fine. Classic operator trap.

Someone sets -Xmx, looks at heap, and thinks we're fine. Then Kubernetes kills the pod anyway. And everyone stares at the graphs like the cluster betrayed them personally. But JVM heap is not the whole memory footprint. Metaspace, direct buffers, thread stacks, native memory, JIT. GC overhead, and off -heap usage still count towards the container memory limit.

Kubernetes does not care that your heap looked reasonable. It cares that the process crossed the limit. So if your Java pods are getting OOMKilled, do not only look at -Xmx. Look at total container memory and leave headroom. Because production loves punishing tight memory math. Third, LLM-generated SQL can be wrong in ways that still run. A broken query that fails loudly is annoying, but at least you know it failed.

A query that returns plausible nonsense is worse. The dashboard loads. The numbers look reasonable. Someone puts it in a slide deck. And now your business metric is powered by a hallucinated join and a missing filter. So text to SQL needs guardrails. Read-only roles. Query limits. Schema-aware validation. Known templates where possible. Human review for anything important.

The danger is not always that AI writes broken SQL. Sometimes it writes SQL that is wrong quietly. And quiet wrongness is how bad decisions get confidence. The Human Closer this week is about postmortem action items. Because if there is one place that engineering organizations consistently lie to themselves, it is the bottom of a postmortem document. Not maliciously, just optimistically. The incident happens.

People jump in. The team writes a timeline. There is a good discussion. Nobody blames anyone. Everybody agrees on what went wrong. Then the action items show up. Improve monitoring. Review runbooks. Add better alerting. Investigate retry behavior. Clean up ownership. These are not action items. These are wishes wearing a Jira costume. Incident .io had a good piece on why postmortem action items die.

The reasons are painfully familiar. No named owner, wrong tracking place, vague wording, and no follow -up cadence. That's basically the whole game. A postmortem action item without an owner is not an action item. It's a group hallucination. An action item that lives in a doc that nobody opens again is documentation tax. And an action item that says improve monitoring is not an action item. It is a mood.

A real action item sounds more like Maria adds a replication lag alert for the payment database by Friday. Or Kevin removes production deploy access from the old CI token before the next release. Named owner. Specific verb. Clear outcome. Real tracking location, due date. That does not make the work easy, but it makes it real. This connects back to every story this week.

The CISA leak is not fixed by saying review GitHub practices. AI RCA is not useful if the follow-up is improve incident response. Computer using agents are not safe because someone wrote ensure governance. Kubernetes seccomp is not handled because someone says harden workloads. This is the staff and principal engineer part of the job that does not always look exciting on a roadmap.

Turning vague risk into specific work. Turning incidents into system changes. That is where reliability actually improves. Because production does not care how good the postmortem sounded. Production cares whether anything changed. Okay, that's it for this week of Ship It Weekly.

We covered the CISA contractor GitHub leak, AWS DevOps agent and automated root cause analysis, Microsoft Copilot Studio computer using agents, Atlassian agentic pipelines with Claude code, CVE-2026-46333, and Kubernetes seccomp defaults, plus a lightning round on GitHub OIDC support, Java pods getting OOMKilled, and LLM-generated SQL.

If this episode was useful, follow or subscribe wherever you are listening or watching. If you're on YouTube, hit subscribe. If you're in a podcast app, follow the show there. And if you know somebody on a DevOps, SRE, platform, security, or engineering leadership team who is dealing with secrets, agents, Kubernetes defaults, or postmortem follow-up, send this one to them. It helps the show grow.

And honestly, it helps me keep making this kind of content for people who actually live with these systems. You can find the weekly brief at OnCallBrief.com and more episodes and show notes on ShipItWeekly .fm. I'm Brian from Teller's Tech, and thanks for listening.

And remember, if your repo is named Private-CISA, your agent can click buttons, your pipeline can call Claude, and your postmortem action item says improve monitoring, maybe take a breath then go find the owner the token the default and the ticket because production does not run on good intentions it runs on the stuff that someone actually fixed

AI agents just got APIs. They got identity. And they're starting to plug into the automation tools teams already use to change real systems. So the question is moving past, can AI write code? The better question is, what happens when AI can open pull requests, call tools, authenticate to services, and trigger operations workflows? Because at that point, you did not build a chatbot.

You built a coworker with API access. I'm Brian Teller from Teller's Tech, and this is Ship It Weekly. Welcome back to Ship It Weekly, the show where we look at DevOps, SRE, cloud, platform, and security stories that actually matter when you're the person who eventually has to keep the thing running.

This week we're looking at GitHub making Copilot cloud agent tasks available through a REST API, Auth0 bringing authentication to MCP servers, Red Hat positioning Ansible as an execution layer for agentic IT operations, and OpenAI Daybreak pushing AI deeper into security research and remediation.

Then we'll step away from the AI cycle for a really good Discord engineering story on automating ScyllaDB operations at scale. And in the lightning round, we'll hit AWS GuardDuty and crypto mining detection, queues and backpressure, and why an index scan can still ruin your day. The theme this week is authority, not intelligence, not productivity, authority. What can these agents reach? What can they change?

Who approved the action? And when something breaks, who owns it? That's the thread for this episode. So let's get into it. First up, GitHub Copilot Cloud Agent Tasks can now be started through the REST API. This is the right place to start because it sounds like a small product update, but it changes the shape of the thing.

GitHub says Copilot Business and Enterprise users can now programmatically start Copilot cloud agent tasks through a new Agent Tasks REST API, currently in public preview. The Copilot cloud agent works in the background in its own development environment. It can make code changes, validate those changes, and open a pull request. That part alone is already interesting. But the API is the bigger shift.

Because now this is not just a developer manually asking Copilot to work on something from inside GitHub. Now another system can kick it off. That means you could wire this into custom workflows. A support escalation, a bug triage process, a security finding, a dependency update workflow, a backlog grooming process. Or whatever else somebody decides to connect. And that's where this gets operationally interesting.

Because once an agent can be started by automation, it becomes part of your automation surface. It becomes something you need to reason about like any other system that can create change. What repos can it touch? What permissions does the token need? Who approved the task? What branch protection applies? Can it create a pull request but not merge one? Can it trigger CI? Can that CI deploy?

And if the workflow is kicked off by another tool, do you still have a clear human owner? That last one matters because it is very easy to imagine a chain like this. A vulnerability scanner opens a ticket. A workflow kicks off an AI agent. The AI agent makes a patch. CI passes. A PR gets opened. Somebody rubber-stamps it because the diff looks boring and the scanner says the vulnerability is resolved.

And maybe that is great. Maybe you just saved an engineer three hours. Or maybe you just created a subtle production issue from a change nobody really understood. The practical takeaway here is not don't use it. The practical takeaway is that agent workflows need the same boring controls we already expect from normal engineering workflows. Branch protection. Required reviews. Code owners. Scoped credentials.

Audit trails. Clear ownership. And a very bright line between agent can propose and agent can ship. The interesting part of AI agents is not that they can do work. The interesting part is that we have to decide how much authority that work gets. That leads nicely into the second story. Auth0 announced that Auth for MCP is generally available.

MCP or Model Context Protocol has become one of those terms that shows up everywhere now. It is basically a way for agents and AI tools to connect to external systems, tools, APIs, and data sources in a more standardized way. And that matters because agents are only as useful as the tools they can reach. A model sitting in a chat box can give advice. A model connected to tools can take action.

And once it can take action, authentication and authorization stop being side concerns. They become the whole game. Auth0's announcement is focused on putting an identity layer around MCP servers. They call out authentication, CIMD registration, and on behalf of token exchange. The plain-English version is this. If agents are going to call tools, those tools need to know who or what is calling them. On whose behalf?

And what that caller is actually allowed to do. That sounds obvious, but a lot on-behalf-of token exchange. The plain-English version that feels like local developer convenience first, production safety second. You spin up a server, you connect it to your agent, you give it access to some tools, and suddenly your agent can read things, write things, query things, maybe even change things. That's fine in a sandbox.

It is not fine... When the tools are attached to customer data, production infrastructure, internal admin APIs, CI/CD, billing systems, or cloud accounts. And this is where identity gets weird. Because with a normal user, we mostly know how to think about it. Brian logged in. Brian clicked a thing. Brian had these permissions. With an agent, the story is messier. Was the action taken by the agent?

By the user who asked the agent? By the application hosting the agent? By a service account? By a delegated token? And when something goes wrong, where does accountability land? That's why I think this Auth0 story is more important than it looks. MCP is not just a cute connector system for demos. It is becoming connective tissue for AI tooling.

And connective tissue needs identity, authorization, logging, and revocation. Otherwise, we're just building a faster way for something to call the wrong API with too much permission. For DevOps and platform teams, this is probably where the real work starts. Not how do we let every team use agents, but how do we let teams use agents without turning every MCP server into an ungoverned production backdoor?

Before we get to the next story, a quick note from this week's sponsor, Guardsquare. If you are building mobile apps, good enough security is usually a problem waiting to happen. Guardsquare focuses on actually protecting your code in addition to scanning it. That means code hardening, runtime protection, testing, and visibility into what's happening once your app is out in the wild.

So if you are responsible for shipping and securing mobile apps, Android or iOS, definitely worth taking a look at guardsquare.com. All right. Back to the show. Third story. Red Hat is pushing Ansible Automation Platform as a trusted execution layer for IT operations in the agentic era. That is a very enterprise sentence. But underneath the marketing language, this is actually a big deal.

Because Ansible is not theoretical. Ansible is already used to patch systems, restart services, configure servers, manage network gear, run operational tasks, and handle a bunch of work that is very close to production reality. So when you connect AI agents to Ansible, you are not just giving an agent a little toy function. You are connecting it to the machinery that already changes real systems.

Red Hat's angle is basically this. Agents may be good at reasoning, planning, or interpreting intent, but enterprises still need a governed, trusted, auditable execution layer. When it is time to actually do something. That is the right framing. Because the dangerous version of agentic operations is not an agent saying, here's the runbook. The dangerous version is the agent saying, I ran the runbook.

And then everyone hoping it did the right thing. Now, to be fair, this is also where something like Ansible can help. Because mature automation gives you structure. You have inventories. You have playbooks. You have idempotency, at least when things are written well. You have logs. You have a known execution path. You have a place to put approval gates.

That is much better than an agent freehanding shell commands on a production box because it read three Confluence pages and felt confident. But the same rules apply here. The agent should not get more authority than the automation deserves. If your existing playbooks are messy, overly broad, poorly scoped, or rely on tribal knowledge, an agent does not magically make them safe. It may just make them easier to invoke.

And that is the part I'd be nervous about. A bad script that an agent can discover and execute through a tool interface is a different class of problem. So the takeaway is not Ansible plus AI is bad. It is actually the opposite. If agentic ops is coming, I'd much rather see agents routed through controlled automation than improvised commands. But teams should treat this as a forcing function.

Clean up your automation. Narrow the blast radius. Split read-only diagnostics from mutating actions. Make destructive playbooks require approval. Add dry-run modes where possible. Make sure the logs clearly say who asked for the action, what agent or system executed it, and what changed.

Because if Ansible becomes the execution layer for agents, the quality of your automation becomes the quality of your agent safety model. Fourth story. OpenAI announced Daybreak, its cybersecurity initiative built around GPT-5.5 and Codex Security. I'm treating this as a follow-up to the Mythos and Project Glasswing episode, not a totally separate story. Because the broader trend is the same.

AI systems are getting better at vulnerability discovery, exploit reasoning, patch generation, and remediation validation. OpenAI describes Daybreak as a way to use AI for cyber defense. The pitch is that it can help identify threats, generate patches, and verify remediation across code and systems. And on one hand, this is exactly what we want. Most organizations are drowning in vulnerability backlog.

They have more findings than time. Some findings are noisy, some are real, some are technically real but not actually reachable. Some are buried in legacy code that nobody wants to touch. And even when the fix is obvious, there is still work. Open the issue. Find the owner. Understand the code path. Patch it. Test it. Get it reviewed. Deploy it. Verify the scanner is happy and hope nothing broke.

So an AI system that can help triage, validate, patch, and verify is genuinely useful. But here's the uncomfortable part. If defenders get this, attackers get some version of it too. Maybe not the same controlled access. Maybe not the same polished product. But the underlying capability trend is not one-sided. That means the bottleneck for security teams shifts. It is no longer just can we find vulnerabilities.

It becomes can we process, prioritize, patch, and safely ship fixes fast enough. And that lands right in the lap of DevOps, SRE, platform, and application teams. Because finding the bug is only step one. The real work is changing the system. And changing the system safely requires all the boring stuff.

Ownership, tests, CI/CD, feature flags, rollback plans, dependency strategy, runtime visibility, asset inventory, patch windows, and enough architectural knowledge to know when the easy fix is actually a trap. This is why I keep coming back to the same point. AI security tooling will probably find more issues. That is good. It will probably also create more pressure. That is complicated.

If your organization already struggles to patch known vulnerabilities, adding AI that finds more of them does not automatically make you safer. It may just make the backlog more honest. So the real question is not can Daybreak find things? The question is, can your engineering system absorb the findings? Can you validate them? Can you prioritize them? Can you patch them? Can you ship them?

Can you prove the fix worked? And can you do all of that without creating a second incident while fixing the first one? That is where this becomes a operations story, not just a security story. Now let's step away from AI for a minute. Because Discord published a really good write-up on how they automate ScyllaDB clusters at scale. And honestly, this is the kind of engineering story that I love.

Discord's persistence infrastructure team runs a lot of ScyllaDB. Over time, they had accumulated Python and shell scripts to help with operations. But those scripts had the usual problems. They were useful. They were also fragile. They were easy to misuse. They relied on humans understanding the right order of operations. And for complex cluster-wide workflows, that becomes a lot of operational risk.

So they built what they call the Scylla control plane. The goal was to safely automate and orchestrate cluster-wide workflows. Things like rolling restarts, replacing nodes, bootstrapping, and doing work that previously required a lot more manual supervision. One of the details that I liked from the write-up is that webhook notifications mattered more than they expected. That sounds small, but it is very real.

There is also a huge difference between babysitting a terminal for two hours and trusting the system to notify you when it needs attention. That's the difference between automation that technically works and automation that actually reduces human load. And that distinction matters. A lot of teams say they have automation, but what they really have is a pile of scripts.

A script can be automation, but it might not be safe automation. Safe automation needs state. It needs preconditions. It needs retries. It needs idempotency. It needs clear failure modes. It needs visibility. It needs a way to resume without making things worse. And it needs to know when to stop. That last one is underrated. Good automation is not automation that blindly completes the task no matter what.

Good automation is automation that knows when the world no longer matches its assumptions. If a node is unhealthy, stop. If the cluster is already degraded, stop. If replication is not where it should be, stop. If the previous step did not converge, stop. That is how you move from script that usually works to operational control plane. And this connects back to the AI stories in a weird way.

Because before we let agents run operational tasks, we need more automation that looks like this. Explicit. Recoverable. Observable. Constrained. Designed around failure. If the future is agents calling tools, then the tools need to be boring, safe, and well-structured. Discord's story is a reminder that the best automation is not magic.

It is just a lot of careful engineering around the parts where humans usually get tired, distracted, or inconsistent. Now let's do a quick lightning round. First, AWS GuardDuty and crypto mining. AWS published a guide on detecting and preventing crypto mining in AWS environments using GuardDuty. This is one of those classic cloud security problems where security, reliability, and cost all run into each other.

A compromised credential does not always turn into a dramatic data breach. Sometimes it turns into a compute bill. Someone gets access. They spin up resources. They run mining workloads. They try to persist. And by the time anyone notices, the incident is both a security problem and a finance problem. The practical question for teams is simple.

If somebody compromised a credential today and started mining in your AWS account, how fast would you know? Would it be GuardDuty? Would it be Cost Anomaly detection? Would it be Datadog? Would it be a budget alert? Would it be a developer asking why their workload is slow? Or would it be Finance two weeks from now forwarding a bill and asking what happened?

That is the difference between having a detection strategy and having a surprise. Next, queues and backpressure. There was a good piece making the point that queues do not absorb load forever. They delay failure. And that is exactly right. Queues are great for smoothing bursts. They are terrible when teams use them to hide sustained overload.

If messages are arriving faster than consumers can process them, the backlog will grow. A bigger queue does not fix that. It just gives you a bigger place to store the problem. Eventually, you hit freshness issues, storage limits, memory pressure, retry storms, customer-facing delay, or some downstream dependency that finally gives up. So the practical takeaway is simple. Monitor queue depth. Monitor message age.

Monitor consumer lag. Have backpressure. Have limits. Know when to shed load. And please, do not call a system resilient just because it has a queue in front of the fire. Last lightning item. Datadog had a nice PostgreSQL performance write-up about inefficient index scans. The short version is that using an index does not automatically mean a query is cheap.

Datadog walked through a production query where the plan used an index scan, but it was still expensive. They changed the indexing strategy and cut average latency from 300 milliseconds to 38 microseconds. That is a ridiculous improvement, and it is a good reminder. You cannot stop at the query uses an index.

You need to understand whether it is using the right index, how many rows it is touching, how selective the predicate is, what the access pattern looks like, and whether the index actually matches the way the query behaves in production. Sometimes the database is not slow. Sometimes your mental model is. The human closer this week is about authority because that is really what all these agent stories come down to.

Not intelligence, not productivity, not whether the model is impressive. Authority. What is this thing allowed to do? What can it read? What can it change? Can it trigger work? Can it authenticate? Can it call tools? Can it run automation? Can it open pull requests? Can it touch production? And maybe the hardest question, who owns what happens next? Because in real operations, ownership is not optional.

If I write a Terraform change and it breaks something, I own that. If I approve a bad pull request, I own that. If I run the playbook against the wrong environment, I own that. AI does not remove that responsibility. It just makes the path to action shorter. And shorter paths to action are great when the guardrails are good. They are terrifying when the guardrails are vibes.

That is where I think a lot of teams are going to struggle. They're going to treat agent adoption like a tooling rollout. Enable the feature, give access, write a quick policy, maybe do a lunch and learn. And then six months later, they will realize that they created a new automation layer that nobody fully owns. That is not a reason to panic. It is a reason to be deliberate. Start small.

Keep agents in proposal mode before execution mode. Treat MCP servers like production APIs. Treat agent tokens like service accounts. Treat agent created pull requests like code written by a junior engineer who is fast, confident, and occasionally very wrong. And before an agent can run a workflow, make sure the workflow itself is worth trusting. Because the future probably is not humans versus agents.

It is humans deciding which agents get authority, where the boundaries are, and what systems are safe enough to let them touch. That is engineering work. And honestly, it is probably some of the most important engineering work we are going to do over the next few years. That's it for this week's Ship It Weekly. We covered GitHub Copilot Cloud Agent Tasks through the REST API. Auth0 bringing identity to MCP servers.

Red Hat connecting Ansible to agentic IT operations. OpenAI Daybreak and the next phase of AI-assisted security. Discord ScyllaDB automation work. And a lightning round on GuardDuty crypto mining detection, queues, and database indexes. If you found this useful, follow the show. Share it with someone who is either excited or mildly terrified by agentic operations. And check out the weekly brief at OnCallBrief .com.

I'm Brian Teller from Teller's Tech. Thanks for listening. And remember, if your AI agent can open a pull request, call an MCP server, authenticate through your identity provider, and trigger Ansible, congratulations. You did not build a chatbot. You built a coworker with API access. Maybe give it a badge. But maybe don't give it production admin on day one.

A lot of the risk in modern infrastructure is no longer hiding in the parts we used to fear most. Sometimes it is a bad kernel bug. Sometimes it is a broken DNS signature at the TLD layer. Sometimes it is a GitOps upgrade that changes behavior under your feet. And sometimes it is an AI agent with way too much authority and not nearly enough guardrails. This week has a little bit of all of that.

And the common thread is pretty simple. Control planes are brittle. Automation is powerful, and identity still decides whether a mistake stays small or becomes a crater. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech. This is Ship It Weekly, where I filter the noise and focus on what actually changes how we run infrastructure and own reliability. Show notes and links are on shipitweekly .fm.

If the show's been useful, follow it wherever you listen. Ratings help way more than they should. And if you're watching on YouTube, subscribe there too. We've got six main stories today, then the lightning round, and we'll wrap with the human closer. We're starting with the Pocket PocketOS and Cursor incident because it is probably the most instantly gripping story in the set and also one of the most revealing.

Not because AI made a mistake, because an agent got access to something it should not have had in the first place. Then we're going to the .de DNSSEC outage and Cloudflare's response, which is one of those reminders that the internet's deepest layers are still capable of ruining everybody's afternoon, all at once.

After that, Bluesky's outage post- mortem, which is a really good incident story because it is weird, specific, and painfully real. Then Argo CD, with version 3.1.16 being the last 3 .1 release and version 3 .4 .1 bringing a behavior change that people really do need to notice. Then the Linux kernel Copy Fail bug, CVE-2026-31431.

Because active exploitation plus broad Linux exposure is not something ops teams get to casually defer. And finally, Google Cloud Agent Identity and AWS MCP Server GA. Because the cloud platforms are starting to treat AI agents as first -class actors instead of weird sidecars bolted onto existing IAM assumptions. Story one. PocketOS and Cursor is really an identity story. Let's start there.

This week's on -call brief summarized the incident this way. On April 25th, a Cursor AI agent reportedly deleted PocketOS's production database in under 10 seconds after a credential mismatch led it to access an API token it should not have had. The result, according to the reporting cited in the brief, was a full production data loss, including backups.

This is obviously a wild headline, but I think the more important part is not the speed or even the AI branding. It is the access path. Because if an agent can see a destructive credential, then the real problem started before the agent ever took action. That is why I think that this story is more useful than just look at what AI did.

It is really about credential sprawl, environment separation, and what happens when people start granting machine -assisted workflows access that was probably too broad even for a human. An AI agent did not invent bad blast radius design. It just exercised it fast. And that is the practical takeaway I'd want teams to hear.

If you are adding AI coding agents or operational agents into real environments, you do not get to treat identity as a cleanup task for later. The difference between helpful automation and production incident is often just one token, one role, one environment boundary, one missing approval step. So yeah, this story is dramatic. But the lesson is boring. And boring is good. Scope the credential.

Separate staging from prod. Assume the agent will eventually try something dumb, overconfident, or surprising. Because sooner or later, it will. Story 2. The The .de DNSSEC outage is a reminder that internet plumbing still has global blast radius. Next up, the .de outage.

Cloudflare says that on May 5th, at roughly 19:30 UTC, DENIC, the operator for the .de TLD, started publishing incorrect DNSSEC signatures for the .de zone. Any validating resolver that received those records was required by DNSSEC rules to reject them and return SERV- FAIL. Cloudflare says 1.1.1.1 was no exception.

That means a bad cryptographic state at the TLD layer was enough to make huge numbers of .DE domains effectively unreachable. That is a great outage story for ops people because it is one of those failures where everything is technically working exactly as designed and that is the problem. DNSSEC is there to make sure responses are authentic. The signatures were wrong.

Resolvers did the correct thing and refused to trust the data. And suddenly, correct behavior becomes mass breakage. That is such a brutal but important reliability lesson. Security mechanisms do not eliminate failure. They sometimes concentrate it. Cloudflare's write -up is also good because it walks through mitigation trade -offs instead of pretending there's always a clean answer.

When the trust anchor above you is wrong, your choices get ugly fast. Do you keep strict validation and break reachability? Or do you apply a temporary exception and accept the risk of bypassing a protection that exists for a reason? That is not really a DNS -only lesson either. It is a control plane lesson. When a higher order authority goes bad, the downstream systems that trust it do not fail independently.

They fail together. So if I'm taking one operator lesson from this, it is this. Any system built on centralized trust, centralized signing, or centralized metadata needs a very honest failure mode conversation. Because when the thing at the top is wrong, your elegant security chain can turn into synchronized outage machinery very quickly.

Before we get to the next story, a quick note from this week's sponsor, Guardsquare. If you are building mobile apps, good enough security is usually a problem waiting to happen. GuardSquare focuses on actually protecting your code, in addition to scanning it. That means code hardening, runtime protection, testing, and visibility into what's happening once your app is out in the wild.

So if you are responsible for shipping and securing mobile apps, Android or iOS, definitely worth taking a look at guardsquare.com. Alright, back to the show. Story 3. Bluesky's outage post -mortem is the kind of weird incident story worth studying. Now for my favorite kind of reliability story. Blue Sky published a post -mortem for its April outage.

And the post says that the service was intermittently down for about half its users for around eight hours. Jim Calabro's write -up says they saw pretty quickly that they were exhausting ports. But the harder part was figuring out exactly where and why. The root issue involved memcached traffic, ephemeral port exhaustion, and a cascade where the debugging path became part of the failure path.

What makes this one so good is that it is not just we ran out of ports. The postmortem explains that logging used blocking write syscalls, and under the load of trying to log huge volumes of errors while still serving traffic, the Go runtime spawned far more OS threads than normal. BlueSky says that extra thread pressure then hit the garbage collector, contributing to the broader failure cascade.

The write-up also shows a custom dialer workaround that randomizes loopback IPs to avoid ephemeral port exhaustion on a single address after restarts. That is such a real incident pattern. The original fault hurts. The observability path amplifies it. The recovery path gets noisy. And then the system that is supposed to help you reason about the incident starts participating in the incident. That is not hypothetical.

That is production. And I also like the honesty in the write -up. BlueSky basically says the signal Bluesky basically says the signal and that you need the discipline and the metrics to cut through it. That lands because it is true. A lot of outages are not invisible. They are just obscured by too many symptoms arriving at once. So my takeaway here is not just watch ephemeral ports. It is broader than that.

Watch where your debugging strategy becomes a scaling liability. Watch where synchronous behavior hides inside paths you think are harmless. And watch for failure modes where saturation causes your tooling to become part of the blast radius instead of part of the recovery. Story 4. Argo CD is giving people one quiet end-of-life and one not-so-quiet behavior change. Next up, Argo CD.

Argo's version 3.1.16 release is the final release in the 3 .1 series. The release notes are very explicit. As of May 6, 2026, 3 .1 has reached end of life and will no longer receive bug fixes or security updates. The same release notes tell operators to move to a supported version, meaning 3 .2, 3 .3, or 3 .4. That alone is worth a mention because GitOps tools have a way of becoming background furniture.

Teams stop thinking about the controller version because the controller is just there doing its thing. Right up until the day it matters a lot. But then there is the second part. Argo CD version 3 .4 .1 is the first release in the 3 .4 series. And the release notes call out an important change. Following Helm 3 .19 .0, Argo CD now aligns its interpretation of Kubernetes cluster version with Helm's behavior.

OnCallBrief points out that this affects application sets that filter clusters by Kubernetes version. That is exactly the kind of change that looks tiny in a release note and then quietly breaks assumptions in real environments. So for me, this is a classic operator story. Not sexy, not viral, very real. One branch is dead. One branch changes parsing behavior.

And if your GitOps setup depends on version -based selection logic, you need to actually test that logic instead of assuming a minor release means minor consequences. The practical lesson is simple. Treat controller upgrades like control plane changes, not package refreshes. Because that is what they are.

If the thing deciding what gets deployed, where, and when changes how it interprets your environment, That is production behavior, not housekeeping. Story 5. Copy fail Story 5. Copy Fail is the kind of kernel bug Now to CVE-2026-31431, also known as Copy Fail. NVD shows that this Linux kernel vulnerability is in CISA's Known Exploited Vulnerabilities catalog.

And CISA's entry gives federal agencies a remediation due date of May 15th. The flaw is an incorrect resource transfer between spheres issue in the Linux kernel. And the reporting around it says it enables local privilege escalation to root on a wide range of Linux distributions. AWS's security bulletin says that with the exception of specifically listed services, most AWS customers are not affected.

But it also lists update timelines for affected services such as Bottlerocket, ECS on EC2, EKS optimized AMIs, and EMR. So this is one of those stories where not everybody is affected should not turn into nobody on our side is affected. You still have to inventory what you run. That is the operational lesson here.

When a kernel LPE lands in KEV and there is an active exploitation, this is not the time for vague patch queues. This is the time for concrete exposure review. Which hosts? Which images? Which managed services? Which self -managed nodes? Which maintenance windows? Which compensating controls until patched?

And honestly, the thing I always come back to with stories like this is that kernel issues collapse abstraction layers really fast. You can have immaculate Kubernetes policy, good IAM, strong workload boundaries, and still get wrecked if the shared kernel underneath is vulnerable and unpatched. That is why these bugs matter.

They turn every higher -level control into a best -effort suggestion until the underlying system is fixed. Story 6. Google and AWS are both telling us agents need first -class infrastructure identity now. Last main story. I wanted to put Google Cloud Agent Identity and AWS MCP Server GA together. Because they are different products, but they point in the same direction.

Google Cloud's new IAM post says that the AI era needs a different security and governance model for autonomous agents. And it introduces agent identity as a new first -class principal type, distinct from human identities and generic service accounts.

Google says these identities are built on SPIFFE, are cryptographically protected, and strongly attested, and can be used to authenticate to MCP servers, cloud resources, endpoints, and other agents. It also ties this into agent gateway policy enforcement, least privilege controls, and runtime defense. AWS's side of the story is different in implementation, but similar in intent.

AWS says that the AWS MCP server is now generally available and gives AI agents and coding assistants secure, authenticated access to AWS services through a small fixed set of tools using existing IAM credentials. The official announcement frames it directly around the problem people keep asking. How do you give an agent real AWS access without just handing it keys to the kingdom?

That is why I think these stories belong together. Both clouds are basically admitting the same thing. Agents are not side features anymore. They are becoming infrastructure actors. And once that is true, the identity model has to change too. Not just more service accounts, not just more static tokens, not just vibes and trust. Real principals, real boundaries, real policy, real auditability.

So my read is that this is where cloud identity is headed next. Human identity was phase one. Workload identity was phase two. Agent identity is shaping up to be phase three. And if teams do not start treating that as real infrastructure design now, they're going to rediscover all of the old machine identity mistakes with an LLM in the loop and a much faster failure path. A few quick ones before we wrap.

Cilium published lessons from securing CI/CD for an open source project. And the reason that I like it is because it is practical. On -call brief highlights controls like tighter CI/CD security practices and lessons learned from operating a real open source pipeline. That is worth a read for anyone treating GitHub actions hardening like a someday project.

Velero version 1.18 .1-rc.2 includes a security fix for CVE-2026 -27141 by bumping golang.org/x/net to version 0.51.0. That is the kind of small release note that matters a lot when you realize it closes a known vuln in a tool people trust for recovery. Google's release notes also carry a couple of quieter but real operational changes.

Google Ads and related measurement APIs moved to a 37 -month retention policy for granular performance stats starting June 1st. And Google's distributed cloud's Kubernetes 1.35 platform update now requires cgroups v2 with cgroups v1 no longer supported for creation or upgrades. Those are not flashy changes, but both can absolutely break assumptions if nobody notices them.

AWS also published a cross -region disaster recovery walkthrough for Amazon EKS using AWS Backup. The post walks through creating backup vaults in source and DR regions, running on -demand backups, and initiating cross -region copies. It is not a glamorous story, but it is the kind of thing that people say they care about right up until they realize they have never actually rehearsed it.

I think the human thread underneath this week's episode is that modern reliability work keeps getting squeezed between two kinds The second kind feels newer: AI agents with too much authority, new cloud identity models that quietly become control planes, build and operational systems that act like sidecars until the day they absolutely are not. And what is tricky is that these two categories are not separate anymore.

The old failures now happen inside environments shaped by the newer automation. The newer automation inherits the blast radius of the old infrastructure. And ops teams end up responsible for both at the same time. So I think the real takeaway this week is pretty simple. Reliability is not just about uptime anymore. It is about authority. Who and what can act? What it can touch? How fast it can fail?

And whether your system design assumes mistakes will stay local when they almost never do. That sounds abstract until you line up the stories. A TLD signs bad data and millions of lookups fail. A social platform runs into port exhaustion and logging makes it worse. A GitOps controller hits end of life while version parsing changes in the next branch. A kernel bug drops into active exploitation.

An AI agent sees the wrong token and production disappears. Different systems, same lesson. Small authority problems turn into large reliability problems very quickly. All right, that's it for this week of Ship It Weekly. Quick recap. The Pocket OS and Cursor database wipe and why it is really an identity story. The .de DNSSEC outage and what happens when the trust chain itself goes wrong.

Bluesky's outage post-mortem and how observability can become part of the incident path. Argo CD 3 .1 going end of life while 3 .4 changes behavior. Copy fail and why active kernel exploitation still cuts through all the higher level abstractions. And Google Cloud agent identity plus AWS MCP server GA. Because the cloud providers are starting to formalize agents as real infrastructure actors.

Then in the lightning round, Cilium's CICD security lessons. Velero's CVE fix. Google's retention and cgroups v2 changes. And AWS's cross -region EKS disaster recovery. Links and show notes are on shipitweekly .fm. You can also find the video versions on YouTube. And if you want the source stack before the episode lands, check out this week's on -call brief.

If this episode was useful, follow or subscribe wherever you listen. And send it to the person on your team who still has to explain that reliability problems are not just outages anymore. Sometimes they are identity problems, tooling problems, and authority problems wearing outage clothes. I'm Brian and I'll see you next week.