Ship It Weekly Host Commentaries

This episode is really about one idea: trusted tools become risky when they become invisible.

Most production incidents are not caused by some mysterious system nobody has ever heard of. A lot of the time, the scary part is something familiar. A developer extension. A CI workflow. A cloud provider account. A control plane API. An SDK retry default. A plugin. A collector. A database driver. The thing everyone uses, nobody reviews closely, and everyone assumes is fine because it was fine yesterday.

That is what stood out to me this week.

The GitHub supply chain stories are the cleanest example. The Nx Console VS Code extension compromise was not just “an extension went bad.” It was a reminder that developer tooling sits right next to source code, terminals, tokens, cloud credentials, package publishing paths, and CI/CD systems. StepSecurity reported that Nx Console version 18.95.0 included malicious code targeting developer credentials, cloud infrastructure tokens, and CI/CD secrets. The Hacker News also reported GitHub confirmed internal repositories were exfiltrated after an employee device was compromised through the poisoned extension.

That makes the developer workstation part of the production attack surface.

Not because it serves customer traffic. It usually does not. But because it touches nearly everything that eventually becomes production. Source code. Deploy paths. Secrets. Cloud access. Build systems. Package publishing. Kubeconfigs. SSH keys. Internal docs.

A popular extension with auto-update, broad workspace access, and a trusted brand name is not “just an editor add-on” anymore. It is code running inside a high-trust environment.

That does not mean the answer is “never install extensions.” That is not realistic. Modern engineering depends on tooling. The better answer is to stop treating dev tools as casual personal preference once they can reach production-adjacent systems. Extension allowlists, endpoint monitoring, token hygiene, short-lived credentials, and real review around high-trust tools all matter more than they used to.

The Megalodon story is the CI/CD version of the same thing. StepSecurity reported more than 5,500 public repositories were hit with malware-laden commits and GitHub Actions workflow abuse. That is not just GitHub drama. It is a reminder that CI/CD is where trust becomes artifacts.

A workflow with cloud credentials is not just a test runner. A workflow with signing keys is not just automation. A workflow with package publishing rights is a release system. If that workflow can be modified by a poisoned commit, then the release path is part of the attack surface.

That is the mental model shift I keep coming back to.

Developer tooling is not around production anymore. It is one of the paths into production.

The Railway outage is the architecture version of this. Railway’s incident report said Google Cloud incorrectly suspended its production account, taking out Railway’s API, dashboard, control plane, databases, and GCP-hosted compute infrastructure. Railway also explained that workloads on Railway Metal and AWS initially stayed up, but their edge proxies depended on a GCP-hosted control plane API for routing data. Once route caches expired, workloads outside GCP became unreachable too.

That is the kind of failure that cuts through the comfortable version of multi-cloud.

Multi-cloud on a diagram is not the same thing as multi-cloud resilience.

You can have AWS, GCP, metal, edge proxies, and nice arrows all over the architecture diagram. But if the routing control plane lives behind one provider account, that provider account is still in the hot path. If failover depends on a centralized identity system, that identity system is in the hot path. If emergency deploys depend on the same CI platform that is down, that platform is in the hot path.

The question is not “how many providers do we use?”

The question is “what has to work during failure?”

That is a harder question, but it is the only one that matters.

Discord’s voice outage postmortem is the distributed systems version. I really liked that writeup because it showed the difference between a routine infrastructure change and the system behavior that change produced. Discord described a Kubernetes migration where terminating too many session management pods in one zone dropped about 17 percent of active sessions. That triggered message floods, reconnect behavior, rate limit pressure, memory spikes, gateway issues, and voice/video routing problems.

That is why I like saying real outages are often interaction failures.

Kubernetes did something understandable. Session handoff existed. Rate limits existed. Downstream systems were designed for normal load. But the shape of the change created a workload the system was not ready for.

That is the part “just autoscale it” misses.

Sometimes the bottleneck is not CPU. Sometimes it is mailbox length, fanout, retries, reconnection behavior, a queue, or the helper service that gets buried trying to clean up the mess. Graceful shutdown is not just a pod lifecycle setting. It is a system behavior.

AWS changing SDK retry behavior is the boring version of the same idea. And boring is not an insult here. Boring is usually where the production risk hides.

AWS is updating retry behavior across SDKs and tools, with opt-in available now and defaults changing in November 2026. The changes affect standard and adaptive retry modes, retry quotas, backoff behavior, throttling behavior, and how transient errors are treated.

That sounds like documentation furniture until you remember retries shape how your app behaves during partial failure.

Your app might think it is “calling S3 once.” The SDK may actually be deciding how long to wait, how many times to retry, how much pressure to apply, and when to fail fast. During a service-side problem, that hidden behavior can affect latency, thread pools, connection usage, downstream load, and customer-visible errors.

Retries are invisible infrastructure.

They can protect you from transient failure, and they can also help create a client-side storm during a partial outage. Both are true. That is why this is worth testing before the default changes.

The RabbitMQ AWS plugin bug is the plugin version of trusted-tool risk. AWS published CVE-2026-9133 for an arbitrary file read in the rabbitmq-aws plugin, caused by debug code accidentally shipped in production builds. The plugin can fetch things like TLS certificates, private keys, passwords, and other secrets from AWS services, so a file read bug in that process is not just “some plugin issue.” It is a secrets and blast-radius issue.

Debug code in production should always make people stop blinking for a second.

Debug paths often bypass the clean shape of the system. They inspect directly. They read directly. They validate differently. They exist for convenience during development, and convenience is exactly what you do not want exposed to a real user or attacker.

The Bedrock Reddit story adds the cost angle. It is not a formal incident report, so I would not treat it the same way as an AWS bulletin. But as a pattern, it is very believable: exposed cloud keys plus AI services can become a fast money fire. A compromised credential used to often mean crypto mining, data access, or infrastructure abuse. Now it can also mean model inference, agent workflows, or API calls burning through money at a rate that makes finance start typing in all caps.

That is where security and FinOps are starting to overlap more directly.

If a key can spend money, it is a financial control too.

The lightning round all fits under that same theme.

OpenTelemetry graduating from the CNCF is a huge milestone, but the collector is still production plumbing. Graduation does not mean every collector upgrade is safe or every telemetry pipeline is boring. The thing that observes production can still break production.

The Claude Code RCE story is another reminder that AI coding tools are not just editors. If they have filesystem access, repo context, terminal access, commands, deeplinks, and workflow integration, then they are part of the developer execution environment. That needs a threat model.

GitLab Secrets Manager moving into public beta is interesting because it brings secrets closer to the CI/CD system where a lot of secrets risk actually lives. It does not solve every secrets problem, but it is directionally right. Pipeline credentials should be treated as first-class production risk, not a pile of masked variables everyone hopes are fine.

Google Cloud AI spend caps are useful, but they are also a reliability design question. A hard cap can prevent a surprise bill. It can also pause API traffic if your application depends on that AI service. That means a spend cap is not just a FinOps control. It can become an availability behavior.

The Redshift Python driver RCE is a reminder that clients are part of the execution boundary too. AWS said versions 2.1.13 and earlier could allow a rogue server or man-in-the-middle to execute arbitrary code on the client. That is not “just a driver.” It is code running somewhere important, trusting a remote endpoint.

The common thread is trust.

Modern systems are built out of trusted tools. They have to be. You cannot run everything from scratch, manually inspect every package, manually deploy every change, manually parse every log line, manually route every request, and manually retry every call. That is not engineering. That is punishment with YAML.

But trust needs visibility.

What does the tool have access to?

What can it change?

What happens if it is compromised?

What happens if it disappears?

What happens if it retries differently?

What happens when cached state expires?

What happens if the workflow runs on a poisoned commit?

What happens if the plugin can read files it was never supposed to expose?

That is not paranoia. That is operational hygiene.

The staff and principal engineer job is often about seeing these hidden dependency shapes before they become incident writeups. Noticing when a dev tool is actually a production path. When a retry default is outage behavior. When a multi-cloud architecture still has one hot dependency. When a telemetry collector is availability-sensitive. When a CI workflow is a release system. When a “temporary” credential is now an archaeological artifact with admin rights.

The takeaway is not to stop trusting tools.

The takeaway is to make trust visible.

Map permissions. Review workflows. Scope credentials. Test failure paths. Patch clients. Constrain plugins. Treat CI/CD as a release system. Treat developer workstations as production-adjacent. Treat retry behavior like part of your reliability model. Treat cloud spend controls like they can affect availability.

Because they can.

Trusted tools are not automatically safe.

They are just familiar.

And familiar is exactly why we stop looking closely.

Extra links worth including on the episode page:

GitHub internal repositories breached via malicious Nx Console VS Code extension
https://thehackernews.com/2026/05/github-internal-repositories-breached.html

OpenTelemetry graduates from the CNCF
https://opentelemetry.io/blog/2026/otel-graduates/

Claude Code RCE flaw
https://devops.com/attackers-can-exploit-a-claude-code-rce-flaw-to-take-command-of-system/

GitLab Secrets Manager public beta
https://about.gitlab.com/blog/secrets-manager-in-public-beta/

Google Cloud AI spend caps
https://cloud.google.com/blog/topics/cost-management/introducing-spend-caps-ai-cost-visibility-next26

Redshift Python driver CVE-2026-8838
https://aws.amazon.com/security/security-bulletins/2026-033-aws/

AWS Bedrock cost spike Reddit thread
https://www.reddit.com/r/aws/comments/1tm3ydo/aws_bedrock_cost_spike_14000_usd/

For this Conversations episode, the thing I kept coming back to is that bare metal is not really the story.

The story is that a lot of teams are tired of paying for complexity twice.

First in the bill. Then again in engineering time.

The cloud made infrastructure dramatically easier in a bunch of ways. I do not think anyone who has actually racked servers, waited on hardware, dealt with colo networking, or tried to manage a random pile of machines should pretend the old world was simple. It was not. There is a reason the cloud won. It gave teams APIs, managed services, fast provisioning, easier experimentation, and a way to stop treating every infrastructure change like a procurement project.

But now a lot of teams are far enough into the cloud era that they are seeing the second-order effects.

The bill is one part of it. Bandwidth, managed service pricing, data transfer, always-on environments, overprovisioned Kubernetes clusters, and a bunch of “we’ll clean that up later” infrastructure that somehow becomes permanent.

But the bigger part, at least to me, is the platform complexity tax.

You start with a simple goal. Run the app. Deploy it safely. Scale it. Keep it reliable. Keep cost under control.

Then a few years later, you have Kubernetes clusters, managed databases, object storage, IAM policies, Terraform modules, Helm charts, GitOps controllers, autoscalers, service meshes, CI/CD runners, observability agents, secrets systems, and a platform team trying to make all of that feel like one coherent developer experience.

And sometimes that is the right tradeoff.

But sometimes it becomes this weird situation where the cloud was supposed to make infrastructure easier, and now the team has built a private bureaucracy on top of managed services. Every abstraction has an owner. Every exception has a support path. Every “simple” request turns into a thread with five teams, three Terraform repos, and someone saying, “I think this is handled by the platform.”

That is why I liked this conversation with Jake.

He is not making the lazy “cloud is dead” argument. That argument is boring, and usually wrong. Cloud is not going anywhere. Kubernetes is not going anywhere. Managed services are not going anywhere. For a lot of teams, they are still the correct answer.

But the more useful question is: what are you actually optimizing for?

If the answer is speed of experimentation, global managed services, low operational ownership, and you have the budget for it, the cloud might be the right answer.

If the answer is deep customization, unusual workloads, custom kernel requirements, or very specific infrastructure control, Kubernetes might still be the right answer.

But if the answer is predictable cost, performance density, data sovereignty, simpler primitives, and giving developers a sane path to deploy without building a giant platform engineering machine around it, then private cloud and bare metal start getting a lot more interesting.

Not because bare metal is nostalgic.

Not because racking servers was secretly fun.

It was fun for about four hours. Then it became inventory, cabling, firmware, network weirdness, drive failures, and “who has the crash cart?”

What is interesting now is whether teams can get some of the economic and ownership benefits of bare metal without taking back all of the old operational pain.

That is where platforms like Cycle fit into the conversation. Not as “everyone should use this,” but as an example of a broader shift: teams want the infrastructure underneath them to be simpler, more predictable, and less dependent on a massive pile of glue code.

Jake’s point about opinionated platforms also stood out to me.

As engineers, we love optionality. We like knowing we can customize the thing. We like having access to every dial, every plugin, every escape hatch, every config field, every weird little setting that might matter someday.

But optionality has a cost.

Every knob is a decision. Every decision becomes tribal knowledge. Every deviation becomes something the platform team has to support. And eventually, the platform becomes less of a paved road and more of a choose-your-own-adventure book where half the endings page the on-call engineer.

That does not mean opinionated platforms are always better. Sometimes opinionated tools box you in. Sometimes they hide too much. Sometimes they are great until you hit the edge of the product and suddenly the workaround is worse than the original problem.

But there is a real argument for boring primitives and good defaults.

Especially for teams that mostly need to run containers, maybe some VMs, expose services, attach storage, manage networking, and deploy applications without turning every developer into a part-time infrastructure engineer.

The other part I thought was interesting was the gravity of cloud ecosystems. Even when teams want to move away from a hyperscaler, they may still need to stay near it. Jake mentioned companies wanting bare metal near us-east-1 because the services they depend on are still there. That feels very real. Infrastructure decisions are rarely clean. You can move the workload, but you may not move all the dependencies, all the latency requirements, all the third-party services, or all the operational habits built around the old platform.

That is the part that usually gets missed in the “should we leave the cloud?” conversation.

It is not just, can we run this somewhere else?

It is, what does this thing depend on?

What data does it need?

What services does it call?

What latency does it assume?

What operational model does the team already understand?

What managed services are we actually using, and which ones are just convenience glue we forgot became critical?

That is where the real work is.

So my takeaway from this episode is not “move back to bare metal.”

It is more like: periodically re-check your assumptions.

The cloud decision you made five years ago might still be right. Or it might be right for some workloads and completely wrong for others. Kubernetes might still be the right foundation. Or it might be an expensive control plane for apps that only needed a much simpler runtime. Managed services might be saving your team. Or they might be quietly locking you into cost and operational patterns nobody has revisited in years.

Infrastructure choices age.

Team size changes. Compliance changes. Cost pressure changes. Latency requirements change. The talent on the team changes. The business changes.

And when that happens, “this is how we’ve always done it” is not an architecture strategy. It is just drift with better branding.

That is why I think this private cloud and bare metal conversation is coming back. Not because the industry wants to rewind the clock, but because teams are trying to find a better balance between control and convenience.

More ownership, without becoming hardware janitors.

Better cost predictability, without building everything from scratch.

More performance, without turning the platform into a science project.

More developer self-service, without pretending every team wants to become DevOps experts.

That is the useful middle ground.

And honestly, that is where a lot of the best infrastructure conversations live. Not in declaring one model dead and another one the future, but in being honest about the tradeoffs, the cost, the people, and the operational reality after the architecture diagram becomes production.

This episode is really about one idea: automation does not remove the boring work. It makes the boring work matter more.

That sounds backwards, because most automation is sold as a way to avoid the annoying parts. Less clicking. Less digging through logs. Less manual triage. Less “who owns this?” Less staring at a dashboard trying to remember which service writes to which topic, which database, in which region, for which customer path.

And honestly, I want that too.

Nobody gets into platform or SRE work because they want to spend their best years spelunking through CloudTrail, Kubernetes events, CI logs, and one Confluence page last updated by someone who left in 2021.

But the stories this week all point to the same uncomfortable thing.

The more powerful the automation gets, the more expensive your old mess becomes.

The CISA contractor GitHub leak is the blunt version. GitGuardian said it found a public repository called Private-CISA with 844 MB of exposed material, including plaintext passwords, AWS tokens, and Entra ID SAML certificates. KrebsOnSecurity also reported that the repo exposed credentials for AWS GovCloud accounts and files showing how CISA builds, tests, and deploys software internally. (blog.gitguardian.com)

That is not just a “whoops, rotate the key” story.

That is context exposure.

A leaked credential is bad. A leaked credential plus Terraform, Kubernetes manifests, Argo CD files, CI/CD logs, internal deployment docs, and GitHub Actions workflows is worse. At that point, you may have leaked not just the key, but a pretty good map of how the system works.

That distinction matters.

A lot of teams treat secrets as the only scary artifact. They run secret scanning, rotate tokens, and move on. But attackers do not only care about credentials. They care about shape. Naming conventions. Deployment paths. Control planes. Environments. Build steps. Internal assumptions. Which systems trust which other systems. Which scripts look abandoned but still work.

The floor plan matters.

And that is where the staff/principal engineer alarm bell should go off. Not because every leak is catastrophic in the same way, but because operational context is part of your attack surface.

Old repos, contractor-owned repos, personal forks, demo projects, migration backups, Terraform state, kubeconfigs, CI logs, and zip files named something like prod-final-backup-really-final are not harmless just because they are boring. Boring is where production risk hides, mostly because boring things stop getting reviewed.

The AWS DevOps Agent story is almost the opposite side of the same coin. Instead of leaking operational context, AWS is showing an agent trying to gather it during an incident. Their post walks through automated RCA across Datadog and Elasticsearch, with EKS access for Kubernetes objects, pod logs, and cluster events, plus CloudTrail deployment context. (Amazon Web Services, Inc.)

That is useful. I can absolutely see the value.

A lot of incident response is context reconstruction. What changed? What deployed? Which pod restarted? What metric moved first? What log line started showing up? Which dependency decided to become a learning opportunity at 2:13 PM on a Tuesday?

If an agent can assemble that timeline faster, great.

But automated RCA is one of those places where the output can sound more certain than it deserves to be. A clean summary with “probable root cause” in bold can become the thing everyone believes, especially when the channel is noisy and everyone is tired.

So the question is not “should we use AI for incident response?”

The better question is: where does the agent sit in the decision chain?

Is it a scribe?

An investigator?

A summarizer?

A hypothesis generator?

Or is it becoming the person in the room everyone quietly defers to because it sounds confident and nobody wants to keep digging?

That boundary matters.

The same thing shows up in Microsoft Copilot Studio computer-using agents. Microsoft says computer use in Copilot Studio is generally available, and its docs describe agents interacting with websites and desktop apps through graphical user interfaces. (TECHCOMMUNITY.MICROSOFT.COM)

That sounds amazing if you live in the real enterprise world, where half the important systems either have bad APIs, no APIs, or APIs technically exist but somehow the only supported process is still “log into the portal and click the thing.”

Computer-using agents are going after that mess.

But they also make the boundary fuzzy.

API automation at least gives you endpoints, scopes, schemas, logs, and a reasonably clear mental model. UI automation is more like, “the agent looked at the screen and clicked what seemed right.”

That may be fine when the button is “Download report.”

It is a little less fine when the button is “Approve,” “Delete,” “Submit payment,” or “Yes, I understand this is permanent.”

Again, the tool is not automatically bad. The failure mode is lazy governance. If an agent can use a UI, then the UI is now an automation interface. That means restricted accounts, audit logs, test environments, approval gates, and very strong feelings about bulk updates.

Atlassian adding Claude Code support to Bitbucket Agentic Pipelines is another version of this. Atlassian says Agentic Pipelines lets teams embed AI agents into Bitbucket Pipelines steps to analyze code, troubleshoot failing pipelines, fix flaky tests, and more. Atlassian also has separate guidance about third-party agent providers, including Claude, and what that means for permissions and data handling. (Atlassian Support)

That is the part I keep coming back to.

CI/CD is not “developer tooling around production.”

CI/CD is the path code takes to become production.

So when agents enter CI/CD, they are not just helping with chores. They are entering the delivery path. That means the boring questions matter immediately.

What code does the agent see?

What logs does it see?

What secrets are available?

Can it modify tests?

Can it open pull requests?

Can it generate security triage notes that people treat as fact?

Can it make the pipeline pass without making the system better?

That last one is not theoretical. Humans do it constantly. We just call it temporary, put it in a PR description, and then let it survive three reorgs.

The Kubernetes seccomp story is the grounding wire for all of this.

After all the agent talk, CVE-2026-46333 is a reminder that your old defaults still matter. Kubernetes seccomp docs describe how seccompDefault can apply the RuntimeDefault profile when no profile is specified, while otherwise workloads may run unconfined depending on configuration. (Microsoft Learn)

That is not flashy. It will not win a keynote. Nobody is making a cinematic launch video for “check your pod security defaults.”

But those are the kinds of settings that decide whether a theoretical exploit path becomes a practical one.

The boring defaults are not boring. They are latent decisions.

And every once in a while, a CVE shows up and asks what you decided.

That is also why the lightning round fits the episode.

GitHub expanding OIDC support for Dependabot and code scanning is not flashy, but short-lived identity-based access is healthier than long-lived registry secrets sitting around forever. Java pods getting OOMKilled even when heap looks fine is a reminder that abstractions leak, and Kubernetes does not care that your -Xmx looked reasonable. LLM-generated SQL that returns plausible but wrong results is a reminder that failure is not always loud.

Sometimes the system breaks quietly.

Sometimes the dashboard loads.

Sometimes the query runs.

Sometimes the postmortem gets published.

Sometimes the action item says “improve monitoring,” and everyone nods like that is a plan.

That is why the human closer matters.

Postmortem action items die because they are often not real work yet. They are good intentions with vague verbs. “Improve monitoring.” “Review runbooks.” “Clean up ownership.” “Investigate retries.”

Those are not action items.

They are vibes in ticket form.

A real action item has an owner, a clear outcome, a tracking location, and a due date. incident.io’s piece on failed postmortem actions points at the same basic reasons: no named owner, vague wording, wrong tracking place, and no follow-up cadence. (Atlassian Support)

And that is the part that ties the whole episode together.

The CISA leak is not fixed by saying “review GitHub practices.”

AI RCA is not useful if the follow-up is “improve incident response.”

Computer-using agents are not governed because someone wrote “ensure controls.”

Claude Code in CI/CD is not safe because someone said “be careful with third-party providers.”

Kubernetes seccomp is not handled because someone said “harden workloads.”

At some point, someone has to turn the vague thing into real work.

Name the owner.

Find the repo.

Rotate the token.

Delete the archive.

Scope the account.

Document the data flow.

Apply the default.

Track the exception.

Close the loop.

That is not the glamorous part of engineering, but it is the part that compounds.

The staff and principal engineer job is often less about having the cleverest take and more about turning fuzzy risk into specific work that actually changes the system.

Automation is going to keep getting more powerful. Agents will get better. RCA tools will get faster. Pipelines will get more intelligent. UI automation will keep reaching into systems that never had proper APIs.

Fine.

But if the ownership model is messy, the secrets are stale, the defaults are unknown, the CI permissions are broad, and the postmortem actions are vague, then automation does not save you.

It scales the mess.

That is the lesson I keep taking from these stories.

Production does not run on good intentions.

It runs on the stuff someone actually fixed.

Additional links worth including somewhere if you have room: KrebsOnSecurity’s CISA leak coverage, Microsoft’s computer-use docs, Atlassian’s third-party agent provider guidance, Kubernetes seccomp docs, GitHub’s Dependabot/code scanning OIDC changelog, Readyset’s LLM SQL piece, and incident.io’s postmortem follow-up article. (Krebs on Security)

This episode is really about one idea: authority is the new blast radius.

For the last couple years, most of the AI conversation in engineering has been about productivity. Can it write code faster? Can it explain logs? Can it summarize an incident? Can it help junior engineers get unstuck? Can it save senior engineers from staring at the same YAML for the 900th time?

All of that still matters. But this week’s stories point at something bigger.

AI agents are not just getting smarter. They are getting places to run, APIs to call, identities to assume, and automation systems to trigger.

That changes the conversation.

A coding assistant that suggests a function is one thing. A cloud agent that can be started through an API, work in its own environment, make changes, validate them, and open a pull request is a different thing. At that point, the agent is not just helping a developer type. It is becoming part of the software delivery path.

That does not mean the sky is falling. It does mean the mental model has to change.

GitHub Copilot cloud agent tasks through the REST API are interesting because APIs are how tools become platforms. As soon as something can be started programmatically, other systems will start wiring into it. A ticket can start work. A vulnerability finding can start work. A dependency update can start work. A support escalation can start work. That is useful, but it also means the agent becomes another automation actor inside your engineering system.

And once something becomes an actor, you have to care about authority.

What repository can it touch? What branch can it write to? What token does it use? Can it trigger CI? Can that CI deploy? Can it comment on issues? Can it open a PR against production code? Can it modify tests to make its own change look correct? Who reviews the result? Who owns it if the change breaks something?

That is the part that is easy to skip because the demo feels productive. But production incidents do not care how impressive the demo was.

The Auth0 MCP story is the identity version of the same problem. MCP is quickly becoming one of the connective layers between AI agents and real tools. That means MCP servers cannot just be treated like fun local adapters. If an MCP server can reach customer data, cloud APIs, internal systems, source code, CI/CD, or production operations, then it needs to be treated like a production API.

That means authentication. Authorization. Logging. Revocation. Delegation. Auditability.

The weird part is that agent identity is not as clean as normal user identity. With a normal user, we can say Brian logged in, Brian clicked the button, Brian had these permissions. With an agent, the action might be requested by a human, executed by an application, delegated through a token, and carried out by a model calling a tool.

That is not impossible to manage, but it is different enough that lazy answers will hurt people.

The Red Hat and Ansible story makes this even more concrete. Ansible is not a toy. It is not just a dev environment helper. It is a real automation platform that teams use to patch servers, restart services, configure systems, manage infrastructure, and run operational workflows. When AI agents start connecting to something like Ansible, the agent is suddenly much closer to the machinery that changes production.

That might actually be the right direction. I would much rather see agents routed through governed automation than freehanding shell commands on production systems because they read three stale wiki pages and felt confident.

But that only works if the automation itself is worth trusting.

A messy playbook does not become safe because an AI agent invoked it. A broad inventory does not become scoped because a model called it. A dangerous script does not become governed because it has a nicer interface. In some cases, AI may just make old operational debt easier to trigger.

That is the risk.

Not that agents exist.

The risk is that agents expose every sloppy permission, every overpowered workflow, every unsafe runbook, every “only Bob knows how to run this” script, and every service account that was supposed to be temporary three years ago.

OpenAI Daybreak fits into this from the security side. AI-assisted vulnerability discovery, patch generation, and remediation validation are going to be useful. I do not think that part is controversial. Security teams are already drowning in findings, and anything that helps triage, validate, patch, and verify could be a real improvement.

But it also changes the bottleneck.

If AI finds more issues, the hard part becomes absorbing the output. Can the organization validate the findings? Can it prioritize them? Can it find the owner? Can it patch safely? Can it ship quickly? Can it prove the fix worked? Can it do all of that without breaking production in the process?

Security does not end when the issue is found. For a lot of companies, that is where the real pain starts.

That is why Daybreak is not just a security story to me. It is an engineering systems story. If your delivery process is slow, brittle, under-tested, or full of unclear ownership, AI-generated security findings may not make you safer right away. They may just make the backlog more honest and more painful.

We also mentioned our special on Project Glasswing / Claude Mythos:

Episode 33Apr 15, 2026⏱️ 16:28Special: Claude Mythos Preview and Project Glasswing: AI Exploit Discovery, Zero-Day Risk, Business Fallout, and What It Means for DevOps, Cloud, and Platform SecurityEpisode: Special: Claude Mythos Preview and Project Glasswing: AI Exploit Discovery, Zero-Day Risk, Business Fallout, and What It Means for DevOps, Cloud, and Platform Security

The Discord ScyllaDB automation story is the useful counterweight to all of this. That is the kind of automation we should be aiming for before we get too excited about agents doing operational work.

Their story is not magic. It is not “AI fixed databases.” It is a team looking at fragile scripts and turning them into a more reliable control plane with state, preconditions, resumability, notifications, and safer workflows.

That is the boring work that actually matters.

A lot of teams say they have automation, but what they really have is a pile of scripts that work when the right person runs them on the right day in the right order with the right assumptions in their head. That is better than nothing, but it is not the same as safe operational automation.

Safe automation knows when to stop. It checks assumptions. It notices when the cluster is degraded. It does not blindly plow forward because the script got to line 47. It gives humans visibility. It reduces babysitting. It makes the system more predictable instead of just making the command shorter.

That matters even more in an agentic world.

If agents are going to call tools, the tools need to be boring, constrained, observable, and designed around failure. Otherwise we are not building reliable operations. We are just giving a very confident system a faster way to trip over our old mistakes.

The lightning stories all point back to the same general theme.

GuardDuty and crypto mining are a reminder that cloud abuse often shows up as cost before it shows up as drama. A compromised credential might not immediately become a headline breach. It might become a weird bill, degraded performance, or a mining workload hiding in an account nobody checks closely enough.

Queues and backpressure are the reliability version. A queue can smooth bursts, but it cannot magically absorb sustained overload forever. It just stores the problem somewhere else until message age, lag, retries, or downstream failure finally make the truth obvious.

And the Datadog index scan story is a nice reminder that labels can lie to your intuition. “Using an index” sounds good until the query is still expensive. The plan can be technically correct and still operationally painful. That is true for databases, and honestly, it is true for a lot of AI and automation too.

The label is not enough.

“Agentic” is not enough.

“Authenticated” is not enough.

“Automated” is not enough.

“Uses an index” is not enough.

The details matter.

What is it allowed to do? What path does it take? What assumptions does it make? What happens when those assumptions are wrong? Who gets alerted? Who can stop it? Who owns the outcome?

That is where I think a lot of engineering teams need to focus.

Not on whether AI agents are good or bad. That debate is already too broad to be useful. The better question is where they sit in the system and how much authority they have.

An AI agent with read-only access to logs is one kind of risk.

An AI agent that can open pull requests is another.

An AI agent that can trigger CI/CD is another.

An AI agent that can call MCP servers attached to internal tools is another.

An AI agent that can invoke Ansible against production systems is another.

Those are not the same thing, and we should stop talking about them as if they are.

The more authority an agent has, the more it needs to look like a real production principal. Scoped access. Clear ownership. Good audit logs. Human approval at the right boundaries. Dry-run modes. Kill switches. Reviewable output. Strong defaults. No mystery tokens hiding in a demo server someone forgot about.

None of that is anti-AI. It is just operations.

The funny thing is, AI may end up forcing teams to clean up the parts of their systems they should have cleaned up anyway. Bad runbooks. Overpowered service accounts. Weak CI permissions. Unowned scripts. Unclear release paths. Missing rollback plans. Poor observability around internal automation.

Those were already risks.

Agents just make them harder to ignore.

So the takeaway from this episode is not “do not use agents.”

The takeaway is to label them correctly.

An agent with repo access is part of your software delivery system.

An MCP server with production reach is part of your control plane.

An automation workflow that changes systems is production infrastructure.

A security tool that generates patches is part of your remediation process.

A queue hiding overload is not resilience.

An index scan is not automatically fast.

And an AI-generated change is still owned by the humans and systems that allowed it to ship.

Authority is the new blast radius.

The teams that handle this well will not be the ones that block everything. They will be the ones that give agents useful jobs, narrow permissions, clear boundaries, and safe paths to action.

The teams that handle it poorly will accidentally build a coworker with API access, hand it a badge, and then act surprised when it finds the side door to production.

What stood out to me this week is that the failure modes were all over the stack, but they kept pointing back to the same thing: authority.

The PocketOS and Cursor story is the obvious example. It is easy to frame that one as “AI went rogue,” but that’s not really the useful lesson. The useful lesson is that an agent got access to a token it should not have had, and once it had that authority, the rest happened fast. On the other end of the spectrum, the .de outage was not AI at all. It was classic Internet plumbing: bad DNSSEC signatures at the TLD level, validating resolvers doing exactly what they were supposed to do, and millions of domains effectively disappearing behind SERVFAIL. Different systems, same theme. Give the wrong thing too much trust, or centralize trust in the wrong place, and the blast radius gets big fast. (Teller's Tech)

That’s also why I liked the Bluesky postmortem so much. It is the kind of outage write-up operators actually learn from because it is not clean or elegant. They were exhausting ports, but the debugging path and the logging behavior helped amplify the pain. That is a very real production pattern. The first problem hurts, then the systems you rely on to reason about it start adding load, noise, or contention of their own. A lot of outages are not one bad component failing in isolation. They are a cluster of small, understandable behaviors that turn pathological together. (Pckt)

Argo CD and the kernel bug were the quieter stories, but maybe the more familiar ones for day-to-day operators. Argo CD 3.1 hitting end of life while 3.4 changes Kubernetes version interpretation is exactly the kind of thing teams wave off until a controller upgrade lands and selection logic stops behaving the way people assumed. CVE-2026-31431 is the same kind of reminder from a different angle. Kernel bugs do not care how nice your abstractions are. If the shared base layer is vulnerable and actively exploited, your higher-level controls stop feeling very absolute. That’s why the boring work still matters: controller version hygiene, image inventory, maintenance windows, patch review, and all the stuff nobody wants to talk about when there is a shinier story on the page. (GitHub)

The other piece I kept coming back to is that the clouds are starting to admit agents are no longer a novelty feature hanging off the side of existing IAM. Google is introducing Agent Identity as a first-class principal type built on SPIFFE, and AWS is pushing MCP access as something that should be secure, authenticated, and bounded through a fixed tool surface. That is a pretty big signal. We are watching cloud identity move from human identity, to workload identity, to agent identity. And if that sounds abstract, it really is not. It just means teams are about to rediscover every old machine-identity mistake they already made once, except now the actor on the other end can move faster and make stranger decisions. (Google Cloud)

So my takeaway from this episode is simple. Reliability is still about uptime, latency, and recovery, sure. But more and more, it is also about who or what is allowed to act, what it can touch, and whether your environment assumes a mistake will stay local when it probably will not. That applies to DNS trust chains, GitOps controllers, kernel exposure, backup design, and AI agents with credentials. Different layers, same question: where does the authority actually live, and how much damage can it do before something stops it? (Teller's Tech)