EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

This episode is about safety nets.

That sounds comforting, and honestly, it should. Rollback buttons are good. Better alarms are good. Zone-aware routing is good. Secret scanning with more context is good. Internal advisory workflows are good. Better agent guardrails are good. I do not want fewer safety nets in production systems. I want more of them.

But safety nets create a weird kind of risk too. They can make teams feel like the hard part is solved. And usually, it is not.

A rollback feature does not mean the rollback is safe. An alarm does not mean the signal is worth waking someone up. A scanner finding a secret does not mean rotation is easy. An AI coding agent finding a fix does not mean it should run commands. And zone-aware routing does not mean you understand your failure domains.

That is really the thread through this episode. The tools are getting better. The recovery paths are getting better. The visibility is getting better. But the systems underneath are still complicated, still connected, and still full of places where one bad assumption can turn a small problem into a production incident.

Start with EKS version rollbacks.

This is a genuinely useful feature. For a long time, Kubernetes upgrades have felt like a one-way door. You check your cluster version. You check release notes. You check deprecated APIs. You check add-ons. You check admission controllers. You check node versions. You check whatever chart everyone is scared to touch. Then eventually, you upgrade.

And if something breaks, the path back is not always clean.

So EKS adding version rollback is a big deal. It gives platform teams a recovery option when a Kubernetes minor version upgrade exposes some compatibility issue they did not catch ahead of time. That helps, especially if you run a lot of clusters. Especially if you have strict change windows. Especially if every Kubernetes upgrade already involves a calendar invite, a Jira ticket, three approvals, and someone saying “should be fine” in Slack.

But rollback does not erase the work. It changes the risk model. That is the important distinction.

The existence of rollback does not mean workloads are compatible. It does not mean add-ons are safe. It does not mean admission webhooks behave correctly. It does not mean node groups are ready. It does not mean disruption budgets are sane. It does not mean the rollback itself will be instant. And it definitely does not mean the cluster goes back in time like nothing happened.

Rollback is an operation. It has requirements. It has timing. It has dependencies. It can fail in its own special way.

That is why this feature belongs in the upgrade runbook, not in the “we’ll figure it out if it breaks” bucket. The better question is not “can we rollback?” The better question is “have we tested the rollback path?”

What happens to nodes? What happens to add-ons? What happens to workloads? What happens with EKS Auto Mode? What happens if disruption budgets slow the rollback down? What happens if the thing that broke during upgrade also affects the path back?

A rollback button is great. But if nobody owns the upgrade path, the rollback button can become false confidence. That is the thing to watch.

Cordyceps hits the same theme from a different angle.

This one is about GitHub Actions and CI/CD supply chain risk. And the reason it matters is not just “workflows can be vulnerable.” We already know that. The reason it matters is that a lot of teams still treat workflow YAML like it is not real code.

It is in .github/workflows, so it feels like config. It gets copied from a blog post. It gets borrowed from another repo. It gets tweaked until the build is green. And then it sits there for years with production authority.

That is the problem.

GitHub Actions workflows can run commands. They can read secrets. They can publish packages. They can assume cloud roles. They can sign artifacts. They can push containers. They can cut releases.

That is not harmless automation. That is privileged execution.

And the nasty part with CI/CD attacks is that the dangerous path is not always obvious. It might not be one line that screams “this is bad.” It might be a low-trust workflow producing an artifact, then a higher-trust workflow consuming it later. It might be a branch name landing in shell. It might be a pull request title becoming command input. It might be a cache crossing a privilege boundary. It might be a comment, issue body, label, tag, or generated file moving from untrusted context into trusted automation.

Each step looks normal enough on its own. Together, you accidentally built an attack chain.

That is why pipeline security is hard. The boundary is not always one file. It is the relationship between triggers, permissions, artifacts, caches, tokens, branches, environments, and who is allowed to influence each of those pieces.

And now AI makes this more interesting.

Because if AI coding agents are generating CI/CD configuration, they can reproduce bad patterns quickly. Not because the agent is evil. Because insecure workflow patterns are everywhere. If the training data, examples, old repos, or copied snippets include bad assumptions, the agent may hand you a pipeline that works beautifully and trusts the wrong thing.

That is a rough combination.

Fast generation. High privilege. Weak review. Production authority.

So the fix is not “stop using GitHub Actions.” That is not realistic, and it is not the point. The fix is to treat workflow logic like production code.

Review it. Threat model it. Limit permissions. Separate trusted and untrusted workflows. Be careful with pull_request_target. Watch shell interpolation. Be suspicious of artifacts and caches crossing privilege boundaries. Assume your CI/CD system is part of your supply chain, because it is.

And attackers already treat it that way.

Then agentjacking takes the trust boundary even closer to the day-to-day work.

This story is about fake telemetry influencing AI coding agents. And I think this one matters because it changes how we think about logs, errors, and incident context.

For years, telemetry has mostly been evidence. An error happens. A log line appears. A stack trace gets captured. A Sentry issue gets opened. A human reads it and decides what to do.

The text might be noisy. It might be wrong. It might include user input. But a human is still sitting between the evidence and the action.

AI agents change that.

Now the same text may be read by something that can act. It might inspect a repo. It might suggest a patch. It might run tests. It might install packages. It might execute commands. It might open a pull request. It might touch credentials sitting in the local environment.

That changes the boundary.

The question is no longer just “can an attacker compromise the server?” The question becomes “can an attacker put text somewhere the agent will trust?”

That is a much lower bar.

And it is uncomfortable because our systems were not always designed with that in mind. We sanitize logs for display. We redact secrets. We limit access to observability tools. But we have not always treated error text as something that could steer an automated actor.

Now we have to.

If an agent can read hostile context and then take action, that context is part of the security model. Logs are not just logs. Issue text is not just issue text. Stack traces are not just stack traces. Runbooks are not just documentation. They can become instruction material.

So the control cannot just be “the agent is smart.” That is not a control.

The control is authority.

What can the agent read? What can it run? What can it write? Can it inspect without executing? Can it propose without applying? Can it separate trusted remediation guidance from user-controlled error content? Can you audit what it saw? Can you audit what it did? Are there live credentials nearby when it makes a bad decision?

That is the platform question.

Not “does this make developers faster?” That is a product question.

The platform question is “what happens when it is wrong, and what can it touch while being wrong?”

That is where agentjacking lands for me. It is not just an AI story. It is a trust boundary story. The text around your systems can now cause action. So the trust model has to include the text.

CloudWatch log query alarms are a much more normal operations story, but they fit the theme too.

This is a useful feature. A lot of real incidents show up in logs first. A specific exception. A repeated auth failure. A dependency timeout. A database connection error. A queue consumer screaming before the metric catches up.

For years, turning that into an alarm meant extra glue. Metric filters. Custom metrics. Another vendor. A Lambda path. A weird pipeline where a log pattern becomes a metric and the metric becomes an alarm.

Now CloudWatch can create alarms from log queries directly. That is good. It shortens the path from signal to action.

But it also shortens the path from random scary log line to pager noise.

And that is where teams can hurt themselves. Because the hard part of alerting is not creating the alert. The hard part is deciding whether the alert should exist.

Who owns it? What does it mean? How often does it happen? Is it user-impacting? What threshold makes sense? What does the on-call do when it fires? Does it flap? Does the log format change every deploy? Is it a symptom, or just a noisy internal detail?

If nobody can answer those questions, you did not create an alert. You created a search shortcut with a pager attached.

And every on-call team has lived through some version of that.

Someone sees an error. Someone says “we should alert on this.” Nobody wants to be the person arguing against “better monitoring.” So the alert ships. Then three weeks later, it is waking people up for a condition nobody understands and nobody owns.

That is how observability becomes punishment.

So yes, use log query alarms. They are useful. But use them for known failure modes. Clear owner. Clear threshold. Clear action. Clear reason to interrupt a human.

That is the difference between faster signal and faster noise.

The lightning round keeps adding to the same shape.

ECS Service Connect zone-aware routing is a good example. On the surface, it sounds like a networking improvement. Prefer same-AZ traffic when possible. Reduce latency. Reduce cross-AZ cost. Improve behavior during partial failures.

That all sounds great.

But it also reminds us that service-to-service routing is never just “can service A reach service B?” It is where traffic goes. How much it costs. What happens when an Availability Zone gets weird. What fails open. What fails closed. What happens when an old service needs a redeploy to pick up the new behavior.

Networking defaults matter because defaults become architecture.

etcd 3.7 is another quiet but important one.

Most people do not wake up excited to talk about etcd. Fair. But Kubernetes reliability still depends on the boring distributed database underneath it.

RangeStream, performance improvements, cleanup of older v2 store remnants, protobuf updates, dependency updates. None of that is flashy. But control plane reliability is built out of boring things working correctly under pressure.

The API server, scheduler, controller manager, kubelet, CNI, CSI, admission webhooks, and etcd all have to hold together. So when etcd gets better, Kubernetes gets a little more room to breathe.

GitHub innersource advisories are also easy to underplay.

Public CVEs get attention. Internal dependencies often do not. But a lot of companies run on private packages, internal platform libraries, shared GitHub Actions, private modules, and internal service templates.

Those things need security workflows too. Disclosure. Affected versions. Dependabot notifications. Remediation paths.

A vulnerable internal package can still be a real supply chain problem, even if it never touches the public internet.

Secret scanning metadata is similar.

Finding the secret is not the whole incident. It is just the beginning.

Is it real? Is it live? Who owns it? What system uses it? Is it expired? Can it be revoked safely? Will rotating it break production? Does it need a workspace URL, endpoint, project ID, or other context to validate?

Better metadata matters because incident response is full of these small, painful questions.

The scanner saying “I found something” is useful. The scanner helping you figure out what it is, where it belongs, and whether it is still active is much more useful.

And CloudWatch Application Signals service events hit the same operational theme.

When something is slow, you do not only need to know that it is slow. You need to know what changed.

Was there a deploy? Was there an exception spike? Was there a latency event? Did the problem start after a rollout? Did the shape of traffic change?

Observability gets better when the signal and the context live closer together. But again, only if someone owns the events and knows how to use them.

Otherwise, it is just another wall of things that happened.

That is why the closer lands on ownership.

The episode is not anti-tool. It is the opposite. Use the tools. Use the rollback button. Use log query alarms. Use zone-aware routing. Use better scanner metadata. Use internal advisories. Use AI agents where they help.

But do not confuse better tooling with transferred responsibility.

The system is still yours. The pager is still yours. The blast radius is still yours. The customer impact is still yours. And the judgment is still yours.

So the practical questions this week are pretty simple.

Where are the safety nets? Who owns them? Have they been tested? What do they not cover? What do they make easier? What do they make riskier? What authority do they inherit? What happens when they are wrong? And who is responsible when everyone assumed the tool meant the thinking was done?

That is the work.

Safety nets are good.

But they do not replace ownership.

---

Extra links worth including:

Amazon EKS Kubernetes version rollbacks
https://aws.amazon.com/blogs/aws/upgrade-amazon-eks-clusters-with-confidence-using-kubernetes-version-rollbacks/

Novee Security: Cordyceps supply chain research
https://novee.security/blog/cordyceps/

Tenet Security: Agentjacking through fake Sentry errors
https://tenetsecurity.ai/blog/agentjacking-coding-agents-with-fake-sentry-errors/

Amazon CloudWatch log query alarms
https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-cloudwatch-log-alarms/

ECS Service Connect zone-aware routing
https://aws.amazon.com/about-aws/whats-new/2026/07/ecs-service-connect-zone-aware/

etcd 3.7 announcement
https://etcd.io/blog/2026/announcing-etcd-3.7/

GitHub innersource security advisories
https://github.blog/changelog/2026-07-08-innersource-security-advisories-are-generally-available/

GitHub secret scanning extended metadata and multipart validation
https://github.blog/changelog/2026-07-07-secret-scanning-extended-metadata-and-multipart-validation/

CloudWatch Application Signals service events
https://aws.amazon.com/about-aws/whats-new/2026/06/cloudwatch-service-events/

This week’s On Call Brief
https://learn.tellerstech.com/on-call-brief-news/2026-W28/

More Ship It Weekly episodes
https://shipitweekly.fm/