CodeBreach in AWS CodeBuild, Bazel TLS Certificate Expiry Breaks Builds, Helm Charts Reliability Audit, and New n8n Sandbox Escape RCE

Transcript

This week is a reminder that the stuff we treat

like glue is now a primary failure domain. CI

trigger roles, cert renewals, helm charts, automation

tools. One tiny assumption goes sideways and

suddenly you're dealing with a supply chain risk,

a global build break, or an RCE in the thing

that holds all your credentials. Hey, I'm Brian.

I work in DevOps and SRE and I run Tellers Tech.

This is Ship It Weekly, where I filter the noise

and pull out what actually matters when you're

the one running infrastructure and owning reliability.

If something's hype, I'll call it hype. If it

changes how you operate, we'll talk about it.

Quick bit of housekeeping, the show notes and

links are on shipitweekly .fm. If the show's

been useful, follow it wherever you listen. Also,

a rating helps way more than it should. Four

main stories for today. First, code breach. A

CI trigger and filtering issue in a small set

of AWS managed repos that's a perfect reminder

that CI glue is part of your security boundary

now. Second, the Bazel TLS cert expiry incident.

The kind of failure that is boring, binary, and

absolutely capable of blocking your entire engineering

org. Third, Helm chart reliability. prequel reviewed

over a hundred popular charts and the results

are basically a post -mortem template for why

it installed fine is not a reliability guarantee

fourth n8n two new high severity flaws disclosed

by jfrog that can lead to code execution we're

going to treat this one like a mini story because

workflow automation tools are basically a control

plane holding your secrets then a quick lightning

round with a few operator friendly tools and

takeaways and a human closer tying the theme

together alright let's get into it So code breach.

AWS published a security bulletin describing

a misconfiguration involving unanchored account

underscore ID webhook filters for code build.

used by a small set of AWS -managed open -source

repos. AWS says they mitigated it quickly, rotated

credentials, reviewed logs, and added additional

mitigations and protections around build processes

with credentials in memory. Wiz's research frames

the risk clearly. If you can trigger a privileged

build in a repo that's part of a supply chain,

you potentially get access to tokens and credentials

that can be used to push changes or create malicious

artifacts. That's why it's not just CI, it becomes

supply chain. Now, the operator lesson is not

code build is bad. The operator lesson is stop

treating pipeline trigger logic like it's harmless.

If an untrusted event can cause a trusted pipeline

to run, you do not have CI. You have an execution

environment exposed to the internet. And almost

every org drifts towards this risk without meaning

to. Here's the drift pattern. You start with

PR checks, lint, test, build, great. Then someone

adds integration tests. And those tests need

credentials to hit an environment, or to pull

from a private registry, or to call a third -party

API. Then someone adds preview deployments. Those

need cloud creds, or at least some deploy token.

Then someone adds artifact publishing because

it's easier if the PR build produces the image.

Now your PR pipeline can build and push images.

Then someone uses those images in staging. And

now your PR pipeline is part of your release

path. And at that point, you've accidentally

created a supply chain path where a PR can influence

something that runs in your environment. This

is why the most dangerous sentence in CI is,

it only runs in CI. Because CI is usually the

thing that holds credentials that can touch everything

else. So the practical question to ask is simple.

Can untrusted events cause trusted actions? Can

a forked PR run a job that has secrets available?

Can a PR comment trigger a workflow that can

deploy? Can a PR workflow push artifacts that

are later deployed? Can a PR workflow assume

cloud roles? If the answer is yes, or even I'm

not sure, you've got a boundary problem. Now,

I want to give you a do this Monday playbook

that doesn't require a big platform rewrite.

Step one, label pipelines mentally as untrusted

and trusted. Untrusted pipelines is PRs and forks.

No secrets, no publish, no deploy. It answers,

does it compile? Do tests pass? Trusted pipelines

run on merges to main, tags, or explicit manual

approvals. That's the one allowed to publish

and deploy. Step two. Pick one repo, your most

sensitive one, and map the event chain in plain

English. What events trigger workflows? Which

workflows can access secrets? Which workflows

can write artifacts? Which workflows can deploy

or change infra? Step 3. Check for the classic

foot guns. Are secrets injected into your PR

workflows at all? Are you using conditions like

actor checks or branch checks as auth? Are you

relying on the code won't print secrets as a

control? Do you allow PR builds from forks to

run with privilege tokens? If you find these,

the fix is not perfect security. The fix is strong

separation. PR builds can still run. just keep

them in the untrusted sandbox. If you need integration

tests, run them in a separate environment with

a separate low privilege credential set. If you

need to build images, build them but don't push

them to a production registry. Or push to an

isolated registry that cannot be used for deploy.

If you need a preview environment, require approval

before anything privileged runs. And the last

point, even if you fix the triggers, scope matters.

Least privilege for CI tokens is not optional.

Your pipeline credentials should not be able

to do everything. That's story one. Story two

is Bazel. On December 26th, 2025, the TLS certificates

for many star .bazel .build domains expired and

it caused widespread build breakage. The Bazel

team's postmortem says the outage lasted around

13 hours before it was resolved. This is one

of those incidents that's both boring and terrifying.

Boring because it's just a cert. Terrifying because

cert failures are a binary cliff. Everything

works, then it doesn't. And the blast radius

is immediate because every client that depends

on that endpoint fails at the same time. Also,

auto -renew does not prevent this class of incident.

Auto -renew is one link in a chain. The full

chain is issuance, renewal, deployment, Reload

and verification. A lot of real cert outages

are renewal succeeded but deployment didn't reload.

Or new hostname wasn't included. Or monitoring

checked the wrong endpoint. Or DNS validation

broke and nobody noticed. So here's the practical

operator version of this story. You'd need external

monitoring for cert expiry from the outside.

against the actual endpoint users hit. Not an

internal health check. Not a dashboard in the

cert system. The real endpoint. You need ownership.

A named owner. A team. A channel. Someone that

at 2am on call can tag and get traction. And

you need runway. Alerts well before expiry. If

your alert fires 24 hours before expiry, you

are still basically doing incident response.

If it fires 30 days before expiry, you can fix

weird edge cases like DNS changes, migrations,

or validation issues calmly. Now, the do this

Monday pass. Pick your top three engineering

org blockers. These are not always customer facing.

Often they're internal systems that block shipping.

Artifact Registry, Git Host, CI Endpoint, SSO

Login, Webhook Receiver, Package Download Host,

any of those. For each one, answer. Do we have

external monitoring for cert expiry and chain

validity? Does it alert at least 14 days out,

ideally 30? Is there an owner written down? Do

we know where to fix it if renewal breaks? If

the answer is no, that's a cheap reliability

win you can fix without rewriting anything. And

if you want a meta lesson, certs are one of those

dependencies where the failure is so preventable

that it's painful. Which is why teams tend to

overcut corners until they get burned. Don't

wait for the burn. Story 3 helm chart reliability

prequel reviewed 105 popular open source helm

charts and they found the average reliability

score was roughly 3 .98 out of 10 with a median

around 4 out of 10. their point isn't helm is

bad it's that many charts ship demo friendly

defaults not reliability -friendly defaults.

This matters because charts aren't just packaging.

They encode operational behavior, readiness and

liveness, resource requests, update strategies,

disruption behavior, security contexts, sometimes

topology and scheduling assumptions. So when

you install a chart, you are adopting a set of

operational opinions whether you realize it or

not. Here's how this bites teams. A chart has

no resource requests. In dev, it looks fine.

In prod, under pressure, it becomes unpredictable.

Pods are throttled, get evicted, or get starved.

Probes are missing or sloppy. Traffic gets routed

to pods that aren't ready. Or probes are too

aggressive and under load, they trigger restarts.

no pod disruption budget, no topology spread.

And then routing node maintenance becomes a cascading

outage. Everything was highly available until

you drained a node and lost a majority of replicas

in one place. Unsafe update strategy and rollouts

turn into brownouts. And the worst version is

when Kubernetes says everything is green while

your app is melting. That's where chart defaults

turn into long incident timelines. So what do

you do without forking every chart? You create

a baseline overlay and a checklist. Baseline

overlay is a thin layer in your GitOps repo or

Terraform or Helm release config where you enforce

defaults. Resources required. Probes required.

Explicit update strategy. PDB when appropriate.

Spread constraints if the service needs real

availability. security context that matches your

cluster policy. And the checklist is just does

this chart behave predictably under rollout,

under node drain, under load spike. Now, the

do this Monday pass. Pick one chart you run in

production that matters. The one that would page

if it went sideways. Open its values and answer.

Are resource requests set? Are probes set and

meaningful? Do we have safe rollout behavior?

Do we have disruption behavior planned? Will

replicas spread across failure domains, or can

they all land on one node? What happens if a

node gets drained? If you can't answer quickly,

that's your signal. Add the defaults, document

them, and move on. You don't need perfect charts.

You need boring, predictable behavior. Now, N8n.

And this is new. Hacker News covered two high

severity vulnerabilities in N8n discovered by

JFrog. The short version is these flaws can let

authenticated users escape the sandbox and execute

code. One is in the expression sandbox and the

other involves Python code execution in internal

mode. Here's the important part. People hear

authenticated and they relax. In workflow automation

platforms, the permission model is often broader

than you think. Authenticated can include a lot

of people who can build or edit workflows. And

in tools like N8N, workflow editing is basically

code execution. Because workflows can evaluate

expressions and interact with credentials. So

this isn't, oh no, an attacker needs an account.

The real question is, who in your org has an

account? And what can their workflows touch?

And what makes this class of bug extra painful

is these tools often sit in the middle of your

environment holding keys. Slack, GitHub, Jira,

AWS keys, database credentials, webhooks, secret

managers, all of it. So a sandbox escape is not

just someone ran code. It's someone ran code

where the keys live. That's why we keep coming

back to N8N on this show. It's not because N8N

is uniquely bad. It's because the category is

high leverage. Okay, practical actions. First,

patch. Don't debate it. If you self -host N8n,

patch quickly when sandbox escapes drop. The

blast radius is too high to slow walk it. Second,

reduce who can author workflows. Don't treat

workflow editor as a casual permission. Treat

it like can run code in a privileged environment

because effectively that's what it is. Third,

reduce exposure. If your N8n UI is public on

the internet, you are playing on hard mode. Put

it behind SSO, VPN, IP allow lists, whatever

fits your org. You want fewer people able to

even reach the attack surface. Fourth, isolate

it. If it's holding keys to a bunch of systems,

at least give it a narrow runtime permission

set. Least privilege on the credentials it uses.

Separate credentials per workflow if you can.

Don't run it with god mode access to AWS just

because it was convenient once. And now the tie

back to our last episodes where we talked about

N8N CVEs. The theme has been consistent. Workflow

automation tools are basically control planes.

They need the same operational rigor you'd give

to an internal platform. Patch fast, lock down

authorship, reduce exposure, least privilege

the credentials. If you treat it like just a

tool, it will eventually treat you like just

a breach. All right, time for the lightning round.

First. Use Tusk Fence. It's a lightweight sandbox

for running commands that block network by default.

If you are experimenting with agents, runbooks

that execute, or any workflow where code runs

on behalf of a user, this is the kind of primitive

you want. Safe by default beats clever by default.

Next, HashiCorp agent skills. This is part of

a trend I actually like. Vendors shipping structured

reusable skills and guardrails instead of just

telling you to prompt better and hope. Next,

Marimo. It's a reactive Python notebook that

stores as normal Python. That sounds small, but

it matters because Git -friendly notebooks are

actually useful for incident analysis, runbooks,

or one -off ops experiments you want to keep

without the notebook JSON misery. And a quick

one from the register, the Ralph Wiggum clawed

loop story. It's funny, but the real point is

dead serious. People are already building loops

that keep agents running until they produce output.

Without constraints and verification, that becomes

confident nonsense at scale. The ops lesson is

the same as alert fatigue. If your system floods

you with low -quality output, humans stop trusting

it. Okay, time for the human closer. Every story

today is a glue failure. CI trigger logic becomes

security boundary and someone implemented it

like it was just config. Cert renewals were treated

like solved and then the cliff happened. Charts

were treated like installers, not operational

dependencies. And workflow automation tools are

treated like a convenience layer even though

they hold the keys. So the takeaway isn't stop

using tools. It's treat guardrails like product

work. Make untrusted pipelines truly untrusted.

Make cert monitoring external and owned. Make

helm baselines explicit. Make workflow authoring

privileged. Make credentials least privileged.

Because if you only build accelerators, you are

not building a better platform. You are just

building a faster incident. All right, time for

a quick recap. We talked about code breach and

why CI triggers and filters are real security

boundaries. We talked about Bazel and the CertCliff

problem. And we talked about Helm chart reliability

and why defaults matter more than installs. And

we talked about the new N8n sandbox escape flaws

and why workflow automation needs to be treated

like a control plane. The lightning round was

Fence, HashiCorp Agent Skills, Marimo, and The

Agent Loop Cautionary Tale. Links and show notes

are on shipitweekly .fm. If you got something

out of this, follow the show wherever you are

listening. And if you can, leave a quick rating

or review. It helps a ton. I'm Brian, and I'll

see you next week.

CodeBreach in AWS CodeBuild, Bazel TLS Certificate Expiry Breaks Builds, Helm Charts Reliability Audit, and New n8n Sandbox Escape RCE

Watch this episode here

Chapters

Transcript

Catch This Episode

Host Commentary

Show Notes

More from Ship It Weekly

EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

Ship It Conversations: Gareth Kersey on IaCConf 2026, AI, and Corey Quinn’s Terraform Keynote

containerd CRI Vulnerabilities, Datadog PostgreSQL HA on Kubernetes, AWS DevOps Agent with Datadog MCP Server, EKS Control Plane Egress, and Why Users Feel the Wait

Ship It Conversations: Jake Warner on Cycle.io, Bare Metal’s Comeback, and Why Private Cloud Is Getting Interesting Again

Get the next episode in your inbox