Kubernetes Config Reality Check, EKS Control Planes, and GitHub Guardrails

Transcript

Hey, I'm Brian and this is Ship It Weekly by

Tellers Tech. This episode is very much a running

real systems in production one. Kubernetes just

dropped an official configuration good practices

guide. AWS is quietly admitting that EKS control

planes and networking need more love, and GitHub

is pushing harder into being your platform brain

with OIDC tweaks and copilot customization. Then

we'll hit a lightning round with Terrascan getting

archived, a ridiculous 15 terabit DDoS against

Azure, and CloudFront flat rate pricing. We'll

then close with a human story about writing incident

reports for a future AI SRE and why that actually

makes them better for humans too. Let's start

with Kubernetes config. Kubernetes posted a new

configuration good practices blog and it is very

much a please stop hurting yourselves kind of

document. The short version is configuration

seems small until it absolutely is not. The blog

pulls together a bunch of things most of us learn

the hard way. Stuff like keep configuration in

source control instead of hand editing manifests

on a jump box. Use tools like Customize or Helm

in a consistent way so you do not end up with

five different templating patterns in the same

company. Avoid magic defaults. Be deliberate

about labels and annotations so you can actually

select and observe your workloads later. There

is also a theme of prefer simple explicit config

over clever tricks. Things like avoiding giant

copy pasted yaml blobs across environments, keeping

environment specific bits in overlays instead

of forking charts and validating config. early

with tools instead of discovering issues when

you hit apply on production. None of this is

shocking, but that is kind of the point. The

worst Kubernetes incidents I see are almost never

Kubla exploded. It is usually a small config

problem that slipped through. So if you are running

clusters, here is how I would use this blog in

real life. First, treat it as a checklist. Take

one of your core apps and walk through how you

do config today. Is everything for that app in

Git, or do you still have a couple of hotfix

-only manifests that people kubectl apply from

their laptop? Are you mixing Helm and raw YAML

and customize in the same namespace? Do you actually

have a policy or linterstep in CI that fails

on bad patterns, or is it we just trust reviewers

to catch it? Second, use it to standardize your

approach. Most orgs have grown Kubernetes config

very organically. One team copied a blog post,

another team used a different starter repo. Five

years later, you have four different ways of

doing environment overlays and nobody really

knows which is the right one. The Kubernetes

post gives you a neutral reference you can point

at when you say, okay, we are going to converge

on this pattern. Third, treat config as part

of your platform. If you have a platform or infra

team, one of your jobs is to make it harder for

people to write bad config. That might mean shipping

base helm charts with same defaults, adding OPA

or Kyverno policies, wiring up schema validation,

or even just having a a kubectl diff step before

applies in CI. Little stuff that stops the oops

prod moments. So if you only skim one thing this

week, that config good practices blog is worth

a read and honestly, worth a little internal

workshop with your team. All right, let's talk

EKS. AWS announced something called EKS provision

control plane. This is basically stop guessing

how much control plane you get. Historically,

If you spin up an EKS cluster, AWS handles the

control plane for you. It scales, there is some

magic, and you mostly only think about worker

nodes. But at scale or during big traffic spikes,

you can absolutely run into API throttling and

control plane saturation. You see weird cube

API latencies during deploys or controllers getting

rate limited. With provisioned control plane,

you can pick from scaling tiers that guarantee

a certain level of control plane capacity. The

docs talk about selecting a tier for high, predictable

performance, regardless of current utilization.

So instead of hoping AWS auto scales in time,

you say this cluster needs this much headroom

and you pay for that. I like this for a few reasons.

First, it forces people to take control plane

SLOs seriously. Most teams have metrics and alerts

for node CPU and pod restarts, but almost nobody

writes our cube API P99 latency should stay under

X, or the cluster should be able to handle Y

QPS of controller traffic during a big deploy.

Provision tiers give you a way to align those

expectations with something concrete in the platform.

Second, it clarifies multi -tenant cluster design.

If you are stuffing a ton of teams into a few

big clusters, you are all sharing that control

plane. Being able to size it explicitly and maybe

have a different tier for noisy clusters is a

nice lever. AWS also dropped enhanced container

network observability. for EKS. This gives you

granular network metrics for cluster traffic.

cross AZ flows, and traffic to AWS services with

CloudWatch visualizations and deeper flow insight.

Translated to human, they're making it easier

to answer the question, what the heck is talking

to what inside of your clusters? That helps with

finding cross AZ chatty services that are quietly

burning money and adding latency, spotting weird

egress patterns to the internet or to manage

services. Debugging, this service is slow, but

CPU and memory look fine. If you have ever done

the dance where an app team swears it is not

them causing cross AZ data transfer and you are

trying to prove it, this kind of visibility is

what you want. The pattern here is pretty clear.

EKS is growing features where real production

pain lives. Control plane capacity and networking.

As a platform team, this is a good time to write

some SLOs for your control plane. Decide which

cluster actually need a provision tier. hook

the network observability metrics into your existing

dashboards and alerts rather than letting them

sit in a default cloud watch view nobody opens.

All right, let's shift over to GitHub and talk

about OIDC and Copilot. GitHub shipped a neat

little security and governance improvement for

Actions. The OIDC token that Actions uses to

talk to cloud providers now includes a check

run ID claim. Before this, you could tell this

token came from this workflow run, but it was

harder to tie it to a specific job or check.

Now, if a workflow calls into AWS or an internal

service using OIDC, you can log and enforce policies

based on the exact check run that generated the

token. For platform teams, that means a few things.

You can write more fine -grained IAM or ABAC

policies on the cloud side that say only this

particular job in this repo can assume this role,

instead of any workflow from this org. That is

big for least privilege. You can audit access

better. When someone asks who touched this role

or what did this job actually do in AWS, there

is a clear link between the token, the check

run, and the cloud actions. You can now also

use it for internal services. If you have an

internal deploy API or some platform endpoint

that Action calls, it can now require a check

run ID and log it. instead of just trusting that

anything with a valid token is allowed. On the

AI side, GitHub published unlocking the full

power of Copilot code review. Master your instructions

files. It is basically a guide to making Copilot

code review actually useful instead of spammy.

Instruction files let you tell Copilot what you

care about in reviews. That could be things like,

this repo uses specific architecture. We prefer

structured logging. Avoid unsafe SQL or this

service is latency sensitive. Be careful about

extra network calls. This is very platformy because

you can standardize expectations across repos.

Instead of everyone writing their own reviewer

guidelines in a wiki that nobody reads, you can

put it in an instructions file that your AI reviewer

actually uses. Encode security and performance

concerns as part of the review process. Help

junior devs by having Copilot nudge them towards

patterns your platform team wants. GitHub also

has a customization library for Copilot that

includes custom agents. These are specialized

versions of the Copilot coding agent that you

configure for specific workflows, like implementation

planner, bug fix teammate, or your own workspace

specific helpers. You can define their behavior

in files under .github .agents, and they keep

that persona across a workflow rather than being

a one -off prompt. Where this gets interesting

for platform teams is that you can start to imagine

internal agents that know your stack. your standard

modules, your CI patterns. Know your security

policies and naming conventions. Help people

generate Terraform, Kubernetes manifests, or

CI configs that line up with your platform. Obviously,

you still need guardrails. You do not want an

overeager agent writing IAM policies or production

manifests and merging them without human review.

But it is pretty clear that in the next couple

of years, we are going to be building platforms

for humans and for AI helpers at the same time.

So tying this segment together, GitHub is not

just a repo anymore. It is your off -bridge into

the cloud via OIDC, and it is becoming the place

where you define how AI participates in your

code reviews and workflows. All right, let's

hit a lightning round. Quick hits. Each of these

could be its own rabbit hole, but I will keep

it short. Terra scan. The IAC security scanner

that a lot of folks used for Terraform and Kubernetes

has now been archived on GitHub. The repo shows

that it was archived by the owner on November

20th, 2025 and is now read -only. Tenable also

ended support for TerraScan in their Nessus release

notes earlier, recommending their own cloud security

product instead. So if Terrascan is still in

your pipeline, this is a nudge to treat it like

any other deprecated dependency. Design your

CI so the scanner is a pluggable step, not baked

into everything. Swapping to Trivi or OPA should

be a config change, not a rewrite. Next up, Azure

just blocked a record 15 .72 terabit per second

DDoS attack sourced from an IoT botnet. The Microsoft

post and follow -on coverage talk about over

500 ,000 IPs, multi -vector attacks, and traffic

equivalent to millions of Netflix streams hitting

a single endpoint in Australia. The interesting

takeaway is not, wow, that is big. It is that

this was powered by compromised home routers

and cameras. And Azure's DDoS protections handled

it without customer impact. For us, it is a reminder

to actually understand what DDoS protections

we have on our own providers. Are we just relying

on defaults or have we validated our mitigation

tiers, alerting and run books? On the cost side,

AWS introduced flat rate pricing plans for website

delivery. These are cloud front based plans in

free, pro, business, and premium tiers that bundle

CDN, some security features, and S3 storage credits

into a monthly price with no overages. From a

FinOps and platform perspective, this is interesting

if you have fairly predictable traffic and you

want a cap on surprise bills. It is less exciting

for very spiky workloads or stuff that is still

experimental. But if you are running a big marketing

site or a core app with steady usage, this might

be worth modeling. The key question is, can I

align teams and cost allocation with this kind

of plan without creating a mess? All right, let's

close with the human side. For this episode's

human story, I want to pull from a post by Lauren

Hutchstein called Two Thought Experiments, which

showed up in SRE Weekly Issue 498. One of the

thought experiments is about incident reports

and AI. He basically asks, what if we assume

that our incident reports will be consumed by

an AI SRE tool in the future? What kinds of details

would be useful to that tool in helping troubleshoot

future incidents? And if we wrote with that in

mind, would humans actually get more value out

of those reports too? I really like this framing

because it solves a couple of problems at once.

Right now, a lot of postmortems end up as vague

timelines, a couple of bullet points about root

cause, and some action items that may or may

not get done. There is often not enough detail

about how people reasoned, what signals were

confusing, and what trade -offs they made in

the moment. If you imagine an AISRE reading that,

trying to learn how your systems fail and how

your humans debug them, you realize pretty quickly

that you need to include more of the work in

the writeup. not just service X was slow, but

we first suspected dependency Y because of metric

Z, tried mitigation A, saw it did nothing, and

then switched hypothesis because of this specific

signal. Those same details are exactly what help

new humans learn from incidents 2. So a practical

way to bring this into your own team is not to

say we are writing for robots now, but something

like, let's pretend an AI assistant is going

to read this and use it to help the on -call

next time. What would we need to capture so it

actually learns something? Hypothesis we tried,

signals that turned out to be misleading, constraints

we were under, the parts that were surprising.

If you bake that expectation into your templates

and your review process, you get better incident

reports today, and you also future -proof yourself

a bit for when you do introduce more AI into

your incident response tooling. And the side

benefit is you shift the focus from blame and

root cause into understanding and how people

and systems interact under pressure, which is

usually where the real learning is. All right,

that is it for this episode of Ship It Weekly.

We talked about the new Kubernetes configuration

good practices and how they map to the boring

but critical work of keeping your clusters sane.

We looked at EKS provision control plane and

enhanced network observability and what that

means if you are running big multi -tenant clusters.

We dug into GitHub's updates around Actions OIDC.

copilot instructions, and custom agents, and

how that all ties into platform and AI workloads.

In the lightning round, we touched on Terrascan

being archived, the massive Azure DDoS that was

quietly handled, and CloudFront flat rate plans

for more predictable costs. And we wrapped with

Lauren's thought experiment about writing incident

reports for a future AI SRE, and how that mindset

actually improves things for the humans reading

them today. I will put all of the links we talked

about in the show notes. I am Brian. This is

Ship It Weekly by Tellers Tech. Thanks for hanging

out and I'll see you in the next one.

Kubernetes Config Reality Check, EKS Control Planes, and GitHub Guardrails

Chapters

Transcript

Catch This Episode

Host Commentary

Show Notes

More from Ship It Weekly

EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

Ship It Conversations: Gareth Kersey on IaCConf 2026, AI, and Corey Quinn’s Terraform Keynote

containerd CRI Vulnerabilities, Datadog PostgreSQL HA on Kubernetes, AWS DevOps Agent with Datadog MCP Server, EKS Control Plane Egress, and Why Users Feel the Wait

CISA’s GitHub Leak, AI Root Cause Analysis, Copilot Agents, Claude Code in CI/CD, and Kubernetes Seccomp Risk

Get the next episode in your inbox