Hey, I'm Brian and this is Ship It Weekly by Tellers Tech. This episode is very much a running real systems in production one. Kubernetes just dropped an official configuration good practices guide. AWS is quietly admitting that EKS control planes and networking need more love, and GitHub is pushing harder into being your platform brain with OIDC tweaks and copilot customization.
Then we'll hit a lightning round with Terrascan getting archived, a ridiculous 15 terabit DDoS against Azure, and CloudFront flat rate pricing. We'll then close with a human story about writing incident reports for a future AI SRE and why that actually makes them better for humans too. Let's start with Kubernetes config.
Kubernetes posted a new configuration good practices blog and it is very much a please stop hurting yourselves kind of document. The short version is configuration seems small until it absolutely is not. The blog pulls together a bunch of things most of us learn the hard way. Stuff like keep configuration in source control instead of hand editing manifests on a jump box.
Use tools like Customize or Helm in a consistent way so you do not end up with five different templating patterns in the same company. Avoid magic defaults. Be deliberate about labels and annotations so you can actually select and observe your workloads later. There is also a theme of prefer simple explicit config over clever tricks.
Things like avoiding giant copy pasted yaml blobs across environments, keeping environment specific bits in overlays instead of forking charts and validating config. Early with tools instead of discovering issues when you hit apply on production. None of this is shocking, but that is kind of the point. The worst Kubernetes incidents I see are almost never Kubla exploded.
It is usually a small config problem that slipped through. So if you are running clusters, here is how I would use this blog in real life. First, treat it as a checklist. Take one of your core apps and walk through how you do config today. Is everything for that app in Git, or do you still have a couple of hotfix -only manifests that people kubectl apply from their laptop?
Are you mixing Helm and raw YAML and customize in the same namespace? Do you actually have a policy or linterstep in CI that fails on bad patterns, or is it we just trust reviewers to catch it? Second, use it to standardize your approach. Most orgs have grown Kubernetes config very organically. One team copied a blog post, another team used a different starter repo.
Five years later, you have four different ways of doing environment overlays and nobody really knows which is the right one. The Kubernetes post gives you a neutral reference you can point at when you say, okay, we are going to converge on this pattern. Third, treat config as part of your platform. If you have a platform or infra team, one of your jobs is to make it harder for people to write bad config.
That might mean shipping base helm charts with same defaults, adding OPA or Kyverno policies, wiring up schema validation, or even just having a a kubectl diff step before applies in CI. Little stuff that stops the oops prod moments. So if you only skim one thing this week, that config good practices blog is worth a read and honestly, worth a little internal workshop with your team. All right, let's talk EKS.
AWS announced something called EKS provision control plane. This is basically stop guessing how much control plane you get. Historically, If you spin up an EKS cluster, AWS handles the control plane for you. It scales, there is some magic, and you mostly only think about worker nodes. But at scale or during big traffic spikes, you can absolutely run into API throttling and control plane saturation.
You see weird cube API latencies during deploys or controllers getting rate limited. With provisioned control plane, you can pick from scaling tiers that guarantee a certain level of control plane capacity. The docs talk about selecting a tier for high, predictable performance, regardless of current utilization.
So instead of hoping AWS auto scales in time, you say this cluster needs this much headroom and you pay for that. I like this for a few reasons. First, it forces people to take control plane SLOs seriously.
Most teams have metrics and alerts for node CPU and pod restarts, but almost nobody writes our cube API P99 latency should stay under X, or the cluster should be able to handle Y QPS of controller traffic during a big deploy. Provision tiers give you a way to align those expectations with something concrete in the platform. Second, it clarifies multi -tenant cluster design.
If you are stuffing a ton of teams into a few big clusters, you are all sharing that control plane. Being able to size it explicitly and maybe have a different tier for noisy clusters is a nice lever. AWS also dropped enhanced container network observability. For EKS. This gives you granular network metrics for cluster traffic.
Cross AZ flows, and traffic to AWS services with CloudWatch visualizations and deeper flow insight. Translated to human, they're making it easier to answer the question, what the heck is talking to what inside of your clusters? That helps with finding cross AZ chatty services that are quietly burning money and adding latency, spotting weird egress patterns to the internet or to manage services.
Debugging, this service is slow, but CPU and memory look fine. If you have ever done the dance where an app team swears it is not them causing cross AZ data transfer and you are trying to prove it, this kind of visibility is what you want. The pattern here is pretty clear. EKS is growing features where real production pain lives. Control plane capacity and networking.
As a platform team, this is a good time to write some SLOs for your control plane. Decide which cluster actually need a provision tier. Hook the network observability metrics into your existing dashboards and alerts rather than letting them sit in a default cloud watch view nobody opens. All right, let's shift over to GitHub and talk about OIDC and Copilot.
GitHub shipped a neat little security and governance improvement for Actions. The OIDC token that Actions uses to talk to cloud providers now includes a check run ID claim. Before this, you could tell this token came from this workflow run, but it was harder to tie it to a specific job or check.
Now, if a workflow calls into AWS or an internal service using OIDC, you can log and enforce policies based on the exact check run that generated the token. For platform teams, that means a few things. You can write more fine -grained IAM or ABAC policies on the cloud side that say only this particular job in this repo can assume this role, instead of any workflow from this org. That is big for least privilege.
You can audit access better. When someone asks who touched this role or what did this job actually do in AWS, there is a clear link between the token, the check run, and the cloud actions. You can now also use it for internal services. If you have an internal deploy API or some platform endpoint that Action calls, it can now require a check run ID and log it.
Instead of just trusting that anything with a valid token is allowed. On the AI side, GitHub published unlocking the full power of Copilot code review. Master your instructions files. It is basically a guide to making Copilot code review actually useful instead of spammy. Instruction files let you tell Copilot what you care about in reviews. That could be things like, this repo uses specific architecture.
We prefer structured logging. Avoid unsafe SQL or this service is latency sensitive. Be careful about extra network calls. This is very platformy because you can standardize expectations across repos. Instead of everyone writing their own reviewer guidelines in a wiki that nobody reads, you can put it in an instructions file that your AI reviewer actually uses.
Encode security and performance concerns as part of the review process. Help junior devs by having Copilot nudge them towards patterns your platform team wants. GitHub also has a customization library for Copilot that includes custom agents.
These are specialized versions of the Copilot coding agent that you configure for specific workflows, like implementation planner, bug fix teammate, or your own workspace specific helpers. You can define their behavior in files under .github .agents, and they keep that persona across a workflow rather than being a one -off prompt.
Where this gets interesting for platform teams is that you can start to imagine internal agents that know your stack. Your standard modules, your CI patterns. Know your security policies and naming conventions. Help people generate Terraform, Kubernetes manifests, or CI configs that line up with your platform. Obviously, you still need guardrails.
You do not want an overeager agent writing IAM policies or production manifests and merging them without human review. But it is pretty clear that in the next couple of years, we are going to be building platforms for humans and for AI helpers at the same time. So tying this segment together, GitHub is not just a repo anymore.
It is your off -bridge into the cloud via OIDC, and it is becoming the place where you define how AI participates in your code reviews and workflows. All right, let's hit a lightning round. Quick hits. Each of these could be its own rabbit hole, but I will keep it short. Terra scan. The IAC security scanner that a lot of folks used for Terraform and Kubernetes has now been archived on GitHub.
The repo shows that it was archived by the owner on November 20th, 2025 and is now read -only. Tenable also ended support for TerraScan in their Nessus release notes earlier, recommending their own cloud security product instead. So if Terrascan is still in your pipeline, this is a nudge to treat it like any other deprecated dependency. Design your CI so the scanner is a pluggable step, not baked into everything.
Swapping to Trivi or OPA should be a config change, not a rewrite. Next up, Azure just blocked a record 15 .72 terabit per second DDoS attack sourced from an IoT botnet. The Microsoft post and follow -on coverage talk about over 500 ,000 IPs, multi -vector attacks, and traffic equivalent to millions of Netflix streams hitting a single endpoint in Australia. The interesting takeaway is not, wow, that is big.
It is that this was powered by compromised home routers and cameras. And Azure's DDoS protections handled it without customer impact. For us, it is a reminder to actually understand what DDoS protections we have on our own providers. Are we just relying on defaults or have we validated our mitigation tiers, alerting and run books? On the cost side, AWS introduced flat rate pricing plans for website delivery.
These are cloud front based plans in free, pro, business, and premium tiers that bundle CDN, some security features, and S3 storage credits into a monthly price with no overages. From a FinOps and platform perspective, this is interesting if you have fairly predictable traffic and you want a cap on surprise bills. It is less exciting for very spiky workloads or stuff that is still experimental.
But if you are running a big marketing site or a core app with steady usage, this might be worth modeling. The key question is, can I align teams and cost allocation with this kind of plan without creating a mess? All right, let's close with the human side. For this episode's human story, I want to pull from a post by Lauren Hutchstein called Two Thought Experiments, which showed up in SRE Weekly Issue 498.
One of the thought experiments is about incident reports and AI. He basically asks, what if we assume that our incident reports will be consumed by an AI SRE tool in the future? What kinds of details would be useful to that tool in helping troubleshoot future incidents? And if we wrote with that in mind, would humans actually get more value out of those reports too?
I really like this framing because it solves a couple of problems at once. Right now, a lot of postmortems end up as vague timelines, a couple of bullet points about root cause, and some action items that may or may not get done. There is often not enough detail about how people reasoned, what signals were confusing, and what trade -offs they made in the moment.
If you imagine an AISRE reading that, trying to learn how your systems fail and how your humans debug them, you realize pretty quickly that you need to include more of the work in the writeup. Not just service X was slow, but we first suspected dependency Y because of metric Z, tried mitigation A, saw it did nothing, and then switched hypothesis because of this specific signal.
Those same details are exactly what help new humans learn from incidents 2. So a practical way to bring this into your own team is not to say we are writing for robots now, but something like, let's pretend an AI assistant is going to read this and use it to help the on -call next time. What would we need to capture so it actually learns something?
Hypothesis we tried, signals that turned out to be misleading, constraints we were under, the parts that were surprising. If you bake that expectation into your templates and your review process, you get better incident reports today, and you also future -proof yourself a bit for when you do introduce more AI into your incident response tooling.
And the side benefit is you shift the focus from blame and root cause into understanding and how people and systems interact under pressure, which is usually where the real learning is. All right, that is it for this episode of Ship It Weekly. We talked about the new Kubernetes configuration good practices and how they map to the boring but critical work of keeping your clusters sane.
We looked at EKS provision control plane and enhanced network observability and what that means if you are running big multi -tenant clusters. We dug into GitHub's updates around Actions OIDC. Copilot instructions, and custom agents, and how that all ties into platform and AI workloads.
In the lightning round, we touched on Terrascan being archived, the massive Azure DDoS that was quietly handled, and CloudFront flat rate plans for more predictable costs. And we wrapped with Lauren's thought experiment about writing incident reports for a future AI SRE, and how that mindset actually improves things for the humans reading them today. I will put all of the links we talked about in the show notes.
I am Brian. This is Ship It Weekly by Tellers Tech. Thanks for hanging out and I'll see you in the next one.
Scroll inside the box to read the full transcript, or expand for a larger view.
