AWS re:Invent for Platform Teams, GKE at 130k Nodes, and Killing Staging

Transcript

Hey, I'm Brian and this is Ship It Weekly by

Tellers Tech. It's reinvent week, which means

AWS has firehosed us with announcements. Instead

of trying to read you the keynote, I want to

pull out the stuff that actually matters if you

run platforms, Kubernetes, or CI in the real

world. So here's the plan for this episode. First,

we're going to hit the AWS updates that I think

are worth caring about if you own networking,

clusters, data, or security. That includes things

like regional NAT gateways, Route 53 global resolver,

EKS capabilities, ECS express mode, S3 vectors,

50 terabyte S3 objects, Aurora dynamic masking,

code commit coming back from the dead, and IAM

policy autopilot. Then we will step outside AWS.

We will talk about Google's 130 ,000 node GKE

cluster and what lessons from that actually apply

if you are just trying to keep your 20 node prod

cluster sane. After that, we will get into a

piece called It's Time to Kill Staging and talk

about what testing in production should and should

not mean. In the lightning round, we will hit

a Terraform MCP server that lets AI tools speak

your Terraform modules, a neat EC2 instance ranking

tool for right sizing, and an SRE story from

Airbnb on adaptive traffic management. And then

we will close with a human story about the fate

of small open source and what that means for

all the tiny projects your platform probably

depends on. All right, let's start with AWS.

I am not going to cover every AI chip and marketing

bullet from reInvent. I want to split this into

four buckets. Networking, compute and platform.

data, and DevTools plus security. Let's start

with networking because that is where a lot of

the quiet pain usually lives. AWS announced regional

availability mode for NAT gateway. Instead of

spinning up one NAT per AZ and wiring custom

routes to each one, you can now create a single

regional NAT that automatically spans all the

AZs in your VPC and scales with where your workloads

actually are. Practically, that means simpler

route tables, fewer moving parts to keep in sync,

and a more straightforward story when you talk

about high availability for private subnets.

You still need to think about cost and IP space,

but the model is more one service per region

rather than a little cluster of pets per AZ.

On the DNS side, AWS introduced Route 53 Global

Resolver. This is an Anycast DNS service that

sits in front of both your public and private

DNS, and it adds some smarts on top of it. It

can filter queries to suspicious domains and

uses algorithmic analysis to detect things like

DNS tunneling and weird domain generation patterns.

Not just is this domain on a known bad list.

There is also an accelerated recovery pattern

in the docs for managing public DNS records faster

and more safely. The recent US East 1 DNS pain

is still fresh in a lot of people's minds. So

this is a good moment to ask yourself, if Route

53 has a bad day again, how fast can we move?

And do we actually know where all of our critical

DNS records live? So if you own networking, here's

how I would use these. First, look at where you

are today with NAT. If you have a mix of per

AZ NATs, some services still hairpinning through

old instances, and a bunch of legacy route table

entries, the regional mode might be a nice forcing

function to clean that up. Use this as a chance

to revisit how you allocate IPs and whether you

can make IP, AM, and prefix lists do more work

for you instead of hand curated cider spreadsheets.

Second, treat global resolver as part of your

threat model, not just a neat new service. If

you have any compliance or data exfiltration

concerns, ask how do we want DNS to behave for

protected environments? And what logs do we need

out of this to actually detect weird behavior?

All right. Networking ran over. Let's talk compute

and platform stuff. On the container side, AWS

launched Amazon ECS express mode. This is basically

an easy button for ECS. You point it at a container

image and it wires up an ECS service, cluster,

application load balancer, out 53 records, auto

scaling, the usual plumbing with production ready

defaults. You still have access to all of the

underlying resources if you want to tweak them,

but the entry path for a new service is much

simpler. On the Kubernetes side, Amazon EKS capabilities

went GA. Think of this as a fully managed bundle

of platform features on top of VKS. It gives

you Kubernetes native components for workload

deployment, cloud resource management, and resource

composition. The idea is AWS runs and patches

a bunch of the core platform bits, and you interact

with it using familiar Kubernetes APIs. The story

here is pretty clear. AWS is trying to give you

paved paths for app teams. If you are a smaller

shop or you do not have the people to build your

own golden path. ECS express mode and EKS capabilities

are an attractive let AWS worry about more of

the platform option. If you already have a strong

platform story, these are still worth watching

as a reference point for what batteries included

looks like. Layered on top of that, we have Lambda

durability functions. These let you write long

-running, stateful workflows directly as Lambda

functions. They can checkpoint progress, pause

for up to a year, resume after failures, and

you do not have to bolt on your own state machine

or air handling engine. That overlaps a bit with

what folks use step functions or DIY orchestrators

for today. I would not rip anything out just

because durability functions is shiny, but if

you were about to build a workflow system where

functions need to wait on AI agents or external

callbacks, I would at least prototype it this

way and see if it simplifies your life. Now let's

talk data and storage. S3 vectors is now generally

available with scale bumping up to billions of

vectors per index. and trillions per bucket.

It's the first time one of the big clouds has

said, yeah, object storage can natively store

and query vectors rather than forcing you into

a separate vector database. The marketing line

is up to 90 % lower cost compared to specialized

vector stores and tighter integration with bedrock

knowledge bases and open search. This is a pretty

big deal if you are doing RAG or semantic search.

You no longer have to manage a completely separate

database just for embeddings. You can treat vectors

as another dimension of your S3 data lake. There

are still plenty of reasons to use a dedicated

vector store for certain use cases, but for a

lot of internal tooling, this is going to be

good enough and way easier to operate. S3 also

quietly increased the maximum object size from

5 terabytes to 50 terabytes. That changes how

you think about backups, big media, and AI training

data. The days of having to shard every giant

file into dozens of pieces just to fit it into

S3 limits are mostly gone now. On the database

side, Aurora Postgres picked up dynamic data

masking using the PG column mask extension. You

can define policies at the column level so certain

roles see full values, others see masked values,

and so on, enforced in the database itself. That

is interesting if you have BI users, contractors,

or internal tools that should see some shape

of the data, but not raw PII. It also gives you

another tool for compliance stories where keeping

masking closed to the data is a plus. Just remember

masking is not encryption and it does not replace

good role design or auditing. Finally, dev tools

and security. AWS officially walked back the

we are de -emphasizing code commit thing. Code

commit is back to full general availability with

AWS saying clearly that customers still want

a fully managed Git service that lives inside

their AWS estate. If you are in a heavily regulated

environment or you just like having repos inside

the same blast radius as everything else, that

is probably a relief. It also raises some awkward

questions for teams that did a big migration

off of CodeCommit after the original deprecation

plan. There is a meta lesson here about how much

you want to depend on any vendor's this service

is here forever statement. And then there is

IAM policy autopilot. This is a new open source

MCP server that reads your application code and

helps generate IAM policies that match what you

are actually doing, instead of star everything

in hope. It is designed to plug into AI coding

tools so they can propose least privileged policies

as part of your workflow. On one hand, this is

fantastic. Writing good IAM is tedious and anything

that helps teams stop shipping wildly overly

permissive policies is welcome. On the other

hand, this is one of those great power, great

responsibility things. I would absolutely run

its output through human review and test, and

I would be very careful about letting a model

both propose and apply policies without a person

in the loop. So if you zoom out, what are the

big AWS themes for platform folks this year?

Networking and DNS get simpler and a bit smarter.

Containers and Kubernetes get more paved roads.

Data and AI workloads move closer to S3. Dev

tools get more tightly integrated with IAM and

AI. The work for you is deciding whether you

want to lean in and let AWS carry more of the

platform and where you still want to keep your

own opinionated stack. There's still more reinvent

to go. So if anything huge drops after this recording,

we'll pick it up in a future episode. All right,

let's step out of Las Vegas and switch gears

and talk about Google for a minute. Google published

a blog about how they built a 130 ,000 node GKE

cluster. This is experimental, not a new default

limit, but it is still a wild number. The officially

supported limit today is 65 ,000 nodes per cluster.

So they basically doubled that for this project.

The post talks about demand for massive AI and

batch workloads. Think training or serving large

models, very large scale simulations, things

where packing as much work as possible into a

single control plane has operational benefits.

They had to do a bunch of architectural work

to make this even remotely practical. Things

like sharding control plane traffic and carefully

tuning API server scaling so you're not just

DDoSing your own Kubernetes API. Being very deliberate

about how many objects live in ETCD since you

are easily into the millions of pods and other

resources. Using job -oriented tooling like Q

to manage scheduling and fairness. So one noisy

job does not starve everything else. Here's the

thing though. Most of us are never going to run

a 130 ,000 node cluster, and that is fine. The

real lessons that I think are useful at a normal

scale are control plane capacity is a thing you

should care about. Even at 100 nodes, you can

run into API throttling or controller backlogs

during deploys or incident storms. Seeing Google

talk about their control plane SLOs at this scale

is a nice reminder that we should probably have

some for our smaller clusters too. Job and workload

management matters. Whether you have five jobs

or 5 ,000, Being explicit about priorities, quotas,

and fairness is the difference between prod is

fine during big experiments and someone kicked

off a batch job and our customer traffic died.

One cluster is not always better. The temptation

with fancy scale numbers is maybe we should consolidate

everything into one giant cluster. For most orgs,

blast radius compliance and team boundaries are

better served by multiple smaller clusters. Even

if that means a bit more overhead in the tooling.

So my recommendation here is not go chase 130

,000 nodes. It is steal the thinking, read the

post, look at how they reason about control plane

scaling and scheduling, and then ask what the

equivalent version of that would look like at

your scale. All right, let's talk about staging

environments. There is a new stack article making

the rounds called it's time to kill staging,

the case for testing in production. The short

version is that staging environments are slow,

expensive, and often lie to you. And more teams

should lean into testing directly in production

with the right safety rails. I have mixed feelings,

which probably means the piece is doing its job.

On the one hand, a lot of us have worked in places

where staging is a bottleneck. 50 developers

all merging into a shared staging cluster that

does not really look like prod, then sitting

in a queue waiting for a staging sign -off that

is mostly vibes. When that staging environment

inevitably diverges from reality, you waste time

debugging issues that would never happen in prod,

and you miss issues that only show up under real

traffic patterns. On the other hand, test in

production without guardrails is just break production.

I think the healthy middle ground looks like

this. You treat staging as limited and cheap,

not sacred. Use it for fast feedback on basic

integration, maybe some performance smoke tests,

but do not pretend it is a perfect mirror. You

keep it simple enough that it is not its own

full -time job to maintain. Then you build serious

safety mechanisms into production. Feature flags

so you can roll changes out to 1 % of traffic

or only to internal users or only to specific

regions. Progressive delivery so you can ramp

traffic up and down based on real SLOs. Not just

it seems fine. Shadow traffic or replay so you

can feed realistic requests into new versions

without exposing users yet. good observability

and alerting so you know if the experiment is

hurting real people. The article's core point

is good. The only environment that exactly behaves

like production is production. So if you want

to be confident, you need to be able to experiment

there. Just do it intentionally. So if you are

listening to this and you own a platform, here

is a question to take back. If someone on your

team said, we want to turn off staging in six

months, what would you need in place in production

to feel safe? List those things. That is probably

your roadmap, whether or not you actually kill

staging. All right, let's hit a lightning round.

First quick one, Zachary Lober wrote a Terraform

custom module MCP server and released a project

called Terraform Ingest. It is a CLI and MCP

server that crawls your Terraform module repo,

summarizes them, and exposes that to AI tools

so they can understand your existing modules

instead of hallucinating new ones. This is exactly

the direction I expect a lot of teams to go.

Instead of asking a model, write me some random

terraform. You point it at your real modules

and you say, compose with these building blocks.

If you care about standardization and avoiding

weird snowflake stacks, this kind of pattern

is worth watching. Next, a small but very useful

tool, Runzons EC2 Instance Family Ranking. It

is a page that ranks EC2 families by passmark

CPU performance, split across x86 and ARM, and

it lets you dig into detailed benchmarks and

even pricing via their API. If you ever picked

an instance type purely by habit, this is the

antidote. Before you copy paste M5 large, again,

you can check where M7G or M8A sit on the performance

curve and what that means for your workloads.

It is a nice way to bring a little data into

those instance choice conversations without running

your own benchmark suite. And the last lightning

item, an SRE story from Airbnb. They published

From Static Rate Limiting to Adaptive Traffic

Management in Airbnb's Key Value Store. It is

about their Key Value Store muscle and how they

evolve static QPS limits into a more adaptive

system that looks at short -term latency relative

to long -term baseline and adjust limits dynamically.

The interesting idea here is using ratios like

current P95 over trailing P95 as a signal that

the system is under stress. the rate limiter

react before things really fall over. Even if

you do not copy their exact design, it is a nice

example of moving beyond fixed per customer limits

into something that responds to real conditions.

All right, let's close it with a human story.

Nolan Lawson wrote a piece called The Fate of

Small Open Source. It is about tiny libraries

and tools and the stuff that might be a few hundred

lines of code that still get millions of downloads

and quietly sit at the bottom of everyone's dependency

graph. He talks about one of his own packages

that has been around for about a decade, still

getting 5 million downloads per week and how

LLMs change the equation. If you can just ask

a model to spin out a custom helper function,

do you really need another dependency? And if

people do keep using these tiny libraries, what

does that mean for the one maintainer handling

issues and security reports for free? I think

this hits home for platform teams in two ways.

First, look at your own tooling. Terraform providers,

little CLI helpers, internal scripts, custom

controllers. A lot of that probably rests on

one or two small open source projects that someone

wrote on nights and weekends. TerraScam being

archived recently is a good reminder that tools

you depend on can go away when the incentives

for the maintainers shift. Second, we are starting

to see companies say, why add a new dependency

when an LLM can just generate the five lines

of code we need? That might - reduce supply chain

risk in some places, but it also raises questions

about how new utility libraries get created and

maintained in the first place. So what do you

do with that? Practically, I would make a list

of small critical dependencies in your platform.

Things where if the repo went read -only tomorrow,

you would be in trouble. Ask what your plan B

is. Could you fork it? Vendor it or replace it

if needed. Maybe consider sponsoring a few of

these maintainers. Even a small amount can make

a difference and is a good signal to the rest

of your org that this stuff matters. Think twice

before outsourcing important functionality to

a single tiny project without at least acknowledging

the risk. And when you are tempted to say the

AI can just generate this, maybe also think about

whether that code will need to be maintained,

audited, and shared across teams in the future.

Sometimes the boring little libraries with tests

and a maintainer is still the better choice.

All right. That is it for this episode of Ship

It Weekly. We walked through the AWS reinvent

updates that actually matter for platform teams.

from regional NAT gateways and Route 53 global

resolver to ECS express mode, EKS capabilities,

S3 vectors, 50 terabyte objects, Aurora dynamic

masking, code commits return to GA, and IAM policy

autopilot. We looked at Google's 130 ,000 node

GKE cluster and used it as a lens on control

plane scaling and cluster design at more normal

sizes. We dug into the kill staging test in production

argument and how to make that safe with feature

flags and progressive delivery. In the lightning

round, we talked about Terraform MCP servers

for module -aware AI, EC2 instance rankings that

help you right -size with data instead of vibes,

and Airbnb's adaptive traffic management for

their key value store. And we wrapped with Nolan

Lawson's piece on the fate of small open source,

and what that means for all the tiny projects

your platform silently leans on. I will put all

of the links we talked about in the show notes.

I am Brian. This is Ship It Weekly by Teller's

Tech. Thanks for hanging out and I'll see you

in the next one.

AWS re:Invent for Platform Teams, GKE at 130k Nodes, and Killing Staging

Chapters

Transcript

Catch This Episode

Host Commentary

Show Notes

More from Ship It Weekly

Ship It Conversations: Gareth Kersey on IaCConf 2026, AI, and Corey Quinn’s Terraform Keynote

EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

Ship It Conversations: Jake Warner on Cycle.io, Bare Metal’s Comeback, and Why Private Cloud Is Getting Interesting Again

CISA’s GitHub Leak, AI Root Cause Analysis, Copilot Agents, Claude Code in CI/CD, and Kubernetes Seccomp Risk

Get the next episode in your inbox