Cloudflare’s Workers Scheduler, AWS DBs on Vercel, and JIT Admin Access

Transcript

Thank you. Hey, I'm Brian Teller. I work in DevOps

and SRE and I run Teller's Tech. Ship It Weekly

is where I filter the noise and pull out what

actually matters when you're the one running

infrastructure and owning reliability. If something's

hype, I'll call it hype. If it changes how you

operate, I'll break it down in plain English.

Most weeks, this is a quick news recap. In between

those, I drop interview episodes with folks across

the DevOps world. Happy holidays. Merry Christmas,

all that good stuff. It's the day after Christmas,

so if you're listening while hiding from your

family, checking one thing real quick, I respect

it. Quick piece of housekeeping, the new site

is live at shipitweekly .fm. That's where I'm

putting links and show notes. Also, I'm looking

for interview guests. If you're building real

infra or platform stuff and you want to come

on for a chill conversation episode, hit the

email on shipitweekly .fm. If you've got war

stories, even better. And one quick ask before

we start. If the show has been useful, hit follow

or subscribe wherever you are listening. And

if you've got 10 seconds, a rating or review

really helps way more than it should. All right,

three main stories for today. First, Cloudflare

wrote up how they built an internal maintenance

scheduler on workers. This is real platform engineering.

Memory limits. data modeling, query optimization,

caching, Parquet for historical analysis. Second,

AWS databases are now available directly in Vercel

marketplace. It's a quiet shift, but it's a big

one. Devs can click button real AWS databases

from inside Vercel, but you still have to own

governance, billing, and the blast radius. Third,

there's an open source project from AWS called

Team. Temporary Elevated Access Management, and

it's built around IAM Identity Center. It's approval

-based, time -bound access. This is one of those

everybody wants it, few implement it cleanly

problems. Then we'll do a lightning round, and

we'll close with Mark Brooker's What Now? Handling

Errors in Large Systems. Let's get into it. Cloudflare

has a really good post on how they built an internal

maintenance brain on workers. The core problem

is kind of obvious once you hear it. when you

run infra at their scale you cannot rely on humans

to remember every dependency and every weird

routing rule and every if these two things go

down at the same time a customer special setup

gets wrecked scenario so they built a centralized

scheduler that treats maintenance like a set

of constraints like we must always have at least

one of these routers active or this customer

pins traffic through these data centers, so don't

take all of them out at once. The fun part is

how they got it working within worker's limits.

Their first naive approach was basically load

everything into one worker. All the relationships,

all the product config, all the metrics, then

compute constraints. And even in proof of concept,

they hit out of memory errors. So they took a

step back and said, okay. Workers have limits.

We can't treat this like a giant in -memory analytics

job. We need to only load the data that matters

for the specific maintenance request. If you

get a maintenance request for your router in

Frankfurt, you probably do not need to load Australia.

You need the dependency neighborhood around that

router. that pushed them into graph modeling

they describe constraints as objects and associations

basically vertices and edges routers are objects

pulls are objects and the dependencies are associations

and then they built a typed association interface

so the constraint logic stays simple but the

backing implementation can get smarter over time

then they flip their data fetching style instead

of pulling down huge responses and filtering

locally They started doing targeted requests

through the graph interface. They claim response

sizes dropped by 100 times in one spot. That's

huge. Also, it's the exact kind of win you get

when you stop shipping your entire dataset into

your app layer just to throw most of it away.

Of course, that created the next problem. sub

-request limits. They traded a few massive requests

for a ton of tiny requests. And then they started

breaching sub -request limits. So they built

a fetch pipeline with request deduping, a small

LRU cache, edge caching via caches .default,

and sane retry and back off. After tuning, they

were seeing about a 99 % cache hit rate on those

fetches. That's wild. And it's It's also how

you make something like this survive at scale

without tuning your internal APIs into a creator.

Then there's a super relatable metric story.

They use Thanos for Prometheus queries, and they

call out what a lot of teams do by accident.

Ask for everything. Get megabytes back, parse

JSON into single -threaded runtime, then filter

most of it out. That's basically self -inflicted

pain. instead they use the graph to find the

specific relationships first then issue much

more targeted thanos queries they say average

response size went from multiple megabytes to

about one kilobyte in one case so the theme of

this story is stop dragging huge blobs of data

into your application just so you can toss 99

of it and then they bring it home with historical

analysis real time is one thing historical is

a different because now you're scanning months

of data to see if your logic is actually safe

and accurate. They talk about how Prometheus

TSDB blocks are not really designed for object

storage access patterns and how that turns into

a lot of random reads. So they adopt a parquet

conversation layer for historical data. Columnar

format. better stats, and you can fetch what

you need without slamming object storage with

random IO. Takeaways you can steal even if you're

not Cloudflare. If you're building platform brains

like schedulers, deploy orchestrators, policy

evaluators, you will hit limits. Memory, CPU,

API quotas, request fanout. You win by changing

the shape of the problem, not brute forcing it.

Graph interfaces are a cheat code for dependency

-heavy domains. And targeted queries plus caching

plus backoff is still undefeated. All right,

let's move from Cloudflare did adult engineering

to developers can click -button databases. AWS

announced that AWS databases are now available

on the Vercel marketplace. The headline is simple.

From Vercel, you can provision and connect to

Aurora Postgres, Aurora DSQL, and DynamoDB in

seconds. And here's the part platform folks should

not ignore. The onboarding path is basically

create a new AWS account from Vercel with some

starter credits. So the dev experience is you're

already in Vercel. You click a thing, and now

you have a database, and the app is wired up.

That is great for Velocity. It is also the kind

of thing that bypasses governance if you don't

get in front of it. Even the AWS side is legit.

You still have to deal with who owns that AWS

account long term. Is it inside your AWS organization

or is it an orphan account that exists because

Vercel made it easy? How do SCPs apply? How do

guardrails apply? How do you do tagging and cost

allocation so finance doesn't show up later asking

why there are mystery accounts? What does networking

look like? Are public endpoints acceptable? Do

you need private connectivity? Do you have a

VPC strategy that fits Vercel -first teams? What's

your audit baseline? Cloud trail? Config? Detective

controls? All the boring stuff. Also, Region

selection matters for data, residency, and latency.

It's not just pick whatever. Vercel also hinted

this is evolving. They're talking about coming

soon support for provisioning into an existing

AWS account, not just a new one. If that lands

cleanly, that's the version I'd actually want

as a platform team because now you can meet developers

where they are without losing governance. So

the takeaway, if your dev platform can create

AWS resources, your governance has to meet it

there. The database is easy. The ownership model

is the hard part. All right. Story 3 is about

access, which matters even more once you have

more accounts and more surfaces. TEAM stands

for Temporary Elevated Access Management. It's

an open -source solution built around AWS IAM

Identity Center. The pitch is basically approval

-based, time -bound elevated access to AWS accounts.

Users request elevated access for a specific

period of time, with a reason. Approvers approve

or deny it. If approved, access is granted. When

time expires, it is automatically removed. That

automatic removal is the whole point. Because

most orgs fail here. Someone gets admin just

for this incident, and then it stays for months.

Privilege creep becomes the default. Team also

leans into auditing and visibility. Who requested

what, who approved it, when it expired, plus

session logging. How I'd frame it. This is not

your break glass story. Break glass is the world

is on fire and we need access right now. It should

be rare, noisy, and heavily monitored. Team is

the daily, I need admin for 45 minutes to do

this legit change workflow. If you want this

to actually work culturally, approvals need to

be fast enough that people don't route around

it. And the default permission sets need to be

sane so elevation is actually meaningful, not

just ceremony. If you are already on IAM Identity

Center and you've been hand -waving we should

do JIT access, team is at least worth a look.

Even if you don't adopt it, it's a good reference

for what time -bound elevation can look like

without building everything from scratch. Alright,

time for the lightning round. GitHub Actions

improved performance on the workflows page. Small

change, but if you live in Actions, you'll feel

it. Big workflows render better now, lazy loading,

and you can filter jobs by status so you can

just see failures or in -progress stuff. During

an incident, this is a real quality of life upgrade.

Next, the weird one, Lambda managed instances.

This is basically run Lambda functions on EC2,

but AWS manages the lifecycle of those instances

for you. It's meant for steady state workloads

and specialized compute needs. It also changes

concurrency and execution assumptions a bit.

So you actually need to care about thread safety

and shared state in ways you might not with regular

Lambda. Interesting, slightly cursed. But I get

why it exists. Atmos quick hit. There's a Cloud

Posse Atmos issue where vendoring a component

with an invalid URL triggers a weird GitHub username

prompt. I'm mentioning it less because of the

bug and more because Atmos is clearly turning

into a bigger workflow ecosystem now. CLI, dev

containers, IDE integrations, the whole thing.

And last, k8sdiagram .fun. It's a free Kubernetes

diagram builder that can also generate YAML for

common resources. I would not blindly apply auto

-generated YAML to prod, but for teaching, prototyping,

or explaining architecture to humans, it's actually

super handy. All right, let's close with the

human story, because this ties into everything

we just talked about. Mark Brooker wrote a post

called, What Now? Handling errors in large systems.

It's basically an interactive error handling

game. You decide whether a system should crash

or keep going when something goes wrong. And

then he explains his take. The key idea is simple.

And it's something we forget all the time. Error

handling isn't a local decision. It's a global

property of the system. We love to argue about

one line of code. Should this crash? Should this

retry? Should this be best effort? Mark's point

is, that decision only makes sense if you understand

the architecture around it. He asks questions

like, are failures correlated? If the same bad

input can hit every node, crashing can amplify

the blast radius. Can a higher layer handle the

air? Some architectures are designed to tolerate

a few crashes. None are designed to tolerate

a ton of crashes continuously. Is it actually

safe to keep crashing? Is it actually safe to

keep running? Sometimes continuing means silent

corruption, which is worse than a crash. Then

he ties it to blast radius reduction. Cell -based

architectures, independent regions, isolating

failures so you don't have a single mistake become

a global outage. That connects to today's stories

perfectly. Cloudflare's scheduler exists because

humans will guess wrong sometimes, and the systems

need to prevent correlated failure. The Vercel

Marketplace DB story is a new surface area. The

failure modes aren't just technical, they're

governance and ownership failures. And team is

literally about reducing the risk of one person

with standing admin turning a mistake into a

disaster. So yeah. If you want a good mindset

going into next year, stop treating air handling

like a code style preference. Treat it like architecture.

All right, that's it for this episode of Ship

It Weekly. We covered Cloudflare's maintenance

scheduler on workers and the platform limits

force better design lessons. AWS databases inside

the Vercel marketplace and what that means for

governance and blast radius. And team as a practical

path to time -bound elevated access with IAM

Identity Center. If you got something out of

this, hit follow or subscribe wherever you are

listening. And if you can, leave a quick rating

or review. It's annoying how much that helps

the show. Links and show notes are on shipitweekly

.fm. And one last reminder, I'm looking for interview

guests. If you want to come on and talk through

real DevOps or platform work you're doing, hit

the email on the site. I'm Brian. Thanks for

listening. Happy holidays. And I'll see you next

week, which is technically next year.

Cloudflare’s Workers Scheduler, AWS DBs on Vercel, and JIT Admin Access

Chapters

Transcript

Catch This Episode

Host Commentary

Show Notes

More from Ship It Weekly

EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

Ship It Conversations: Gareth Kersey on IaCConf 2026, AI, and Corey Quinn’s Terraform Keynote

containerd CRI Vulnerabilities, Datadog PostgreSQL HA on Kubernetes, AWS DevOps Agent with Datadog MCP Server, EKS Control Plane Egress, and Why Users Feel the Wait

Ship It Conversations: Jake Warner on Cycle.io, Bare Metal’s Comeback, and Why Private Cloud Is Getting Interesting Again

Get the next episode in your inbox