Ship It Weekly Transcripts

AI is making it easier than ever to write code. That sounds great. Until you are the team responsible for everything that has to exist after the code gets written. The infrastructure. The deploy path. The secrets. The permissions. The cost. The rollback plan. The weird Terraform change someone generated at 4:30 on a Friday and swears is probably fine.

Because when app teams move faster, platform and infrastructure teams do not magically get less work. They get more change, more pressure, more risk, and a lot more questions about whether the systems around delivery can actually keep up. That is really what this conversation is about.

Not just AI, not just Terraform, not just infrastructure as code, but what happens when software delivery speeds up and the people running the infrastructure have to figure out how to keep things safe, reliable, governed, and still usable. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech.

Ship It Weekly is where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability. Most weeks, it's a quick news recap. In between those, I do conversation episodes with people who are building platforms, running infrastructure, organizing events, and thinking through where this whole industry is actually headed. Today is one of those conversations.

I'm joined by Gareth Kersey to preview IACConf 2026. IACConf is a free virtual conference focused on infrastructure as code, platform engineering, DevOps, and infrastructure operations. And the theme this year is basically how infrastructure teams keep pace when AI is changing the speed of software delivery. And I like that framing because it gets past the lazy version of the AI conversation.

This is not just can AI write Terraform. It probably can. Sometimes badly. Sometimes usefully. Sometimes like a tourist who learned just enough of a language to confidently order the wrong thing. The better question is, what happens after that? What happens when developers can generate more code, ship more changes, and push product roadmaps faster?

What does that mean for DevOps, SRE, platform, and infrastructure teams who still have to deal with the blast radius.

In this conversation, Gareth walks through the IACConf agenda, including Corey Quinn's keynote, sessions on AI and infrastructure operations, platform engineering panels, Kubernetes, Argo CD, AI agents managing infrastructure as code, governance, policy, and the risk of 10x code velocity turning into 10x operational risk.

We also talk about how IACConf has grown, why people seem hungry for an event that goes deeper on infrastructure as code, and how they are trying to keep it community focused instead of turning into just another vendor marketing conference. That part stood out to me because infrastructure as code is one of those topics that sits at the intersection of almost everything now.

Terraform, OpenTofu, Pulumi, Crossplane, Kubernetes, GitOps, Policy, Security, Cost. Developer experience, internal platforms, and now AI-generated changes being pushed into systems that were already complicated enough before the robots showed up.

So if you work around infrastructure, platform engineering, DevOps, SRE, or you are just trying to figure out how to keep your delivery system sane while everything around it speeds up, this one should be worth your time. All right, let's jump in. Today, I'm joined by Gareth Kersey from IACConf.

We're doing a fast preview of IACConf 2026, what the theme is, what talks are worth bookmarking, and what infra teams should be doing now that AI is changing the pace of shipping. Gareth, thank you for joining me. Thanks, Brian. Excited to be here. So I'm interested to hear, what does keeping pace mean?

Yeah, so it's not a surprise probably to anyone with whether you're in engineering, You're in any sort of field operations function. What AI is doing to kind of everyone's day -to -day function, right? Everyone's feeling the pressure to move faster, do more with less. And I think one of the biggest places that has been upended and really drastically changed is software engineering, right? The rate at which you can spin up an application, you can code with cloud code.

What does that actually mean on the other side for operations teams well now you've got developers are shipping way faster that product roadmap is getting through a lot faster but that application code isn't always going to be the same maybe level of quality or the thoroughness that are security checks what does that mean for infrastructure all the kind of the downstream operations that are needed to actually sustain

That application deployment is that actually keeping pace so that's what we mean by keeping pace and obviously with it being infrastructure as code conference, IACConf.

We're really focused on the infrastructure operations components of that. But we do have a lot of sessions that touch on more than just infrastructure as code, because realistically, whether you're a DevOps, SRE platform, IAC is one piece of the puzzle that you're working with. And so we've got several talks that talk a little more broadly about keeping pace in terms of like the app operations standpoint.

So that's a good precursor. What are the talks? Let's kind of dive into that. Yeah. I saw some of them. You had mentioned some of the speakers on the IACConf page. Can you give me just a high -level overview? Who is speaking? What are they speaking about? Yeah, for sure. Yeah, we've got a really good agenda this year.

It's a mix of, you know, a couple panels, a couple talks that are more, you know, framework, point of view on the market, and then a few talks which are down in the weeds, more demos. You know, so we've tried to create a nice array of topics. So this is, let's see, we did IACConf, the first one in May of 2025. That was 13 sessions, really great response, a lot of good feedback.

Then we did a security focused virtual spotlight in August. And then we did an AI focused one. And every time we do it, we collect feedback on what people want. And people like to have the in -talk or the in -depth demos, but they also, you know, like to hear the panel discussions and people just talking about what they're building. So we've tried to create that variety.

So I can kind of give you some of like the high level agenda overview and and kind of what these these different talks are shaking out, shaking out as. So first, we've got Corey Quinn. He's running the keynote. If you're involved with Amazon Web Services, you know, AWS, you probably know Corey Quinn. I think I came across him first, maybe like 2020. And then I saw him do the AWS reinvent like vendor crawl.

And it was just like he was like dressed up as a zookeeper, like going across these different vendors. And it was just hilarious. So his talk is what he's labeling AI speaks Terraform like a tourist. Really, really excited for him to kind of kick it off. When we were talking about his keynote, you know, there's no right answer in any of this is what it comes down to, right?

Whether AI writing Terraform is going to help in some areas, whether going completely away from IC. In general for some of these things and having AI just directly provisioned infrastructure and everywhere in between, the keynote is meant to kind of set the stage of, hey, there's a lot of places that you can go to. And obviously it has a little bit of Corey's humor built into it. Awesome.

So from there, we've got Matt Gowie, who's a founder and CEO of a consulting company called MasterPoint. They're a partner of Spacelift. He's got a great perspective on the industry. He has a lot of clients. So he's talking about. Kind of how AI is upending infrastructure operations. We're running two tracks at different times of the day. We're starting off with two tracks after Corey's keynote.

And we've got two folks, Emin Elmedar and Flavius Danu, who are practitioners. They've worked at Spacelift, but they're going to talk about how to get started pretty easily with infrastructure as code with an...

Ai flavor and they're actually going to run into an open source project called intent about how you can actually stand up very quickly some ai provisioning both with iac and then you know an ai driven workflow path from there we're going to jump into anta babenko who if you all know tf weekly he's uh you know a big voice in the terraform infrastructure's code space and he does a lot with terraform modules that's what

He's known for producing these modules that everyone loves and goes to repeatedly so he's got a session specifically around replacing Terraform module forks with policy transformation rules.

We then jump into one of the first panel of the day. We've got two panels, and this one is really interesting. So it's going to be moderated by Luca, who is over at Platform Engineering, or if you know PlatformCon, Luca Galante is all things platform engineering. And we've got three folks on that panel. The first is Chris Haas, who is the CTO of Mondelez. So Mondelez is the massive food.

A manufacturer I think they own cad berries and all different sorts of uh confectionaries so he started as a director of platform engineering and he's made his way up and now he's the cto of the organization so great in -house expertise we've got an individual fasal who is the principal of platform engineering for a services firm called ahead so he's got a lot of experience going talking to all the services partners and his clients and then we've got Eric Maxwell, who's actually a co -author of DORA and works at Google.

So that panel is going to be really interesting perspectives from three different individuals all about platform engineering and what does it actually mean with now AI agents being consumers of your platform. So then we get into some more technical sessions. We run this on two tracks. We call the main stage and the builder stage.

On the main stage, we've got Amin talking about when 10x code velocity could mean 10x operational risks. So that idea of application. Code development is rapidly increasing, but what does it actually mean for your downstream operations? And then Davlet from Cloud Genie is going to be talking about how to safely deploy AI agents that write and manage your IAC.

On the build track, we've got a couple of interesting talks there. So one is this gentleman by the name of Joseph who... The title of this session is really interesting. I deleted 4 ,000 lines of AWS CDK and shipped faster. So he's going to get down into the details of a specific scenario he ran into. We actually simplified his AWS deployment, was able to go faster. So again, keep in pace.

And then Atesh Main from Philo is going to talk about infrastructure at scale using Argo CD. Kubernetes deployments, we're talking about what might be some future topics for IACConf. A lot of Kubernetes interest. Maybe not surprisingly, in a lot of session submissions and ones we've done in the past. So I think that's going to be a popular session. Around 2 .30 Eastern, we go into our second panel of the day.

And so this panel is actually a combination of different tech founders from a couple of different vendors in the DevOps platform engineering space. So we've got Marcin Wyszynski, who's the co -founder and head of R &D at Spacelift. We have Chris Evans, who's the field CTO and co -founder of Incident .io. And then we've got Ganesh Dutta, who is the CTO and co -founder of Cortex.

And what they're going to be talking about is... Yeah, we're seeing the, this is working for our customers. These are the people who are actually talking to people every day, building products that DevOps platform teams are using to try to keep that demand and how they're thinking about building those products.

I think those would be really interesting for maybe those platform engineering teams who are thinking about almost... Internal product development for their teams. What does that actually look like inside one of these vendors? This is also going to be moderated by a woman named Serena, who used to be the head of JP Morgan and Chase product developer platform from a UI UX standpoint.

So she's got some really interesting experience. She used to be at Red Hat on the product UI UX teams. That's going to be a very, I would say, product -centric panel thinking about how do you build this platform. For this 10x velocity and all the things that, again, they're hearing from customers, which I'm sure there'll be a lot of customers of any one or maybe a combination of those products in the conference.

Very cool. At three o 'clock, we're going to move over to, again, two tracks. So Dimitri Vlachos, who's the... CMO of Spacelift. This is our second year running the infrastructure automation report. And we surveyed 400 individuals across platform and DevOps engineering functions and asked them about their adoption of AI. How's it going?

What's the adoption rate of AI among their development teams versus their operations, specifically infrastructure operations? And what does that gap look like? There's some really interesting findings there. Drop a little bit of a hint.

Some of the biggest gaps are around the governance around those AI deployments and what does that actually mean in terms of productivity and time kept back of having to go back and, you know, check what's been done because of lack of guardrails that are in place. Alexander is going to run a session. This is very much a build session of something he's built around AI enforced architecture.

And then we're going to close out the day with John, who is a principal engineer at Sanofi, the large pharmaceutical company. And he's going to kind of round out the day about talking about. Platform engineering in general and how to do that without slowing developers down. Again, rounding out the theme around keeping pace and what does that mean in 2026? Interesting.

So I've seen like over the last year, IACConf has grown, especially with this next upcoming conference. And there's a lot of that just from feedback. Like you mentioned, we're doing two quote unquote stages now. There are panel discussions and what has caused like the changes and where do you see it going long term?

On the first one we did, which was in May of 2025, we were optimistic, but the response rate was more than we anticipated in terms of the registrations, the attendance. But then also after the event, we surveyed everybody. Hey, what did you like? What do you want to see more of? What suggestions do you have? And the response was overwhelmingly positive.

From the logistics standpoint, even people saying this is the type of event I would actually travel to because there was a... I mean, there still is a gap in the market in terms of something that's specific to infrastructure as code as a conference. You have your KubeCons, you've got, you know, the AWS Summits and the Reinvents, Google Next. Those are all great conferences, but they're quite broad, right?

And so this filled a gap for, you know, what is the niche of infrastructure as code? And I think with the content that we curated, it hit the mark. And so the feedback that we heard was... Well, one, keep it up. This is exactly what we want more of. People want stories. They want demos. They want real world scenarios.

I think there's always the balance of being able to have that kind of framework discussion of here's what good looks like versus like the actual real world scenario. Yes. And here's how I applied that in my actual environments. And here's how you can go on Monday.

Turn around and start doing it yourself so that's the biggest thing that we've tried to do every one of these events we run we go and survey the audience and say what did you like what did you not like what do you want to see more of and try to do that and we've also done that with trying to run these in person we've done two so far we did the first at cubecon when it was in atlanta in november and then we did a

Meetup in amsterdam back in march because The other half of it is people just want to connect, network, meet other people who are working with infrastructure as code and build their own network.

So I think that's the extension of what we want to make this more of is, yes, the primary is an events platform, allowing people to have a platform to share their story, share their knowledge, a place for people to come learn, but then also a platform for people to go connect with others who are working with infrastructure as code and whether that takes the shape of. The next place we've done was in -person events.

You know, from here, we want to build a content board, a jobs listing board. There was a lot of comments around, hey, we could have a Slack community or some sort of community where people can go and have real -life chat.

And so that's the objective is to keep building it, allow people to connect, still have the events platforms where people can share their stories and learn from each other, but create more of those forums, whether they're in -person or virtual, for people to actually connect with each other.

Do you see the cadence keeping up with that pace as far as like what one or two virtuals per year and then one or two in -persons or? Yeah. So we did Amsterdam. So for in -persons, we did Amsterdam. We're going to do Salt Lake City, which is, I think it's November. A lot of demand for doing something at reInvent.

Vegas is a bit crazy at reInvent, but it would be great to try to put something together there, even if it's a networking meetup style or a happy hour style. We have discussed some ideas of doing some more regional city -based events throughout the year, perhaps partnering with a local DevOps meetup or AWS has a lot of user groups. So I'd love to explore that. I'm in Boston.

Talking about trying to do that shortly after IACConf 2026. In terms of the virtual cadence, so yeah, we'll definitely keep with what we call Spotlight Series. So we have this annual event, 13 sessions. It runs from 11 to 4:30 Eastern. But then we did two of these spotlights last year. So one was security. So this intersection of infrastructure as code and security, that was three sessions, and that was August.

And then we did one on AI in January. Again, three sessions. And those went really well. So I definitely think there's more of those we can do. I mentioned Kubernetes being a really hot topic. Definitely think we can do a session based off of the submissions we've got and the interest we have on Kubernetes and that intersection of infrastructure and Kubernetes. Open source is another topic that's come up a lot.

And then there's, again, there's a lot of like those intersections we could have of infrastructure and going deep in one specific area. And then we've also talked about, we get a lot of submissions. I think we got 50 submissions for IAC Conf 2026, and we can only take so many. I'd love to be able to do one -off events. Hey, this is a submission we didn't get to.

We're going to run it on a Thursday in June and just give somebody that platform to share their story. You know, it helps them build their credibility. It's something they can add to their resume. They can add to their LinkedIn profile. This is a presentation I've done, but it also gives the community another.

Story they can go and listen to and think about how they can take that to their environment yeah for sure and so it is a free conference to be clear too so it's free yeah is it recorded so like let's say I couldn't attend can I see those talks after the fact yeah yep so we have a youtube channel so we post everything there typically within a week we'll get things posted up so yeah if you If you miss it, you can go catch it on YouTube again a week, 10 days later.

If you're also not sure you can attend, register because as soon as we do send an email out a few hours after the event, and then you can go access the events platform we use and you can go browse any of the sessions before they get to YouTube. So yeah, even if you're not sure you can attend, register and you can go access the platform immediately afterwards.

So given all the submissions that you have gotten, that maybe you weren't able to get to this time around. Have you seen any trend? Like, is there any industry trends around AI or around Kubernetes specifically that you've seen with these submissions? I have to go back and look a little bit.

I mean, so it was on this IACConf 2026, there was a pretty widespread of submissions around, again, this idea of keeping pace and what does that mean? We had some that were more keeping pace from the application side. And ways that they're thinking about the application development.

And I think there's ways to actually create some, there could be some interesting discussions around bringing people who are more focused on the application side and then someone else on the operations side. And maybe it's a team who's worked together to figure that out really well. On the Kubernetes, well, one, Kubernetes is such a broad topic.

I think people were, last year, Crossplane was one of the most, I would say, sought -after sessions we had at IACConf. Now, if you actually looked at the chatter and the questions that were coming up in that, and actually even the presentation itself, no one was super confident on Crossplane. Now, it doesn't mean, but people are still interested in how can I make Kubernetes work for...

My infrastructure and what does that intersection look like? So the demand is certainly there. Obviously, Kubernetes is not going away. We've got conferences dedicated to it. So there's plenty of people out there looking to learn and explore and see what's possible. I think my overarching, I guess, overarching takeaway in looking at the submissions is there's no right answer. People are still figuring this out.

There's a lot of experimenting. I think what's going to be really interesting from this is where the conversation kind of...

Centers around because the chat becomes really interesting during these sessions where people a speaker says something and then it seems to spark a conversation in the chat and you can like almost you know see through the screen people's like light bulbs going off like yes I get that that makes sense and there were certain sessions last year that triggered that and given that this year's topic is a bit more focused on this keeping pace and well just the environment May 2025 to May 2026.

The world of software engineering and what that means for operations has changed dramatically, right? So I think it'll be really interesting to see kind of what those spark moments are in some of these talks, because that will be a good indication for us. And we'll definitely publish it once we do that, because we did this last year.

We can see all the data on the back end, like what those topics were and what sparks those, what are the questions asked? So we get to actually kind of get some sort of an idea of what the themes are. We'll do a write -up and we'll definitely publish that once we have it, because I think. It'll be interesting for a lot of folks to read that.

So like a report around like the analytics, I guess, of what you're able to glean from the conference itself, like from the participants? Yeah, like what was the most attended session? Like what was the highest interest rates of the content? Yeah. Yeah. That sounds interesting. I'd be curious to read it for sure.

I mean, reading like the DORA report every year, it's very interesting to see just the state of the industry, where it's headed, especially over the last few years. As you've said already, AI has kind of changed everything for everybody. Yeah. There's been a real shift. Did you have Crossplane talks previously? Or you just had submissions around Crossplane? We had a Crossplane talk in IACConf 2025. Okay.

And how does that jive with Spacelift, your IC platform? Yeah. It's kind of like promoting a separate product in a way? I don't know. Yeah, yeah. So I'm on the Spacelift team. I'm employed by Spacelift. I'm on the marketing team. But we have been very deliberate in IACConf. Spacelift funds it, right? But we do honestly try to keep it separate. So that means we open up the CFPs.

We want to get people in talking about things. What are the popular topics? Crossplane was a popular topic. It was a great submission. And so... For the better of the community, that's what people want to hear about. It doesn't really matter if that, you know, is a competitor of Spacelift. In the end, it's not really a competitor of Spacelift.

You know, I think it's more of a, maybe a cultural difference of the team adopting it, right? If I want to go all in on Kubernetes versus I want to go all in on traditional, you know, terraforming GitOps approach, well, that's a separation once you make that decision. That's a bit of a competing, maybe philosophical, is that the right word? I don't know. Methodology of how you actually want to deploy infrastructure.

But even if it wasn't that, you know, GitOps Terraform space, again, if people, the community wants to hear about certain topics, that's what we want to create IOC Conf for. I've even thought about, it would be really interesting to have like a tacos panel, like get some of the founders from all these different platforms and have them talk about what they think the future is of, you know.

Terraform orchestration platforms. The IACConf is just the platform to make those conversations happen. It's great to hear. I mean, so I've been to a lot of like vendor sponsored conferences and sometimes they lean too heavily in marketing. It's nice to hear that IACConf is more interested in relaying the technologies that matter to the practitioners that are attending over trying to. Sponsor Spacelift as a product.

I mean, that's not the only motivating factor. Like I said, I don't want to name names, but I've been to other vendor -led conferences that were very heavy into that vendor ecosystem at all costs, which can be detrimental to the overall conference itself. So it's good to hear. Yeah, we have our own set of, you know, Spacelift events and sessions where we can talk Spacelift, but that's not what IACConf is for.

It's a platform for the community and, you know, Spacelift just helps fund it. So if there was, I had time to watch one talk, is there one or top three talks that you would say ahead of time that I should maybe check out? Let's say I was very busy. I just didn't have time. I don't want to put you on the spot. If you don't, if you don't want to name one, that's fine too.

But yeah, well, I haven't, all the, all like the first round drafts of presentations are due in like later this week. So I actually haven't seen anything yet, but I mean, there's a few talks that I think are going to be interesting. I don't know if I can like give you an order, but well, like Corey.

Again, if you've seen any of his videos, his sense of humor and his approach just talking about AWS and all things infrastructure is always entertaining. So I'm pretty excited about that. I'm really excited about this gentleman, Amin, whose session is when 10x code velocity could mean 10x operational risk. If you were to put a session, not so much focused on infrastructure, but on this keeping pace topic.

That's exactly what his session is about. And what does that actually mean for all the downstream operations teams? So I think that one will be really interesting. The panel in the agenda is the Platform Engineering Experts panel with Luca, Eric Maxwell from DORA, Frisal from AHEAD, and Chris from Mondelēz. I think that's going to be really interesting because you've got...

Three very different but aligned perspectives. You've got someone who's in -house, started as a director of platform engineering, now the CTO. You've got someone at a major services partner who's working with a lot of clients. And you've got someone at Google who co -authored Dora. So I think there's just going to be a lot of really interesting perspectives on that.

And then the one that I think is, you know, his background looks really interesting. The submission also looks really interesting is Alexandra from Anovo, AI -enforced architecture fitness functions at scale. On the builder's track, a bit more technical, a bit more down in the weeds. That one's at just around three o 'clock Eastern. I think that'll be a really interesting session that'll get some good engagement.

Very cool. So for anyone looking to register, where should they, how do they check you out? Yeah. So IACconf .com is the main website. We've got, there's an event section and you can see within there, it's the featured event. You can also find us on LinkedIn. We've got an IACconf profile page. We're releasing every few days, featuring a new speaker.

And so those are probably the two best places to go get some more information. Awesome. Great to hear it. Anything else you'd like to leave our listeners with, Gareth? Yeah, I'd say the reason that... IAC Conf has grown to what it has is by listening to feedback from the community.

So if infrastructure as code is of interest to you, you're either trying to learn it, use it daily, or you lead IEC initiatives at your organization, register, get on the list, attend, and let us know what you think. Again, we're building this community. For this audience. And so the best way to help us build is to let us know what you like, what you don't like, when you want to hear more of.

So this is a really exciting project. We want to do more of these events. And so, yeah, please join us on May 14th. I'm looking forward to it personally. Everyone else, please check it out as well. Gareth, thank you so much for coming on. Really appreciate your time. Thanks, Brian. All right. That was my conversation with Gareth Kersey previewing IACConf 2026.

My biggest takeaway from this one is that keeping pace sounds simple until you actually unpack it. Because for infrastructure and platform teams, keeping pace does not just mean moving faster. It means absorbing faster change without turning your platform into a junk drawer. It means giving developers better paths to production without giving up governance.

It means figuring out where AI helps, where it creates risk, and where human judgment still needs to sit in the loop. And honestly, that feels like the real infrastructure conversation right now. Not “AI is amazing.” Not “AI is useless.” More like, okay, the code is coming faster now. The pull requests are coming faster. The experiments are coming faster. The platform requests are coming faster.

So what has to be true for infrastructure teams to handle that without becoming the bottleneck or the cleanup crew? That is where infrastructure as code still matters. That is where policy matters. That is where reusable patterns matter. That is where platform engineering matters.

And that is where events like IACConf are interesting because they are bringing together people who are actually dealing with this stuff from different angles. Practitioners, platform teams, vendors, consultants, open source folks, people deep in Terraform, Kubernetes, GitOps, governance, and AI -driven workflows. I also appreciated Gareth being pretty clear that IACConf is trying to be a community event.

Not just a vendor event with a nicer logo. Spacelift helps fund it, but the topics are broader than Spacelift. They have had Crossplane talks. They are talking Kubernetes. They are talking Argo CD. They are talking platform engineering, DORA, incident management, governance, and the bigger operational impact of AI speeding up software delivery.

That matters because practitioners can smell a thinly disguised product pitch. From a mile away. And this seems more like a useful place for infrastructure people to compare notes on where the work is actually going. IACConf 2026 is free, virtual, and happening on May 14th.

Gareth mentioned that even if you cannot attend live, it is still worth registering because the sessions will be available afterward through the event platform and later on YouTube. I'll put the link in the show notes. If you enjoyed this episode, follow Ship It Weekly wherever you listen to podcasts. If you want the show notes and links from this conversation, head over to shipitweekly.fm. Thanks for listening, and I'll see you later this week.

Developer tooling used to feel like the safe part of the system. Production was the scary place. The database, the cluster, the load balancer, the pager. And then everything around that was just tooling. GitHub, CICD, merge queues, package publishing, code review bots, AI agents. But that line is getting harder to defend.

Because when the tooling can merge code, publish artifacts, leak secrets, spend money, or decide what gets shipped, it is not just sitting beside production anymore. It is part of production. And this week, that idea showed up everywhere. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech.

This is Ship It Weekly, where I filter the noise and focus on what actually changes how we run infrastructure and own reliability. Show notes and links are on shipitweekly .fm. If this show's been useful, follow it wherever you listen. Ratings help way more than they should. And if you are watching on YouTube, subscribe there too.

We have six main stories today, then the lightning round, and we'll wrap with the human closer. We're starting with GitHub's critical Git Push remote code execution vulnerability, including the AI reverse engineering angle from Wiz. Then we'll talk about prompt injection against AI agents tied into GitHub workflows.

After that, Elementary's CLI supply chain incident, where a GitHub actions workflow became the path to a malicious release. Then GitHub's April 23rd Merge Queue regression. Then cal .com going closed source, which is probably the most argument -starting story in this set.

And finally, GitHub Copilot moving to usage -based billing because AI tooling is becoming a governance and cost management problem, not just a productivity experiment. Then in the lightning, we'll hit MinIO, Ghostty, Docker -hardened images, and Azure DevOps security updates. Story one. GitHub's GitPush RCE is a toolchain wake -up call. Let's start with GitHub.

GitHub disclosed a remote code execution vulnerability in the GitPush pipeline. The issue affected GitHub .com, GitHub Enterprise Cloud, GitHub Enterprise Cloud with Data Residency, Enterprise Managed Users, and GitHub Enterprise Server. GitHub says Wiz reported the bug through the bug bounty program on March 4th.

GitHub reproduced it internally within about 40 minutes, fixed GitHub .com in under two hours, and says it found no evidence of exploitation. That is the reassuring part. The uncomfortable part is the shape of the bug. The vulnerability involved user -supplied Git push options. Those push options ended up influencing internal metadata passed between GitHub services.

Because of how that metadata was formatted, an attacker could inject extra fields that a downstream service treated as trusted internal data. And once user input can shape trusted internal metadata, things get ugly fast.

GitHub says the researchers showed they could override the environment where the push was processed, bypassing sandboxing around hook execution, and ultimately execute arbitrary commands on the server handling the push. That is a lot hiding behind the phrase Git push. And that is why this belongs in a DevOps and SRE conversation, not just a security newsletter. Every team treats Git as normal plumbing.

Humans push code. Bots push code. Automation responds to pushes. CI kicks off. Release workflows start. Security scanners run. Deployment paths wake up. But behind that one command is a hosting platform. Internal service boundaries. Hook execution. Sandboxing. Audit logs. And identity. That is infrastructure.

For GitHub .com and enterprise cloud users, GitHub says the platform was patched and no customer action is needed. For GitHub enterprise server customers, this is a patch now situation. GitHub specifically recommends upgrading supported G -H -E -S releases and reviewing audit logs for suspicious push operations. But the bigger story may be how Wiz found it.

Dark reading covered the AI -assisted reverse engineering side. Wiz used AI tooling to help analyze GitHub Enterprise Server, understand compiled binaries and internal service behavior, and find a bug that historically may have taken weeks or months to uncover. They reportedly went from the idea to working exploit in under 48 hours. That is the part that should stick with us.

Not because AI found a bug, like the robot woke up and became a hacker. The real story is that AI lowered the cost of understanding complex systems. A lot of software has been protected informally by being annoying. Hard to reverse engineer. Hard to understand. Too much glue. Too many binaries. Too many internal assumptions. That was never real security, but it was friction. And friction matters.

If AI removes enough of that friction, then systems that were probably fine because nobody will spend the time become more interesting targets. So the takeaway is not panic. It is that boring security engineering matters more when discovery gets faster. Input validation. Trust boundaries. Least privilege. Patch velocity. Audibility. Because when the cost of finding bugs drop, the cost of sloppy assumptions go up.

Story 2. Prompt injection is now in the CI -CD blast radius. Next up, AI agents and GitHub actions. The register covered research showing prompt injection attacks against AI agents from Anthropic, Google, and Microsoft tied to GitHub workflows. The basic shape is pretty simple.

AI agents read GitHub data, pull request titles, issue bodies, issue comments, review comments, markdown, the stuff developers and contributors write all day. So if an attacker can place malicious instructions into that data and an agent later processes it with access to tools or secrets, the attacker may be able to steer the agent.

In the Claude case, researchers showed that a malicious PR title could influence the bot and leak credentials into a review comment. With Gemini, issue comments and prompt injection were used to expose an API key. With GitHub Copilot Agent, malicious instructions could be hidden in an HTML comment that humans would not see in rendered markdown.

The victim assigns the issue to Copilot, but the payload is already sitting there in the context. The researchers demonstrated theft of Anthropic and Gemini API keys, multiple GitHub tokens, and potentially any secret exposed to the GitHub Actions Runner environment. That last phrase is the one I care about. The GitHub Actions Runner environment. Because this is not just an AI story. This is a CICD story.

If your agent runs in GitHub Actions, and GitHub Actions has access to tokens, deployment credentials, package publishing rights, cloud permissions, Slack webhooks, or JIRA credentials, then prompt injection becomes part of that blast radius. The researcher called this comment and control, which is a play on command and control. And it fits. The attacker may not need malware. They may not need an external callback.

They may not need to trick a human into clicking anything. They can write data into GitHub, let automation read it, and let the automation leak the secret back into GitHub. That is a nasty little loop. And this is where I think teams are a little too cautious about agents. A code review bot should not have more access than it needs. An issue summarizer should not have repository write access.

A documentation bot does not need cloud keys. And if you are passing untrusted GitHub content into an agent that can use tools, run commands, or read secrets, that is not a harmless productivity flow. That is untrusted input hitting an execution environment. We have had that category of problem for decades. AI did not invent it. AI just made the parser weird. Story 3.

Elementary's CLI incident is the supply chain story in miniature. Now let's talk about elementary. Elementary Data published a security incident report for a malicious release of the elementary open source Python CLI version 0 .23 .3. On April 24th, a malicious CLI package was published to PyPI, and a malicious Docker image was published to their registry. Those artifacts were not produced by the elementary team.

According to elementary, the attacker opened a PR with malicious code and exploited a script injection vulnerability in one of their GitHub actions workflows to publish the release. That is the modern supply chain anxiety loop in one story. A pull request. A workflow vulnerability.

A release -looking artifact, PyPI, Docker, and downstream users who now have to assume any credentials accessible to the environment where that CLI ran may have been exposed. Elementary removed the malicious release, published a safe 0 .23 .4, removed the vulnerable workflow, rotated affected credentials, and moved the OIDC authentication where possible.

That response sounds pretty solid, but the attack path is the real lesson. A lot of teams still think of GitHub Actions as a convenience layer. But for many projects, GitHub Actions is the release authority. It builds containers. It publishes packages. It signs things. It pushes to registries. It decides what gets shipped.

So if that workflow has script injection, broad permissions, long -lived tokens, or too much trust in PR -controlled input, then your release system can become the attack path. The question is not only do you trust your source code? It is also do you trust the automation that turns source code into artifacts? Those are not the same question anymore. You can have a clean main branch and a vulnerable release workflow.

You can have signed releases and a compromised signing path. You can rotate app credentials and still forget that the package publisher token has been sitting in CI forever. So the practical playbook is boring but important. Check workflow permissions. Avoid long -lived publishing tokens where OIDC is available.

Be careful with shell interpolation from PR titles, branch names, issue comments, tags, and commit messages. Split, build, and publish. Protect release environments. And threat model GitHub Actions like production. Because if GitHub Actions can publish your package, it is production. It might not serve customer traffic, but it controls customer trust.

Story four, GitHub's merge queue regression is a control plane reliability story. Next one is also GitHub, but from a reliability angle. On April 23rd, GitHub had a merge queue regression. GitHub says whole requests merged through merge queue using squash merge produced incorrect merge commits when a merge group contained more than one pull request.

In some affected cases, changes from previously merged PRs and prior commits were accidentally reverted by later merges. GitHub says that 658 repositories and 2 ,092 pull requests were affected. No data was lost because the commits still existed in Git. But affected default branches ended up in an incorrect state. And GitHub could not safely repair every repository automatically. That is a painful incident.

And it is painful because merge queues exist to reduce risk. That is the whole point. They are supposed to make high -change environments safer. Instead of everyone racing to merge against a moving base branch, the queue group changes, validates them, and lands them in a more orderly way. So when that control plane gets a bad behavior, it creates a weird kind of incident. Not the site is down.

Not the database is corrupted. But the mechanism we trust to preserve correctness just produced a branched state that is not what we think it is. That is subtle. And subtle is dangerous. Because if a deploy fails loudly, you know. If an API goes red, you know. If a merge queue quietly reverts earlier changes, you might not notice until the wrong code is already in production.

The practical takeaway is not don't use merge queues. I still like merge queues. The takeaway is that any environment with merge authority needs SRE thinking. Good audit trails. Easy ways to identify what changed. Detection when a branch state does something weird. Clear ownership when the thing that protects main becomes the thing that damages main.

Merge queues, feature flag systems, CI schedulers, artifact registries, deployment orchestrators, GitOps controllers. These are control planes. They may not be in the request path, but they shape what production becomes. Story five. Cal .com going closed source is the argument everyone will want to have. Now for the spicy one. Cal .com announced that it is going closed source.

They said that after five years as open source champions, they are moving the production code base closed because AI has changed the security landscape. Their argument is basically this. In the past, exploiting an application required skilled humans. Time, and patience. Now AI can be pointed at an open -source codebase and used to systematically scan for vulnerabilities.

Cal .com says that being open -source is increasingly like giving attackers the blueprints to the vault. They are keeping an open version available as cal .diy under the MIT license. But they say that the production Cal .com codebase has diverged, including rewrites in areas like authentication and data handling. I have mixed feelings on this one. On one hand, I understand the concern.

This week's GitHub and Wiz story gives some support to the idea that AI can make vulnerability discovery faster and cheaper. If you run a SaaS product handling sensitive calendar data, authentication flows, integrations, and customer information, it is not crazy to worry about AI -assisted vulnerability mining. On the other hand, closed source equals safer is not a clean argument.

Closed source software still gets reversed. Closed source software still has bugs. Closed source software can lose the benefit of external review. And hiding code is not the same as securing it. So I would not frame this as cal .com is right or cal .com is wrong. The better framing is that AI is putting real pressure on the business and security model of commercial open source.

For maintainers, open source can mean transparency, community, contributions, and trust. For customers, it can mean self -hosting, auditability, and less vendor lock -in. For attackers, it can mean a searchable map of the systems. And for a commercial company, it may mean all of those things at once, plus support burden, vulnerability reports, hosted customer risk, and investor pressure. That tension is not new.

AI just makes it louder. For platform teams, the takeaway is not stop using open source. That would be ridiculous. The takeaway is stop treating project governance and licensing as background noise. When a project changes its source model, repo model, release model, or commercial posture, that is supply chain signal. You do not have to panic, but you should notice.

Story six, co -pilot usage -based billing makes AI a governance problem. Last main story. GitHub Copilot is moving to usage -based billing on June 1st. GitHub says all Copilot plans will transition to GitHub AI credits. Instead of counting premium requests, plans will include a monthly credit allotment, and usage will be calculated based on token consumption, including input, output, and cached tokens.

Base subscription prices are not changing, according to GitHub. Code completions and next edit suggestions stay included. But agentic usage, longer sessions, heavier models, and multi -step workflows move more directly into the cost model. And one detail jumped out at me. Copilot code review will also consume GitHub Actions minutes in addition to GitHub AI credits.

That is what makes this a platform engineering story. Because now AI usage is not just a seat cost. It is tokens, model choice, cached context, review automation. Actions minutes, budget controls, and admin visibility. And this is the normal cloud story happening again. First, it feels magical. Then everyone adopts it. Then the bill shows up. Then leadership asks, why is it growing?

Then teams discover they do not have tagging, budgets, ownership, quotas, or usage visibility. Then platform teams get asked to govern it after it is already everywhere. We saw this with cloud. We saw it with observability. We saw it with CI minutes. Now we are going to see it with AI -assisted development. I am not anti -copilot. I use AI tools constantly. But the economics are changing.

A quick autocomplete suggestion and a multi -hour agentic coding session are not the same cost profile. A lightweight code explanation and a repo -wide refactor are not the same cost profile. A human asking a few questions and an agent iterating across a codebase are not the same cost profile. GitHub is saying that more directly now. And teams should probably take the hint.

If AI behaves like cloud spend, you need cloud spend habits. Visibility. Ownership. Guardrails. Budgets. And somebody who notices before the bill becomes the incident. A few quick ones before we wrap. MinIO's main GitHub repo was archived by the owner on April 25th and is now read -only. The readme says the repository is no longer maintained and points users towards other options.

It also says that the community edition is now source -only and that legacy pre -compiled binaries will not receive updates. That is a big dependency signal if you have MinIO in your stack. Ghostty is leaving GitHub. Mitchell Hashimoto wrote that Ghostty plans to move off of GitHub while keeping a read -only mirror at the current URL. The interesting part is his framing. He says the problem is not Git itself.

The problem is the surrounding infrastructure that people rely on. Issues, pull requests, actions, and platform workflows. Git may be distributed, but your actual engineering workflow usually is not. Docker published a one -year reflection on Docker -hardened images. They say Docker -hardened images crossed 500 ,000 daily pulls with more than 2 ,000 hardened images, MCP servers, Helm charts, and ELS images.

That is a nice counterweight to the rest of the episode. Not every supply chain story is a compromise. Some of the work is making safer defaults easier to adopt. Azure DevOps added one -click CodeQL setup and org -wide alert triage to advance security. Microsoft says teams can enable code scanning at the repo, project, or organization level without manually configuring a pipeline.

And security admins get a combined alerts view across repositories. Not super flashy, but useful when security has to scale beyond fixing repo number 417 by hand. I think the human thread this week is that engineers are still emotionally treating developer tooling like it lives outside the blast radius. And it does not. The build system is not outside the blast radius. The merge queue is not outside the blast radius.

The package publisher is not outside the blast radius. The AI review bot is not outside the blast radius. GitHub Actions is not outside the blast radius. Copilot billing is not outside the governance conversation. The open source project that you depend on is not outside your supply chain just because you starred the repo five years ago. All of this is part of production now.

Maybe not production in the narrow sense, maybe not serving customer traffic at this exact second, but production in the operational sense. If this breaks, we cannot ship. If this is compromised, customers may be affected. If this gets expensive, leadership will care. If this silently does the wrong thing, production may become wrong later.

If this changes its licensing or maintenance model, our roadmap may get weird. That is production. And I think that that shift is hard because developer tooling used to feel like the comfortable part of engineering. It was where we moved fast, where we automated things, where we glued things together with tokens and YAML and a little bit of we'll clean this up later.

But the shortcut became the path, and the path became the control plane. And the control plane now has secrets, merge authority, release authority, billing impact, AI behavior, and platform dependency risk. That does not mean every team needs to panic. It means we need clearer labels. A GitHub Actions workflow that publishes to PyPI is not just CI. It is a release system.

A merge queue that lands code on main is not just a productivity feature. It is a source control control plane. An AI agent with repo access and secrets is not just a bot. It is a principal with tools. An open source dependency whose repo gets archived is not something to look at later. It is a dependency management event. And a co -pilot billing change is not just pricing. It is a governance signal.

Once you label the system correctly, you can manage it correctly. You can put the right permissions around it. You can monitor it. You can budget for it. You can patch it. You can separate trusted and untrusted input. You can stop pretending that the thing with keys to production is somehow not production. Reliability is not just keeping servers up. It is keeping the paths to production trustworthy.

And this week, those paths looked a little more serious than usual. Alright, that's it for this week of Ship It Weekly. Quick recap. GitHub patched a critical Git push remote code execution vulnerability. And the AI -assisted reverse engineering angle may be just as important as the bug itself.

Researchers showed prompt injection attacks against AI agents tied into GitHub workflows, which is a pretty clear reminder that agents are now in this CI -CD blast radius. Elementary Data dealt with a malicious CLI release published through a vulnerable GitHub Actions workflow. GitHub had a merge queue regression that affected more than 600 repositories and more than 2 ,000 pull requests.

Cal .com is going closed source and putting a spotlight on the tension between AI, security, open source, and business models. And GitHub Copilot is moving towards usage -based billing, which means AI tooling is becoming a cost and governance conversation, not just a productivity conversation.

Then in the lightning round, we hit MinIO archiving its main repository, Ghostty leaving GitHub, Docker hardening images crossing 500 ,000 daily pulls, and Azure DevOps making CodeQL setup and org -wide alert triage easier. Links and show notes are on shipitweekly .fm. You can also find the video version on YouTube. If this episode was useful, send it to the person on your team who still says it's just CI. I'm Brian, and I'll see you next week.

You know that moment when a platform stops sounding helpful and starts sounding serious? That's this week. Kubernetes is cleaning house. Gateway API keeps getting more official. AWS is moving on from Copilot. Airbnb is showing why production should not be your first alert test. And Cloudflare is reminding everybody that a bot with a token is still a principal with blast radius. Hey, I'm Brian Teller.

I work in DevOps and SRE and I run Teller's Tech. This is Ship It Weekly, where I filter the noise and focus on what actually changes how we run infrastructure and own reliability. Show notes and links are on shipitweekly .fm. If the show's been useful, follow it wherever you listen. Ratings help way more than they should.

We have five main stories today, then the lightning round, and we'll wrap with the human closer. We're starting with Kubernetes 1 .36 because this feels like one of those releases where the project keeps sanding off older sharp edges while pushing more production grade features into stable territory. Then Gateway API version 1 .5, which is basically SIG network saying the future is not just coming.

It is getting promoted into the stable path now. After that, AWS Copilot CLI end of support because this is a very real platform story. One easy path is aging out and AWS is pretty clearly nudging people towards the next preferred path. Then we've got Airbnb on alert development, which is probably my favorite SRE story in the set because it is really about how better feedback loops beat blaming culture.

And finally, Cloudflare on non -human identities. Because this is one of the clearest examples lately of security vendors saying out loud that scripts, agents, and third -party tools need to be treated like first -class identities, not side characters. Story 1. Kubernetes 1 .36 feels like a maturity release. Let's start there. Kubernetes 1 .36 shipped on April 22nd.

The release includes 70 enhancements with 18 graduating to stable, 25 entering beta, and 25 moving to alpha. That alone gives you the usual big release energy, but the more interesting part is what it says about where the project is spending its maturity budget. One of the clearest examples is service .spec. External IPs.

Kubernetes says that field is now deprecated, calls it as a known security headache, and points back to the long -running man -in -the -middle risk around CVE -2020 -8554. You'll see warnings now, and full removal is planned for version 1 .43.

Kubernetes is also permanently disabling the old Git repo volume plugin in 1 .36, saying that path is closed for good and existing workloads need to move to alternates like init containers or external Git sync style tools. That's why I like this story, because this is not just look at all of the new stuff.

It is Kubernetes continuing to act like a platform that wants fewer half -legacy foot guns hanging around forever. And honestly, that is what maturity often looks like. Not more knobs, fewer weird things everybody knows are sketchy, but keeps tolerating anyway. And there's another angle here that makes this release feel more grown up than flashy.

Kubernetes 1 .36 is also pushing more of the supply chain and packaging story towards cleaner primitives. The release highlights support for packaging read -only application data. Models, and static assets as OCI artifacts and delivering them to pods through the same registries and versioning workflows teams already use for container images.

It also calls out staleness migration work for controllers, which matters because a lot of weird controller behavior does not show up as a dramatic crash. It shows up as a controller acting on stale assumptions at the wrong time and doing the wrong thing very confidently. So the practical read for me is this.

If you run Kubernetes at any real scale, 1 .36 is the kind of release where you should not just skim the release nodes for what's new. You should skim them for what old thing are they finally done tolerating? And what newer path is now stable enough that we should stop calling it experimental in our own heads? Because that's usually where the real platform signal is.

Story 2, Gateway API version 1 .5, is the networking future getting more real. Next up, Gateway API. SIG Network announced Gateway API version 1 .5 on April 21st, called it the biggest release yet, and said the focus was moving existing experimental features into the standard, meaning stable, channel. The release promotes six features that people actually care about.

Listener set, TLS route, HTTP route, CORS Filter, Client Certificate Validation, Certificate Selection for Gateway TLS Origination, and Reference Grant. They also moved to a release train model, which basically means features ship when they are ready at freeze time instead of waiting for some perfect bundle.

That matters because Gateway API is no longer just a nice future -facing idea for people who like cleaner abstractions. It keeps becoming the actual road forward. Especially now that Kubernetes itself is pointing people towards gateway API in places where older patterns are being deprecated. And especially after all of the broader ingress and controller retirement pressure we've already been seeing this year.

So to me, the real read here is simple. The networking control plane in Kubernetes keeps getting more explicit, more standardized, and less willing to leave everything. In the controller specific magic plus annotations bucket forever. And some of the specific gateway API promotions are worth slowing down on for a second.

Listener set is interesting because it gives teams a cleaner way to contribute listeners to a gateway without forcing everything into one giant resource owned by one team. TLS route going stable matters because it makes the TLS pass through and terminate use cases feel more first class.

The CORS filter moving into the standard channel is also one of those small looking things that matters a lot in real life because it is exactly the kind of behavior people used to bury in controller specific config or app side workarounds. And the move to a release train model is its own kind of maturity signal too. It means the project is optimizing less for perfect bundling and more for predictable forward motion.

So if I'm a platform team, the takeaway is not just cool, more gateway API stuff. It is probably how much of our ingress and edge behavior is still trapped in annotations. How much of it is controller specific and how much of it now has a cleaner upstream shaped home we should actually plan forward. Story three, AWS Copilot is reaching end of support. And that tells you a lot. Now to AWS.

AWS says Copilot CLI reaches end of support on June 12th, 2026. It will remain available as an open source project on GitHub. But AWS says it will no longer receive new features or security updates from AWS. In the same announcement, AWS points users towards Amazon ECS Express Mode and AWS CDK Layer 3 constructs as the migration paths they want people evaluating.

This is one of those stories I like because it is not really about the tool alone. It is about the platform preference. Copilot was AWS's opinionated developer -friendly path for developing containerized apps on ECS and AppRunner. Now AWS is pretty clearly saying the newer opinionated paths are somewhere else. That does not mean Copilot users did anything wrong. It just means the center of gravity moved.

And that is a very real cloud platform lesson. Sometimes the easy path you picked was genuinely the right call at the time. Then the provider evolves. The preferred abstractions change. And now your job is not debating whether the shift is fair. Your job is planning the migration before the old path becomes operational debt with a calendar attached to it.

And there is a migration planning lesson here that I think gets missed when people hear end of support. Copilot did a lot more than just offer a nicer CLI. AWS says it used CloudFormation stacks for the app and service layers, which means a lot of teams probably have more Copilot -shaped infrastructure under the hood than they remember. So the right response here is not panic. It is inventory.

What workloads are still on Copilot? Which ones are app runner versus ECS? Which parts of the deployment flow are tied to Copilot convention? And whether the real destination should be express mode for simplicity or CDK L3 constructs for teams that want stronger IAC controls and customization. AWS is pretty explicit.

That Express Mode is meant to preserve a lot of the simplicity that made Copilot attractive, while CDK L3 is the more customizable path. That is the kind of thing I would actually say out loud to a team. Do not turn this into a philosophical debate about whether Copilot was good. Assume it was good for when you picked it. Now ask, what will be annoying to unwind later if you ignore the date now? Story 4.

Airbnb says alert pain was a workflow problem, not a culture problem. This one is probably my favorite in the whole episode. Airbnb says the issue with alert development was not that engineers did not care. It was that their observability as code workflow had a blind spot. Code review could validate syntax and logic, but not actual alert behavior against real -world data.

So production kept becoming the proving ground. Airbnb says they built fast feedback loops to preview, validate, and surface alert behavior before PR submission, cut development cycles from weeks to minutes, and used the workflow to help migrate 300 ,000 alerts from a vendor to Prometheus. They also say what used to take a month of iteration can now take an afternoon. This is such a good SRE story.

Because it is very easy to look at noisy alerts, weak alerts, or slow alert iteration and say, this is just a culture problem, or people just need to care more. Sometimes, sure. But a lot of the time, the workflow is just bad. If the only way to see whether an alert behaves correctly is to merge it and wait, then you built a system where production is the testing harness and on -call is the feedback loop.

That is not a motivation issue. That is a tooling issue. And Airbnb's fix is basically the kind of thing platform teams should love. Earlier feedback, more confidence inside the PR, less wasted iteration after the fact. And Airbnb made another design choice here that I really liked. They explicitly chose compatibility over novelty.

Instead of inventing some exotic, proprietary, alert analyst model, they took Prometheus role groups as the input. Used Prometheus's own rule evaluation engine, and wrote the results back out as Prometheus time series blocks, exposed through the standard query API. That is a really smart platform move because it means the preview and analysis system fits into the workflows engineers already understand.

Instead of becoming one more internal snowflake, everybody has to relearn. That matters because a lot of internal platform tooling dies, not because the idea was bad. But because the workflow becomes learn our special thing first. Airbnb's version sounds more like use the standards, use the real engine, show people the delta before they merge, and make the right thing easier than the lazy thing.

That is honestly a great pattern well beyond alerting. Story five. Cloudflare is saying non -human identity is now the real identity story. Last main story. Last time in episode 34, Cloudflare was talking about the network fabric for agents. This time, they're talking about the identity, token, and permission model around them. Cloudflare's framing here is very direct. Identities are not just people anymore.

They are agents, scripts, and third -party tools acting on your behalf. Their update packages that into three practical areas. Scannable API tokens, better OAuth visibility and revocation, and more granular resource -scoped RBAC. Cloudflare says new tokens are easier for scanners to recognize. Customers now get a central connected applications experience for OAuth access and revocation.

And resource -scoped permissions are available for more resources so both users and agents can be right -sized more tightly. They also say these scopes can be assigned through the dashboard, the API, or Terraform. And I think that this story matters because it is one of the clearer examples of the industry dropping the pretense. An agent with a token is not some cute helper. It is an identity with power.

A script with standing access is not background noise. It is an identity with power. A third -party OAuth app is not just a convenience. It is an identity with power. And once you accept that, the rest of the story gets more normal. Token scanning, connected app visibility, permission scoping, leased privilege. This is just IAM growing up around modern workloads and agent -heavy environments.

And I also like that Cloudflare is not treating this as some abstract future of security thing. The token changes are practical. They added a recognizable prefix and checksum so scanners can identify Cloudflare tokens with much higher confidence. The OAuth work is practical too. You can review the app name.

Publisher, requested scopes, and which accounts the app is asking to access before you approve it, and then see those connected applications later in one place to revoke them if needed. And their resource scoped permissions framing is probably the most useful mental model in the whole post. The token gets you in the building. But scopes should decide which rooms you can enter.

That is the part I think teams should take seriously. If you are letting agents, scripts, and third -party automations pile up with broadstanding access, you do not really have an AI governance problem. You have an IAM hygiene problem that got new branding. Okay, a few quick ones before we wrap.

Microsoft shipped April patches for Azure DevOps Server and says they strongly recommend staying on the latest secure version. The patch fixes a null reference issue that could break pull request completion during work item auto -completion, improves sign -out validation to prevent potential malicious redirects, and fixes PAT connection creation for GitHub Enterprise Server. That is a very practical PatchNow item.

Google also pushed more on OTLP metrics for cloud monitoring. Google says you can send metrics to cloud monitoring through a provider -agnostic OpenTelemetry pipeline, store that data in the same format as managed service for Prometheus, and query it through the same cloud monitoring interfaces. That is a nice observability standard story because it is not just support the protocol.

It is make the protocol path actually first class. I think the human thread underneath this week's episode is that a lot of engineering pain comes from waiting too long to make responsibilities explicit. Kubernetes is making some of that explicit by deprecating or removing paths it clearly does not want to keep carrying forever. Gateway API is making networking intent more explicit.

AWS is making platform preference more explicit by telling people where Copilot stops and where the next preferred paths begin. Airbnb is making alert quality less dependent on vague craftsmanship and more dependent on visible feedback before merge. And Cloudflare is making it harder to pretend that agents and scripts are somehow outside the normal identity and access conversation.

And this is where I think platform work gets misunderstood sometimes. People talk about maturity like it means more automation, more abstraction, more paved roads. Sometimes it does. But just as often, maturity is really about being honest sooner. Honest about which patterns are legacy. Honest about which workflows are broken. Honest about which tools are losing support.

Honest about whether production is secretly your only validation environment. Honest about who or what actually has access in your environment. That is not as fun as a big shiny launch. But it is usually where a lot of the real risk reduction comes from. Because the longer a team stays fuzzy on ownership, the more that fuzziness turns into toil. And the more that toil turns into staffing pain.

And the more that that staffing pain turns into reliability pain. Not because anybody is lazy. Not because people do not care. Usually just because too many systems stayed ambiguous for too long. So yeah, that's probably my biggest takeaway from this week. Better platforms do not just make things easier. They make certain kinds of vagueness harder to sustain. And honestly, that is usually a good thing. All right.

That's it for this episode of Ship It Weekly. Quick recap. Kubernetes 1 .36 and why it feels like a maturity release. Gateway API version 1 .5 moving more core networking features into stable territory. AWS Copilot reaching end of support and what that says about shifting preferred paths. Airbnb proving alert pain was a workflow gap. Not just a cultural issue.

And Cloudflare making the case that non -human identity is now a core security story. Then in the lightning round, Azure DevOps server patches and Google Cloud OTLP metric support. Links and show notes are on shipitweekly .fm. You can also find video versions on YouTube. If this episode was useful, follow or subscribe wherever you listen and send it to the person on your team.

Who keeps having to explain that reliability problems are usually workflow problems long before they became on -call problems. I'm Brian, and I'll see you next week.

A lot of teams say they want better CICD. What they usually mean is they want fewer weird failures, less shared state nonsense, less tribal knowledge, and way less time wasted fighting the delivery system itself. Because the problem is not just that Jenkins is old.

The problem is when your pipelines are noisy, Fragile, hard to reuse, hard to observe, and tightly coupled to a bunch of infrastructure decisions developers should not have to think about. And when you finally fix the problem, you get a different problem. Success. Because once delivery gets fast and easy, people start using it for everything.

Bots, maintenance changes, dependency bumps, mass rollouts, and now the real question becomes, how do you keep the system smooth when everybody finally trusts it? Like and subscribe! Hey, I'm Brian. I work in DevOps and SRE, and I run Tellers Tech. Ship It Weekly is where I filter the noise and focus on what actually matters when you are the one running infrastructure and owning reliability.

Most weeks, it's a quick news recap. In between those, I do interview episodes with people who have actually built things, migrated real systems, and learned what works the hard way. Today is one of those conversations. I'm joined by Stefan Moser from Pipedrive.

He helped lead a big move from Jenkins to GitHub Actions, built a self -hosted runner platform on Kubernetes, moved delivery towards GitOps with Argo CD, and helped roll that model out across a large internal estate with hundreds of services. And what I like about this one is it's not just tool talk. We get into why Jenkins had become painful, from groovy fiction to noisy neighbor problems on shared VMs.

Why GitHub Actions ended up fitting better. How reusable workflows and custom actions helped. Why they chose Argo CD over other deployment options. And how they had to build better internal observability because GitHub alone was not enough at their scale. We also talk about the migration strategy. Which honestly is one of the best parts.

Dogfooding first, migrating in batches, using internal teams as the first proving ground, letting the process get polished before pushing it wider, and building something self -service enough that teams eventually started migrating on their own. And then there is the mobile story, which is its own thing.

Mac minis, messy runner drift, different toolchains and the surprisingly practical path they landed on after testing a few different options for stabilizing mobile CI. If you care about CI CD architecture, platform engineering, GitHub actions at scale, or how to do a migration like this without setting your org on fire, this one is worth your time. All right, let's jump in.

Today, I'm joined by Stefan Moser from Pipedrive. He helped lead a big move from Jenkins to GitHub Actions and built a self -hosted runner platform on Kubernetes, plus a GitOps CD flow with Argo CD. And we're going to talk about what worked, what broke, and what's worth copying. Stefan, thanks for joining me. Thank you. Thank you for the opportunity to talk about this adventure.

Basically, it's not the first time I'm talking about this. I already had... Meetup session that was recorded on YouTube, two blog posts, and basically just sharing again these adventures. But now with a little sparking because something already after one year already changed. So I have more stuff to add to these blog posts and meetup that we had. Awesome. Well, I'm excited to learn more.

So starting off, can you give me a thesis? Why did Jenkins stop working for you, and what were you trying to optimize with this new system? So basically, the first problem that we had with Jenkins is Jenkins decides being the pipelines being written in Groovy. That was the first problem. In Pipedrive, we are mainly working with TypeScript and Go. Groovy was really not very needed for us, so it was a bearer.

For the DevOps teams that we had at that time, plus the engineers trying to do writing something. That was the first issue. The second issue was basically the setup. So we had VMs, and that VMs were not isolated. So that means if a big pipeline lands in one VM, and we had other pipelines like to build the Docker containers, then we had the issue with the nice...

Noisy neighborhood so basically we get a starvation of the resource so it's really not predictable we tried to to improve that so we had a crazy idea first to build a internal ci engine so that was really the most crazy thing they did so basically copy the idea of yaml from github's gitlab ci we build the engine and then start to work on that but then we had another issues People need to learn and use syntax.

And after GitHub Land launched GitHub Actions, people started using it, some developers started using it in their app stores. And then we were thinking, why not experimenting? And the first idea was basically, besides Jenkins, we had another product, CodeChip, that was to do the pull request validation. So every time the pull request is open, we had a typical lint and test work.

So basically linting to building the... Code, leading the code, and doing the unit tests. They were running in the code chip. And I can tell it was worse than Jenkins because every project was individual. So imagine if I had to change or to share something, it's very difficult. So we had an idea. Well, let's try to win the GitHub Actions. And basically, I team up with a colleague, Gregor.

And basically, it was the idea that... Let's replace CodeChip with GitHub. Actually, let's see what happens. And that was really the kickoff. So we get a green card to export idea. So the first thing was thinking that is, okay, I need to run this somewhere. Probably GitHub, even we had a free package, will not be enough. So let's go to find solutions for self -hosting.

We found, after some ideas, we found a community project. It is ARC, Actions Run Controller. At this moment, it's already belonging to or maintained by Gita, but at that time was really purely community -based. One thing that it was very... Important for us, or very convenient for us, it was a Kubernetes controller. So we spent the last years working with Kubernetes.

We are understanding the Kubernetes API, these CRDs, all talking in the Kubernetes way. So it was more easy to work with that. So we define resource, get runners running, fine. One thing that is also awesome, we can grab a pre -built image and then... And put that customization that you want. So basically, developer needs HGK, IKH, or different tooling. Just put it inside of that custom image and then we ship it.

It was even for easy. Then we already know how to restrict and monitoring Kubernetes in production. Just apply the same idea in the...

CI environment so basically I did a thing that is normally not normal standard so basically I don't I didn't set the requests to be the smallest possible most necessary no I just was brute force so I say for example set the request and the limits to be almost the same so that I could have one thing that is pretty bit in the So that every time a developer gets a runner, it's always the same CPU and memory.

So that avoids the problem of noisy neighborhoods. Then we had already ideas how to, we already know how to see and control the resource in Kubernetes. So apply the same metrics. So basically we bring back the, of the really tools that we had in production to.

The ci cluster but then we also did want to be more straightforward in in the process to maintain the clusters basically instead of building the cluster from scratch we use ets to be more easy and then we want to to have scalability in terms of the nodes we decide then to go to the new project at that time was carpenter so basically the the magic way of AWS to spawn nodes faster than a cluster of auto -scaling.

So basically, then we get this solution. So a controller that was listing the GitHub app books to spawn jobs, spawn when the instance creates a new pod that represents a runner. If that runner doesn't have a space, the carpenter then creates a new node.

And that was really bring the flexibility okay I just need to set the max number of thoughts or the runners and the ecosystem for the cluster scale up and scale down when when necessary and this was really basically the why we we moved to uh two changes and first steps we then basically did the migration of code chip to github actions at that time also was released the reusable workflow so that means we are able to

Reusable to create a workflow and spread that workflow to the old repositories so reducing the the repetition that we had and the manual configuration we had with Jenkins with the code chip and Basically, after we dropped out of CodeChip, it was the time that we had the opportunity to revamp CI -CD.

So we bring a group of engineers, I think four engineers plus a developer experience product manager. So basically, we had a product manager dedicated for the developer experience. And then we decided to revamp.

Uh necessity and they basically said to put some kind of competition in in terms of tooling so I remember that the first contender was basically github actions in terms of ci but then also we bring argo workflows and tecton because were two projects that I was curious about using And in terms of deploying, I'm thinking Argo CD plus Flux.

We even tried with Spinnaker, but it was so messy to spin up that system that we just dropped it. And how we chose, basically, first was...

Readability or feasibility or the easiness to create the workflows and the specific customizations so and that is really shine for us the kit of actions basically people complain about the action but really the fact that the actions are written in javascript and then we are using typescript so it's natural to create custom logic with typescript even then we can go to creating composite actions with some more for some batch scripting.

Scripts, we just use that language to create the customization. And then it was really easy to pack up. So basically, we create action, we have a package, like a package, and then can put it a block that we can put in different place. And then the other point was basically about how we can expose or show the workflows to the developers. And that is... The less clicks the developer does, find lots better.

And that, of course, makes the GitHub action to be a big content. So the fact that the workflows was next to the repository, the execution was next to the repository, the developer didn't need to switch to the platforms to see was really a big plus. Yeah. And that's really the reason why we went to... To GitHub Actions. Even at some point after this migration, what we had, it's basically a monorep of GitHub Actions.

And I think we have 50 or 60 GitHub Actions. So basically, we just build our actions and even it's our monorep of all the code base that you want to reuse in the organization. In terms of development, of deployment, one thing that we want to make. Clear is that we want to avoid push code.

So one of the issues that we had always is that with Jenkins and this push idea is that fact that I need to have a runner in the clustered push code. That means that I need to give the Kubernetes credentials to a runner to be able to write stuff. And that means that in some way I can try to capture that runner and do malicious stuff.

Let's do the reverse so that is that the cluster is really in isolation and we get what we have it's someone inside that check something that it's really it's already trusted and use it to um to apply change in the cluster and that is really why you want to go to the to do githubs and of course in githubs we have two two tools to choose argo cd or flags yeah And in that case, it was basically one of the reasons why Argos CDF had a good UI comparing with Flux at that time.

It was like two, three years ago. So that was really the big reason that we picked up with Argos CDF Flux for the UI. And then we had already a way to show to developers what has happened in a more easy way.

Because again, that is that it's easy for developers to visualize doesn't mean that I allow them to change manifest with rcd yeah I know what I allow is to basically scope in inside of the application saying okay this is your application see your status you can go see the status of your resource you can see the logs if you need it you can see if the content the pod is it's restarting or not really um we added that idea.

Yeah, you can show. Yeah, after that idea, I think that picking up the GitHub Actions took me like two days to create MVP of the deployment flow, basically because I was reusing everything that I already did in the CodeChip migration, and we decided to have that MVP, and then... See what are the gaps.

So one thing that I forgot to mention is that initially in the first step or first iteration that we had with migrating to CodeChip, we had a problem with a lack of observability in terms of actions. GitHub at that time didn't provide any statistics about the actions. So if you want to know how long was the runner, we have any idea. You can just... See at the repository and was not good enough.

But especially in our organization that we already have, at that time we had like 700 service. We have architecture of microservice plus libraries. It was impossible to track at the repository level. So what we did is basically we create a service that lists these events and then starting putting everything inside of a database so that could build.

A source that we can then do queries and then find uh what are the the performance of of the the workflow because at some point a manager a director or even cto can comes to us it says what is the failure rate of these uh workflows where what are the workflows that are taking or these guys what are the deployment flows that are taking more time what are the steps that we need to to optimize and then basically this is really our gut feeling that we need that information and we are storing that information.

And until now, we're still using that service to track, even now to tracking more billing information, but it's really the source of truth. Okay, that was the phase one and phase two, phase three. It's basically making the product ready for the consumption. And one thing that we discovered that we lack, it's basically, again, organization -wide of the visibility.

GitHub actually is great to see your workflows, to see the organization. It's bad because we don't have anything. So we create a service. That is basically a registration service plus a UI that we have in our back office. So basically we have a UI in our back office that shows all the deployments, but they need to have some source.

And basically we create a service that consume events that we are producing in our workflows. And after consuming these events can do the mapping of what it's app. And basically we have basically our own interpretation of the events of. Because they have a different interpretation of what is deployment, what is a deploy, what is a region.

And then basically we add that internal logic or that internal understanding what is build, test, unit test, functional test, end -to -end test, deploy to a region, deploy to a... Deployment at a whole scale. And we have that understanding inside of a service.

And basically this was initial, the service that was consuming everything to create observability of what we need to do or what we have in terms of deployments to production. And after that, we have that, after having that UI, we are more confident to go. Now goes the question, how you do this kind of release? 700 service. I think we have five teams or 10 teams at that time. It was really a big migration.

How you start migrations? I think like any product should migrate or should be released, you first try it yourself. So basically... And we decided to migrate our own service. My team doesn't only produce CI, CD. We have a couple of service that supports the deployment flows and support the developer experience. So for example, we had one service that creates and do all the lifecycle of remote dev environment.

So basically we have remote dev environments that developers can use that it's a replica of production. And then we have a system that creates that. For example, this is one of these servers that we need to migrate. And that is really basically like this. So I have scripts, I have workflow, I have documentation to do it. So I deliver to my teammates so that my teammates can migrate. These bring two good things.

First, we test our process to someone that doesn't know about the process. And second, we allow... To share knowledge. So basically, knowledge that was restricted to that five people that create systems starting to be spread to the team because I'm sharing that knowledge by forcing the people to do the migration. And basically, it was basically this alpha grouping. Then we had a different team.

So basically, it was a platform team that it's more experienced than the normal developers around stuff with Kubernetes. And the deployment process and they have more out of the box or out of the standard service, then we go to them and say, okay, we have this Polish workflow. Let's now teach you, do a session and then allow them to migrate.

And then it was basically like the open beta of the system, of the migration. Again, the idea was to... Make everything more polished, more clear for the users. After they give us our feedback and then we improve the workflows, define, we start doing to think how we do all the role. In this case, we pick up the prioritization that SRE team define for the service. So basically SRE define tiers.

So tier one is a service that's very critical. Tier two, it's more or less. And then tier three is less critical. So we decided, okay, let's split this in batch. Let's go team by team, starting in tier three. Then they can move to tier two and then tier one. And that was the idea. And then the plan was basically split it in batch and assign a DevOps engineer to each batch, not to do the...

The migration but to be the assistant so basically the person who does the introduction of the new way ui the new workflow and then this is the the the interaction goal that we have to to do your mind and also to make them so basically if I want everyone one person that's responsible to achieve a goal of migrate x number of of service until the end of the month it will keep the momentum in the team and then we

Started doing that so team by team we started moving we have a schedule we have a limit number of of engineers so basically we have a queue until some moment we had guys from the middle of the queue at the end of saying okay I already saw the other guys doing migration.

I think I can do it alone. Okay, just check the recording that we had. Try yourself. Yeah, and they decide to try alone and starting doing migration alone. So basically, the system was already so well oiled and everything was moving smoothly. They were able to migrate alone. And that's really, I think, the real sex story of doing these migrations by batch.

Doing your dog food and all these things in a way that makes everything automatic or with less human intervention to allow people to use it. And then you also can think this is really the idea of maybe platform engineering, making everything self -service. When you have a product that's already usable for a normal product engineer that doesn't need to have the context of the CICD and how to deploy to Kubernetes.

It's a service that he can use and can work and can do his work autonomously. And yeah, that's then after five months, I think we migrate everything. Yes, we found what we get as issues, basically our success starting break things. So the process was so smoothly. The deployment flow was so good. That then we're starting an introduction of bots to look at deployment for the maintenance tasks.

So basically, the SRA team developed a service to create pull requests to adjust the resource usage in production and then automatically deploy that to production. So basically, it creates a pull request, we introduce a set of resources, and this pull request is already pre -approved, so it moves by all the normal pipeline flow and goes to production. Then we have the PandaBot.

Creating pull requests for the pendants and in some case the developers were already confident enough in the unit test they have in the functional test they have allowed that obligations or updates of of dependence to go without any approval from a human so we had then a situation that we have multiple um departments happen and then was struggling or doing impact in our system. Even we can grow, grow, grow.

Then we add some bottlenecks. At that point, we decide to improve that service that was the deployment registrar to have a queue. So the idea is that you then have a way that you, when you start the deployment, you send event, I have a deployment. And then what happens is that the deployment registrar register that deployment and put in the queue.

Evaluate the queue size so basically we define that we have the number of 50 deployments per available in parallel and also we do a tricky thing that is we only allow 10 of that queue to be used by bots so basically we always want to have some kind of free space for the moments to develop that develop features to ship and basically that's a and so we have a queue with all the idea to put a margin for humans.

And then if everything is okay, then we rerun the workflow that allows to execute everything. Yeah, it was basically that thing that we had to improve. Also, one of the bottlenecks that we had was we had issues with how we commit to an environment state. So basically, it was one of our... Really bottlenecks is when you have multiple deployments, you need to be, only can commit once, one at a time, the deployment.

So basically, we had a strategy that we create one commit per region. So we have a lot, a pile of commits to push to that, this environment site repository. And that was really a bottleneck. Fortunately, we were not very clever at that time to create the queue for that specific step. And then we had to re -engineering all the process because we had the issue that we didn't do like a FIFO.

So basically, we did implement a hot hobby. And that means that some guys was very unlucky that it was the first one to arrive and was not the first one to get their stuff deployed. Yeah. This is really the story that we have. What was missing at that time was in the presentation and blog post was really the migration to the mobile team. I don't know if you want to go to already that story.

It's a new story that I have. Or you want any other questions? Let's actually get to the mobile story in a second. So going back over your CICD process, I'm curious. First off, are you worried about The GitHub change to private self -hosted runners, they're changing the pricing model. So I guess they're going to charge for self -hosted as well now. Is that going to change your implementation at all?

That's the problem. I can think that if they're starting to charge that, then I need to start to thinking, if they start charging like that, then I need to really be very picky in their SLAs.

So I they how they can charge me a fee for the control plane when their control plane it's not really 99 or doesn't yeah meet the slas it certainly hasn't been lately that's for sure that that is really the thing even even we don't even we get that uh we more or less exclude that from our matrix of of um I already starting to think already seeing some page that shows the SLA in the last couple of months.

And then I'm starting to thinking if you charge me for this, I need to have a better SLA. I cannot stop working. And then for the size of what we have, it's basically or I need to find a different CI tooling and going to that path or even can go more crazy. Don't forget that, okay, we are a special case, or probably not a normal case. That is, we are already in enterprise level.

And we have GitHub hosted enterprise server. So basically, you can bring yourself. And if this is starting to be expensive, probably that's the thing I need to do. I bring down and then try to be less dependent of... The ability of the cloud version and then be my concern. And then I can then point to myself. Yeah, your GitHub is down because of myself and not because of some change in the cloud version.

But yeah, it will be really a concern. And if they force us, it's basically now I will go to my legal team and then check the SLA and let's see if they don't like the SLA. They're starting to get some notice from our legal team to get a recharge back or something. So also you had mentioned, that's a fair statement for sure. You had mentioned looking at Flux originally and then going with Argo.

Are you using anything like Crossplane with Argo yet or no? No. So basically the fact that we don't use Crossplane is basically because it's not in our domain of work. Okay. From what I understand, really, the idea is that cross -plane is a good way to provision a resource using Kubernetes as a native language.

Even I try to understand from Victor, Victor Werczek, that's working with cross -plane, the difference between Terraform and cross -plane, and that's really the reason that... We didn't go deep in crossfire because the main player of Terraform and provision infrastructure is the infrastructure team, not the engineering excellence department. Basically, we are the middle layer. Imagine it's like a lasagna or a burger.

Basically, the team is the lowest band and I am the lettuce. And then we have even up on me, we have the burger and then we have the tomato and they have the other. So basically we have.

These layers and I'm in that layer that I consume service from infrastructure team and then I deliver service to to to the to the platform team and to the to the developers so that's the reason that we don't look for crossplane that's thing that I would like to to experiment but then I need to really have a good case of why did a terraform to use crossplan yeah it's more I guess if you want the infrastructure

Definitions closer to the actual service right so if you want that all defined together and there's there's pros and cons for both ways I would say honestly even in our organization we kind of or I've worked in organizations where we've done both yeah even even listen uh what victor said about the idea of cross plan and how to use cross plan and that year for example is one of the examples that I can tell or I can get a model in terraform that's getting a postgres to be from Mother bless.

But probably if I want to talk with the developer, developer wants to have the minimal settings to change. So basically, I think the WinCross plan can abstract that with their internal resources and say, okay, define this YAML manifest or that YAML resource or, sorry, that CRD in YAML and then the controls and everything will set up everything for you. So at this moment, we don't add this kind of...

Need in terms of organization to have exposed so much the infrastructure to developers. And that's real. Without need, I don't have a way to force a tech to be used. Oh, that's fair. It's always a balance. Yes, it's cool, but I need to really, it makes sense to the case. So tell me about this, the new mobile deployments. How is that going? And how did you set that up?

The mobile team was using Jenkins, but even with more strange setup. So basically, they had a farm of Mac minis, and they basically reconnected with the Jenkins controller master. And they were using that. So basically, they were using groovy, pretty groovy with...

In their checking pipelines plus with fast line for one for the ios team the android team will use a different tooling and that was really that massive because part of the team needs to be operations to understand to upgrade the nodes to fix issues and also again the same issue of isolation when we had cases that the nodes were not very identical and a job was if lands in one machine was passing if lands in the other

Machine was failing or less force between artifacts between was screwing up with node versions hubi versions and was really a mess so lack of productivity and then that was really the idea We need to move to GitHub actions, but plus also find a way to make their compute power or compute resource be very stable, having the same identity that we had in the CI -CD for the microservice.

So that was a multiple deployment and isolate. And at that point, it was really a surprise for me, the end solution, but basically it was this case. So I went to research. That basically I put 4Ks on top of the table. First, using VMs inside of Mac. So basically a product from CircleCI. So basically a company that's already providing GitHub Action Runners for Macs and have the system to work.

And basically it's on top of the one tooling at the start. And basically it's a... A nice tooling to spawn VMs inside of Mac. Then the idea, okay, let's try to use Nix for some independence in isolation. And the other idea was basically also to use AWS. So why not spawn Macs in AWS and use it as a GitHub address? And at last I was thinking, I need to at least think in the way of outsourcing this to a company.

This guy is GitHub Actions, but... I could also think to other GitHub action providers or first to GitHub itself and then to other providers because you can see that Blacksmith and I think Depot are providers of GitHub action trends that you can offload and don't depend of GitHub to have the best performance in your machines. Basically, it was really a poor scientific research. I have three, four hypotheses.

I have one month to test it. And I set like one week for each hypothesis and then try to go. And this is really the first. It was in summer of last year. And also that culminate in the appearance of the... AI native mindset. So in this experimental research, after I collided with teachers, with tooling, I found that I had to use start to create VMs. So first hypothesis.

And then I would decide, well, if I try to create a controller, like I have the idea or already the use case that to have a control for Kubernetes, to run runners. So why I don't have an action runner control for that? And that was really the idea.

Starting to think I did a poc with shell script because it was a very easy command but then I decided okay I have a nice shell script let me do a really nice mvp and then I did my first specification development project so basically I used this idea I then signed to define my specifications in a markdown file what I want, what were the toolings, what were the constraints.

And then use, in this case, was already still using GitHub Copilot and say, I have this idea in this file, let's make a plan. And starting elaborating the plan, creating the plan. And then the process was really that way that after I have the plan, we need to have to -do lists or to -dos for each point. And then basically I force the Copilot to use, go for each to -do or each step. Do the implementation, I review it.

Okay, it's fine. Let's move to the next one. Fine. And then also ask to do a summary of each implementation. So basically to have a history of what I did. Just important for me, but also important to share with the team all this process. And then after one day and a half, I got a control. So basically I had a Mac mini in my desk. I put the control there and it was spawning. And doing the lifecycle of the VM.

So basically, a workflow, it's my runner, execute a niche, drop, tear down the VM, start a new one, register against GitHub, like normal flow. And I will say, yeah, nice. I just need to make some improvements. Then I moved to the Nix. AI, in this case, also it was Copilot, helped me a lot. How to build the recipes with Nix, but it fails tremendously just because of the way Xcode works. It was very annoying to work.

And then I had the issue of how to distribute Xcode. At that time, I was running out of time. AWS was not really an option to investigate. And then I started doing some calculations about cost if I use GitHub as a provider of headers. And surprise, surprise. It was cheap for our use case. So it was basically a matter of after spending three or four months collecting metrics in Jenkins, I say, yeah, we can use it.

And basically it was really the idea. I said, okay, this is the amount of money and comparing the working hours that it's necessary for an engineer to fix this, it's a good balance. And then after I convinced my director that it really makes sense in terms of financial terms.

You approve and then we move and then we move to migration and that this migration was I did with the junior the first thing that we did when this migration was really sit down with the developers and I asked anoint us to to to my junior in this case it was to internities sorry but it was really important to that yeah I asked him go for each um jenkins pipeline they have and start doing a flowchart so basically we

Had a flowchart for each Jenkins pipeline with steps and the steps in the way that what is supposed to do and what are the commands that are executed and then I sit down with the with each uh team from android and and ios and then let's go forward for each step and then try to understand as this makes sense this flow I don't care about what is the command that is executed.

Does this step make sense? Does this test make sense? Does this fork in the logic make sense? And then we also understand some good things that is some workflows or some checking job that we have was already redundant. We could refactor the input and combine in one single pipeline. That was really the idea. And then that was the second time we used the AI to speed up.

And this time I already added the session of agenting coding with my engineering department. So basically the engineering excellence create agenting sessions to teachers all to use AI tools in a more agentic way. So not to auto -complete features, but to give context, to give a goal. Have this guy, this AI, as really a partner to execute. And then at this time, already we was using cloud code.

And then, okay, let's bring these flowcharts, convert to something that is more digestible by AI. So I was able to export as a CSV file. And then, okay, these are CSV files, contains flowcharts of our workflows. Let's build GitHub Action workflows. And documentation. And he's starting doing the old workflow. Was not really exactly what we want, but was close enough.

Imagine that this was really a good best draft, a good first draft. This was really, now just adjust this step, this step, this step, and then we just starting building on that. And then, of course, this really make the work very easy. So we add like 17 workflows to migrate. What is the difference from this migration to the other migration? The other migration, we add like one or two workflows for all the process.

So basically, the issue was to replicate that to use workflows to 700 service. In this case, we have only two service, two repositories, the Android and iOS, but we have multiple workflows to migrate. And then we had to rewrite a lot of stuff. And that was the way that we use AI to basically at the pace of each day we migrate a workflow and then cloud code was able to digest some part of the code base.

So I had two issues. First, parts of the customization that they have in the iOS team was using FastLine that is written in Ruby. I don't know Ruby. So I used it to understand what was that.

What was the logic behind that ruby scripts and had extra features then I'm not very well versatile in the ios test and compiler so I never I'm not irs developer I don't know all the quirks about the the process to compile language compile ios app And I use protocol for that. So basically, it was already in a way that I had an issue in my pipeline. And I tell them, OK, I have problems like this.

I have an issue in my pipeline. This is the idea of the workflow. This is the idea of job. This is a step that's failing. Fetch.

Using github cli in that time even we using mcp probably it's best if you have a cli to tell the ai model to use that cli to fetch the information instead of uh bloat the contacts with mcps use the cli fetch the logs in that section and that let's go investigate what it's what it's failing and it was really able at some point I get get surprised because when I go in deep mode of troubleshooting the model in this case

Even the the agent that is called cloud cloud was able to go to the internet and find github issues about the problem pointing out I have found this issue probably it's about this let's double check and then I double check yeah probably makes sense let's try this change and it was really basically it was a more family language or I cannot what I can say it was like while we were in the past doing Googling.

So you put the problem, you try to find the issues. Here, I have the problem. Also find the issues in the internet and get me back the information and then validate with me and then explore. And basically, after four weeks, we migrate everything. We even add extra features they want. And they were very happy. And still, they are very happy. Of course, the initial costs failed because it was more than I expected.

But for one reason, developers were delivering more. So it was really a situation that the workflows and run are so stable, they can focus more in future and then increase the cost. But it's because they are shipping more features than they did in the past. That's a good cost problem to have, you know? Yeah. Yeah. I think that's the thing I have. Do you have any questions about that topic?

I did want to ask, wrapping up, if someone that's listening, they wanted to pull off a big CICD switch like you did, are there some lessons learned that you could give that they could follow so they don't set their org on fire? I mean, because this is a complex... Lift and shift, right?

Going from Jenkins to GitOps and introducing Argo and all the complexities around, you know, the Mac mini pipelines, like be interested. There's some like core lessons learned that you could impart. So the first thing is, I think you need to, even you have a big pipeline, I bet that you have a small part. So a niche. So try to find. A small part that you can replace and do it in isolation.

In my case, it was basically the pull request validation. It's detached in some way of the big flow. Try on that. If you cannot do that, try to find segmentations that you have in your organization in terms of teams. That's a good way to approach so that you can move parts of your team. Basically, if you have 10 teams or five teams, pick one team and try to go in that way.

So this is a way that you can try to reduce the buster radius. And of course, I think that was a thing that was very important for us, basically doing dogfood. I think it's very unfair for someone that is developing tools for developers not using that in their work. So I think this is really the most important thing. It's doing dogfooding.

And then if you want, try to find in each place if you don't have anything think do you have internal tooling that doesn't uh provide for your final customers that can be bad but for trying to apply this to your internal tooling yeah that makes sense cool where can people find your uh your posts and and where can they reach out to you okay so my posts are in the medium probably you can find them have the links in the description also I have that both was based in the talk that I did.

So also in YouTube, I will have that talk. Also, sometimes I do some publishing in LinkedIn. So you can go there, try to reach me in LinkedIn. Awesome. I'll leave the links for your Medium posts and your LinkedIn and anything else in the show notes. Stefan, thanks for coming on. Really appreciate it. Okay. Thank you. All right. That's my conversation with Stefan Moser.

My biggest takeaway from this one is that good CICD is not just about picking a newer tool. It is about building a delivery system that is predictable. Observable, isolated, and usable enough that engineers can trust it without needing constant help from platform teams. That is really the thread running through this whole episode. They did not just swap Jenkins for GitHub Actions.

They reduced noisy neighbor problems. They standardized runners. They leaned into reusable workflows. They moved deployment towards GitOps. They built their own visibility layer when the platform was not giving them enough, and they rolled it out in a way that let teams build confidence instead of forcing a giant overnight cutover. I also liked that he was honest about what happens when the new system works.

Once deploys get easier, people use them more. Bots start shipping changes, automation starts piling up, and then you discover the next bottleneck, whether that is queuing, fairness, or protecting enough room for humans to still get work out. That is the real platform lesson. Success creates new load.

And the better your self -service story gets, the more you have to think about throughput, guardrails, and the system behavior under trust. The other part I liked was his migration advice at the end. Start with a niche. Reduce blast radius. Dog food your own system first. And if you are building tools for developers, use them yourself first before asking everyone else to bet on them.

That is probably the cleanest takeaway from the whole episode. If you enjoyed this episode, follow Ship It Weekly wherever you listen to podcasts. If you want the show notes, links to Stefan, his write -ups, and the resources we talked about, head over to shipitweekly .fm. Thanks for listening, and I'll see you later this week. Thank you.

A lot of infrastructure work gets easier right around the moment it gets more opinionated. Private connectivity becomes a product. Cluster networking becomes a managed default. Ingress migrations stop being optional. Observability config starts acting like a real standard. And the old we'll patch that next sprint stuff still shows up in the lightning round waiting to ruin somebody's week. That's the theme this week.

The platform layer keeps absorbing work teams used to hand roll, babysit, or quietly postpone. Sometimes that is great. Sometimes it is the platform telling you that your old defaults are running out of runway. Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech. This is Ship It Weekly, where I filter the noise and focus on what actually changes how we run infrastructure and own reliability.

Show notes and links are on shipitweekly .fm. If the show's been useful, follow it wherever you listen. Ratings help way more than they should. And if you want more signal between episodes, check out oncallbrief .com. The latest brief this week was heavy on breaking changes, security patches, and platform releases, and that definitely shaped this episode.

We have five main stories today, then the lightning round, and we'll wrap with the human closer.

We're starting with aws interconnect going generally available because this is a pretty clean signal that aws wants private connectivity to feel a lot more like a managed cloud primitive and a lot less like a telecom product then cloudflare mesh which feels like the private network for humans services and agents version of the same broader trend after that gitlab 19 .0 because the move away from bundled nginx ingress

And bundled data services is exactly the kind of breaking change platform teams actually have to plan around.

And that one came straight out of this week's on -call brief. Then we've got EKS auto mode networking and what it means when AWS owns more of the cluster networking path for you. And finally, OpenTelemetry declarative config reaching stability, which is quieter than the others, but honestly kind of foundational. Story one. AWS interconnect is cloud networking getting productized harder. Let's start there.

AWS interconnect is now generally available and the core pitch is pretty straightforward. AWS is offering managed network connectivity in two directions, multi -cloud connectivity between AWS and other cloud providers and last mile connectivity between AWS and your on -prem or enterprise sites. AWS says that the service is turnkey.

Private and designed to remove the usual infrastructure complexity from the customer side. The part that stood out to me is not just the feature list. It's the framing. This is AWS taking something that traditionally felt like network engineering plus vendor coordination plus waiting and trying to turn it into a cloud service with a cleaner control surface. Multicloud starts with Google Cloud.

Azure is coming later this year and the traffic stays on private backbones instead of bouncing over the public internet. The last mile option launches in the US with Lumen. That matters because private connectivity has always been one of those areas where cloud teams and networking teams can end up speaking slightly different languages while the delivery timeline drags on forever.

So when AWS starts saying, we'll manage more of this path for you, that is not just a product launch. That is a platform boundary shifting. And I think the practical takeaway is pretty simple. If you run multi -cloud, hybrid, branch -heavy, or edgish environments, this is the kind of thing worth paying attention to early.

Not because you need to adopt it tomorrow, but because it changes the default expectation of how painful private connectivity is supposed to be. Story 2. Cloudflare Mesh is basically saying private networking should include agents now. Next up, Cloudflare Mesh. Cloudflare launched Mesh as a private networking layer for users, nodes, agents, and workers.

Their pitch is that the client on private networks are no longer just humans and services. Now they're also autonomous agents that need scoped access to internal APIs, databases, and other private systems without exposing those systems to the public internet. And honestly, I think that framing lands.

Because this is one of the first times I've seen a big networking security vendor say out loud that agent traffic is not just a cute add -on to existing access models. Cloudflare is treating it as a first -class access pattern.

Mesh plugs into Cloudflare 1, applies existing gateway, access and posture policies and ties into workers vpcs so workers durable objects and agent workloads can reach private infrastructure directly the interesting part is that this is not being sold like a traditional vpn replacement story it is more like your private network should already know how to deal with services laptops remote servers and agent style

Workloads on the same fabric Cloudflare also says Mesh is many too many private networking over its global network, not a collection of one -off tunnels glued together.

So the bigger read for me is this. Networking vendors are starting to assume the future client is sometimes a human, sometimes a workload, and sometimes an agent acting semi -autonomously. If that assumption keeps spreading, private access tooling is going to look a lot more like policy -driven platform plumbing and a lot less like old -school remote access.

Story three, GitLab 19 .0 is what real platform migration pressure looks like. Now to GitLab. This week's on -call brief highlighted GitLab 19 .0, and I think it is a really good platform story because it is not flashy at all. It is just the kind of change that forces actual planning. GitLab says 19 .0 includes 15 breaking changes.

And one of the big ones for self -managed Helm users is that bundled Nginx ingress is being replaced by Gateway API with Envoy Gateway as the default. GitLab says Nginx ingress hit end of life in March 2026, though it can still be explicitly re -enabled until removal in 20 .0. That alone is enough to matter. But then there is the second punch.

GitLab is also removing bundled PostgreSQL, Redis, and MinIO from the Helm chart and operator path, with no replacement, citing licensing, maintenance, and image availability issues. This is the exact kind of story that feels just operational until you are the team that has to sequence the migration. Because these are not cosmetic defaults.

These are parts of the install path that people absolutely built around, especially in test, POC, or smaller self -managed setup. And now the platform is basically saying that convenience path was temporary. So the takeaway here is not just GitLab changed some things. It is that platform teams need to notice when a project's default architecture starts growing up past its old bundled assumptions.

Gateway API is not just a new thing to learn. For some teams, it is now the road forward whether they wanted a migration project or not. Story four, EKS Auto Mode networking. Is AWS trying to make cluster networking feel less handmade? Next up, EKS Auto Mode. AWS published a good breakdown this week of how EKS Auto Mode handles networking. And the story is really about ownership.

Auto Mode sets up the VPC CNI for you. Manages DNS as a core component, supports node level DNS caching, and lets you request ALBs and NLBs through native Kubernetes resources without a separate load balancer controller. AWS also says it handles CNI lifecycle management as part of the cluster maintenance. That is a pretty meaningful bundle of responsibility.

Because cluster networking is one of those areas where teams can spend years carrying around a bunch of, well, this is just how our cluster works, setup that is really a mix of controllers, tuning, upgrades, and networking assumptions nobody wants to touch during business hours. What AWS is doing here is making a stronger argument that for a lot of teams, that should stop being custom glue. Pods get VPC IPs directly.

Traffic follows normal VPC route tables. You use AWS native networking services and tools you already know. And AWS is taking more responsibility for keeping the CNI path current and sane. Now, that trade is not going to be for everybody. Some teams want the flexibility. Some teams need the knobs. Some teams have enough scar tissue to be suspicious anytime a managed mode says trust us. Fair.

But for a lot of orgs, the real risk is not lack of knobs. It is the pile of half -owned networking glue they already have. Story 5. OpenTelemetry declarative config getting stable is boring in the best way. Last main story. OpenTelemetry announced that key portions of its declarative configuration spec are now stable.

That includes the JSON schema, YAML file representation, in -memory model, parsing and creation operations, and the OTEL config file environment variable used to point SDKs at a config file. The blog also says implementations are currently available in C++, Go, Java, JavaScript, and PHP, with .NET and Python underway.

I really like this one because observability configuration has had a lot of it depends energy for a long time.

Environment variables everywhere slightly different mental models across languages lots of power not always a lot of consistency so when open telemetry says the declarative config model is stable and should increasingly be treated as a first -class ux surface that is real platform maturity they even say the future process should be declarative configuration first and that older environment variable patterns that do

Not interoperate well may get deprecated over time that does not make for a flashy headline But it does make life better for platform teams that want more consistency across languages and services without every team inventing its own telemetry setup philosophy.

And honestly, that is a very DevOps story. Not exciting in the launch demo sense, but very exciting in the maybe we stop relearning config drift in five languages sense. A few quick ones before we wrap. Container D pushed security releases across supported branches, including 1 .7 .31, 2 .0 .8, 2 .1 .7, and 2 .2 .3 to address CVE -2026 -35469.

The release notes describe it as a SPDY stream issue and call out hardening around sanitization errors before returning them over gRPC to prevent possible credential leaks in pod events.

Patch this one github also launched code security risk assessment for organizations github says org admins and security managers can run a free assessment to review code vulnerabilities across their org and the docs say it scans up to 20 repositories and reports severity and autofix eligible findings that is a pretty decent quick win tool if you want a fast read on exposure without building a whole campaign around it And AWS published guidance on secure AI agent access patterns using MCP.

Their framing is that IAM has to become the authorization layer for these non -deterministic systems because coding assistants and agents choose tools at runtime and can act at machine speed. That is a good reminder that it's just an assistant stops being true the second it has real permissions. I think the cleanest closer for this one is pretty simple. Infrastructure keeps moving from assembled to opinionated.

Private connectivity gets wrapped as a managed service. Private networking gets redesigned around workloads, agents, and not just people. Kubernetes install paths grow out of old bundled defaults. Cluster networking becomes more provider -owned. Observability config starts trying to look like an actual stable interface instead of a pile of toggles. And that shift cuts both ways. It can absolutely reduce toil.

It can also remove some of the comfortable ambiguity teams used to hide inside. Because once the platform gives you a clearer default, you have to make a more conscious decision if you want to keep doing things the old, harder way. That is the human part of ops that shows up over and over. A lot of teams say they want simplicity.

What they usually mean is they want someone else to own the painful complexity without taking away the escape hatches that they have grown attached to. Sometimes that works. Sometimes the industry just moves on and your migration project shows up whether you invited it or not. All right. That's it for this week of Ship It Weekly. Quick recap.

AWS interconnect going GA and pushing private connectivity further into managed service territory. Cloudflare Mesh building private networking for users, workloads, and agents. GitLab 19 .0 forcing some real migration planning around gateway API and bundled services. EKS auto mode networking making cluster networking less handmade.

And OpenTelemetry declarative config getting stable in the kind of boring, foundational way that usually matters a lot later. Links and show notes are on ShipItWeekly .fm. You can also find the video versions on YouTube. And if you want more signal before the episode, check out OnCallBrief .com. If this episode was useful, follow or subscribe wherever you listen.

And send it to the person on your team who keeps having to explain that managed does not always mean simple. But it usually does mean the default architecture is changing whether people noticed or not. I'm Brian, and I'll see you next week.