Ship It Weekly Transcripts

Hey, I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech. Ship It Weekly is where I filter the noise and focus on what actually matters when you're the one running infrastructure and owning reliability. Most weeks, it's a quick news recap. In between those, I drop interview episodes with folks who are actually building in the space. Today is one of those interviews.

We're talking with Gracious James about TARS, his human -in -the -loop fixer bot wired into CICD. We get into how he segments incident response into sub workflows, how he keeps agents from turning into SSH with a chat bot, and the guardrails he thinks are non -negotiable if you're going to let AI anywhere near production. Today, I'm joined by Gracious James.

He's been building TARS, a human in the loop fixer bot wired into CICD. And we're going to talk about what it takes to make AI automation safe enough for real teams. Gracious, thank you for joining me. Thanks, Brian. Great to be here. Like you do it. I'm doing really well. Thanks for asking. So I'm really interested in this TARS bot that you built.

Can you give me a thesis of what you were actually building with TARS? And why is human in the loop the core idea? All right. So I am a software engineer. And when I coded and pushed my code into, you know, CI CD pipelines, I saw that it Took a lot of time to get through a lot of teams, especially the DevOps ones.

So I started looking at the processes of how DevOps go through, the incident response and everything like that. There's a lot of steps, manual steps that each and every person has to cover through. Going through it, I realized that all of these things are very manual, very repetitive. And don't get me wrong, everything is important and can break the system at any point.

So having a human in the loop is very important. But looking at the processes, I also see that all of these things can be automated. So having an AI agent where human in the loop approach increasingly helps us to fasten the processes is what got me going into all of this. Interesting. So what problem were you trying to solve that like existing tooling didn't handle? Right.

So all of this, all of the tools that I used into this workflow. Are existing tools, only that these are not custom made for, you know, my requirement, my company or whatever it is. So I had to custom make the workflow using multiple tools from different, different domains.

So I custom made, you know, the SS access in which there are, you know, validated SSH commands that my AI agent can utilize to find out what is going on into my Docker containers. To get a proper system status report.

And in order to do a comprehensive check, I compare those logs or the errors or whatever it is that the Docker is pointing out to my previous GitHub comments, be it the previous ones, the latest ones, or whatever it is, I compare them and get a comprehensive check. And based on those checks, it helps me to figure out where I should narrow my focus to.

And once I do that, I kind of understand what are the fixes that I should be figuring out in a very short incident response time that I get. That makes sense. Okay, so this was originally Friday, and then you morphed into TARS, right? Yes. Can you walk me through this evolution from Friday? Like, what was Friday and what is TARS? Well, Friday in itself is an Aniton workflow. Now, Aniton is a...

High -level abstract API, sorry, multiple API application system where we can connect different programs, subsections, and sub -workflows in which we take the data from one workflow, one action, and use it for the next process and step -by -step so that based on our custom input, we get a custom output which we utilize for something.

But Friday in itself used a lot of tools, like some of the tools I myself created and named, like visit development server, visit production server, visit this API container, that container. And based on all of this, my LLM creates a system status report in which I get to know whether some container is down or up or what is going on, if everything is working fine. And if anything is down.

Then it sends me a report via Telegram. And based on that report, it's kind of an alarm going off like in a plane's cockpit. And I am the pilot. All I see that, okay, Friday is telling me something is wrong over there. So it's my duty to go and evaluate, check what's going on over there. So yeah, it's kind of an alarm alert system. Interesting.

How do you keep this from turning into like a chatbot just with like shell access? Okay, so yeah. Everything is revolving around guardrails. So when it comes to prompt, the AI agent for doing system checks has its own prompt, and the prompt involves not just doing any actions.

So it's very limited to commands like Docker PS, Docker logs, the errors, and all of this information that it can gather to make a comprehensive update as to what the system is going through. It actually cannot do commands that are harmful for the system. And that's given in the prompt. Now I get your question as to like, the LLM can go ahead and do anything if it has the SSH access.

For that, what I have included is that in the tools like visit development server or production server, those are sub workflows. And those sub workflows. Don't just contain the SSH access. It has some validation parameters. It won't just let the LLM run any command it wants. It has validation checks based on what I have programmed it for.

For logs, it has only Docker PS, Docker logs, and it can never actually go and maybe Docker stop or Docker restart because all of those commands I have deliberately blocked in that sub workflow. So it is kind of a manual override over there. But yeah, that is a limitation that I have translated as a guardrail. That makes sense.

So, OK, speaking on guardrails a little bit more online, you had mentioned explicit rollback phases. What do those look like? OK, rollback phases. So, well, as a software engineer, when I do push some version into my development branch or from development to production, there's a lot of chances like. After the CI, CD testing and integration, something does break when it actually goes out into the real world.

So what we do know always as a software engineer is that the previous version, well, right, the previous version was pretty stable. So our go -to response is, you know, of course, fixing everything and putting it into the right place. But if it's going to take time, if we know based on the system status check.

That, okay, this error is a big one and it's going to take time rather than, you know, just sitting on it for a couple of days and, you know, just letting the clients know that, okay, I'm going to take my own time and do this. I'm just going to roll back to the previous stable version so that their work don't get, you know, hindered or whatever. You get it, right? Oh, that makes sense. Okay, so given the guardrails that you have in place, where do you require...

Human approval or is there human approval like in the process yeah there is there is the human approval uh does not come during the system status check so I don't really tell the ai agent you know where to go to read the data where to go to debug the code or whatever it is I have given it a set of instructions like you can check the docker logs you can check the terminal logs you can check the logging system that I

Have implemented per code base you can check the previous github commits to correlate as to what's going wrong but I don't really say it to you know go and check over here I think it might be wrong over there it doesn't do that what the human in the loop approach where it comes in is to do some kind of action now the action involves you know pressing some kind of button or running a pipeline or a workflow that actually changes some things in the system.

All of these things, the system checks and everything is just reading the data and creating a status report. But where it actually comes to change something, change in the system, that's where the human in the loop approach comes in. And without my, you know, verbal API key, me actually telling the agent to run this pipeline, run this workflow. It won't really do that.

Okay, so you had mentioned too that you had prompts with like guardrails. What are the hard never run this rules that you have set up? So it is kind of a corollary. What I'm trying to say is without a specific phrase, like run the CI CD pipeline to deploy this, this version. This is the exact phrase it requires to run the CI CD pipeline. So in my prompt, I have written it so that it.

Looks for the specific phrase to actually activate that function. So without it, without an actual phrase, without those actual wordings, it can't run it. And that's in the prompt, I get that. But as I told you before, as a guardrail, I have put it also into the sub workflow so that it doesn't just run anything. Once Friday gives me the comprehensive system check, I have two options usually.

One is to ask it again for a comprehensive system check where it has access to GitHub repositories and the GitHub actions. And that's when TARS activates because Friday doesn't have access to the GitHub, TARS does. So TARS goes through the normal Docker logs and compares it with the GitHub commits and what might have gone wrong. And it gives me a proper report as to this is the area that I need to focus in.

Now, it does ask me for a response as to what to do next, because after all, something is wrong in the system. So it's waiting for my approval or some kind of message that I need it to get. So I can again ask it to do some kind of comprehensive check.

But if I'm good with whatever information it has given me, what I'll do is I can tell it to, you know, run the CI CD pipeline to redeploy the previous stable version, the previous GitHub commit. So what it does is that based on this specific phrase, it activates the CI CD pipeline because in the sub workflow of the CI CD.

Pipeline, I have actually written that in order to run this, you need the specific phrase, run the CSED pipeline to redeploy the previous stable version. So without this phrase, so if this phrase involves like, okay, run the pipeline without the CSED or run the pipeline that I last deployed in the GitHub actions, it won't ever run because it does not match that specific phrase.

And that's kind of rudimentary, but it gives me more, you know. Believable focus as to, you know, it wouldn't just run anything just because I wrote it. Like, it is not up to its comprehension. That makes sense. Okay, so let's say that a team wants to adopt human -in -the -loop AI SRE automation. Where do you think they should start?

Okay, they should start, honestly, where they feel that a lot of steps that they do are not just rudimentary, but repetitive in tasks. Because that's where I started. I saw that the incident response procedure in itself has a lot of manual and repetitive steps. And not that I wanted to automate all of it, because after all, it requires access to SSH and a lot of system. And I can't just let an LLM run wild into it.

So kind of segmenting. Yeah, that's the right word. Segmenting the procedure into different steps.

Converting those steps into sub workflows and integrating all of those sub workflows into a pipeline one by one where I have the control between the input of one sub workflow to the output of other one so that I get in I keep in control as to what the whole picture is without actually letting one llm know what the entire big picture is so that it can manipulate is the go -to for integrating these type of tools.

How would you pitch human -in -the -loop AI SRE automation to someone that's maybe more skeptical? All right. So if a person has their own skepticism about AI actions, LLMs, thinking on their own, having access to all of these tools, I get that. I completely get that because that's where I started as well. But when you delve deep into how an LLM works, it works on an input and an output.

So as long as you can use that particular stage of giving an input to an LLM and phrasing that output in a way that I want to see and using that output into another LLM so that that LLM only gets the input that I am giving it, keeps the big picture to myself. And never the AI letting it know what the big picture is. So you obviously introduced this because you're trying to, you know, reduce toil.

You're trying to iterate quicker. What was the first thing that you used this AI tool to help solve? Okay, so... Where'd you start? Yeah, it never started as a fixer bot. It actually started just on the basis of, you know, go into the system and tell me what went wrong and why did the Docker container stop? That's all.

What I used to do is creating a simple LLM workflow in which I copied the output of the Docker logs, pasted it into the LLM, letting it know whatever the data I have, I'm giving it to you, tell me what went wrong. So then it explains me what went wrong one by one. And that's when I thought that, okay, this is one portion of it.

Why not I use the same portion for every single container that I have, but I want to automate it so that all I tell it is that, okay, go to this container and check it out rather than me copy pasting the logs and putting it and whatever it is. So for that, I needed to give it access to the terminal, right? So that's where the SSH access came in.

And that became a sub workflow because I still needed to keep it safe so that the LLM just can't, you know, write anything. I put the guardrail over there so that it can only write stuff like Docker PS or Docker logs. And it did the same thing for me, told me what is going wrong or how the system is working. And that's where it started.

And that's where it saved tons of time, me just copy pasting stuff, me just looking in the wrong places, because I have to look to all of those containers to see what is going wrong. So instead of that, I know exactly where I need to narrow my focus. That's where it all started. And then you just iterated on that and started adding more features. Yeah, yeah, absolutely.

What does future development look like for TARS? Like, let's say the next three, six months, assuming you have the runway and you have the time, what features would you like to add? That is a really good question. Well, TARS in itself is, I don't want to say limited. Limited is not the right word because two, three months ago, I was pretty astonished as to what TARS could do with the tools that it had.

And that itself was a huge accomplishment over there. But now after a couple of months when I use it regularly, I feel that I could go more. So I could give it access to more tools. I can give it access to the infrastructure, maybe Terraform and go from understanding what resources my system needs.

And if it's lagging somewhere, what is the focus that I should maybe increase some resources somewhere or maybe give it access to. Change stuff in Terraform or something like that. That's Terraform side of things, infrastructure. And that's one more tool to think about other than just, you know, GitHub actions and SSH access.

Let's say you have teams that are interested in experimenting with agents in CICD, but they're apprehensive. Do you have any advice for how to get started or where to start? Yep, yep. So it never starts with an LLM, I'll tell you that. Yes, we are trying to embed AI into it, but it never starts with an LLM. It always starts with...

Actually getting your thoughts and putting it into a paper, putting it into steps, logical steps that I can program first and then take that program and understand how an LLM will be effective in what positions. Because if you don't really have a work plan or a game plan to begin with, then the AI will also be as confused as you were. Yeah. That's where I'll start.

So what's your take on fully autonomous AI ops agents? It's scary. Because I myself am not a very big fan of, you know, I don't want to say it out loud, but I don't really believe a lot in experience counts. Because I believe if you're smart enough to do the smart thing, then sometimes even you can, you know, outgrow experienced people.

If you know where to look and how things work very logically instead of, you know, just gut feeling. Because experienced people tend to do that more. And that's where the AI and LLM lacks. It's kind of a hypocritical thing to say that thinking about the AI and LLM, always taking the smart choice instead of, you know, just having lots of experience, just a human like a human has.

But what I'm trying to say is that I still believe in the gut feeling that I truly hate in a lot of people. And that gut feeling is never going to be at least.

With the information that I know about AI and LLMs, it has never proved to have a conscience or a gut feeling that even if it sees all the data pointing to one thing, maybe a human knows that where exactly the error is, but the LLM will always go towards the probable cause instead of the gut feeling. So that's what the... That's where LLMs lack.

And that's why they can never replace humans because we have that creative aspect to us. And it's just a machine at the end of the day. Yeah, that's fair. Okay, so going back to guardrails, what do you think? We've talked about guardrails a lot. We've talked about AI and... Autonomy a lot. What would you say is the single most important guardrail for an AI fixer bot?

In terms of maybe single most, I would say validating every output that an LLM gives before you use that output for anything else. Yeah, that's fair. Okay, so closing thoughts, just curious, what would be like one of the main reasons to start using AI and CICD? What can that buy you? Well, AI is like an assistant. And it works to automate a lot of repetitive tasks that humans don't want to do.

And that primarily saves time. And time saved is money saved. But whenever we do tend to automate these things, we tend to get carried away in a way that, okay, I can automate this. I can automate that. I don't need you. I don't need that. I can put all of this into code. Well and good, but... Anything can go wrong when an AI hallucinates. Yeah, for sure.

All right, so closing out, what advice do you have for people using AI in their pipelines? Well, I understand there are a lot of disadvantages for a full -scale AI taking over a traditional system, but don't let that hamper you. Don't let that stop you from understanding your own process, understanding your own thoughts, writing it down.

Writing a program and, you know, running along with it because everyone gets their own time and you know how to utilize that time.

So you can always stop at a point, introspect as to what you have done till now, how it's going to affect the future, how it has changed your past, how it's going to help other people, how it's going to hamper other people and take all of these things into confidence and building the next step. Makes sense. Appreciate it. Thank you, Gracious, for coming on. Thanks a lot, Brian.

It was great being on this podcast with you. All right. That's the conversation with Gracious. I really liked his angle of designing the process first and only then dropping LLMs in as helpers, not magic. If this episode was useful, share it with a platform, SRE, or DevOps friend who's been playing with AI agents and trying not to blow up prod.

Hit follow wherever you listen so you don't miss the weekly news recaps plus these guest interviews. We'll be back with a regular Ship It Weekly news episode later this week. See you then. Thank you.

This week is a reminder that tooling is not neutral. The stuff we call tooling ends up holding secrets, holding budgets, and holding production together. And when that tooling breaks or gets repriced or gets abandoned, you feel it immediately, usually at 2 a .m. So today, a nasty N8n security issue that hits the workflow automation crowd, quietly raising prices on scheduled GPU capacity.

And a Netflix pattern that basically says stop running critical ops as fragile scripts. Hey. I'm Brian Teller. I work in DevOps and SRE, and I run Teller's Tech. This is Ship It Weekly, where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. Quick housekeeping, shipitweekly .fm has all of the links and show notes.

And if you want to come on for a guest episode and talk through real DevOps, SRE, platform engineering, migrations, governance. Incident scars, whatever, there's a contact form on the site. And if the show's been useful, follow it wherever you listen. A rating or review helps way more than it should.

All right three main stories then the lightning round then a closer story one n8n which a lot of teams use as a workflow automation platform shipped a security fix for a critical issue that can turn into local file access and potentially worse depending on how you run it this is another your internal tool is part of your attack surface moment story two AWS is raising prices around EC2 capacity blocks for ML.

Basically, reserved scheduled GPU capacity. If you do any GPU work, cost assumptions just shifted. If you don't, it still matters because it's a signal. Constrained compute is not... Getting cheaper. Story 3. Netflix wrote about using Temporal to make cloud operations and deployments more reliable, claiming a huge reduction in transient failures. Even if you ignore the exact number, the pattern is gold.

Let's get into it. Story 1 is N8n. If you haven't run into it, N8n is a workflow automation platform. The Glue Tool. The thing that ties together Slack, Jira, GitHub, webhooks, cloud APIs, whatever. And that right there is why I care. A critical vuln dropped, CVE -2026 -21858, and multiple write -ups describe it as a worst -case type issue.

Unauthenticated attackers, remote code execution, full compromise potential, depending on how you run it. You'll also see it framed as unauthenticated file access via certain form -based workflows, which can lead to exposure of sensitive info and then escalation from there. Either way, this is not a patch -it -next -sprint situation. Now, the reason this matters isn't just because N8n is popular.

It's because the thing we keep doing in modern ops is we build these automation islands and then we forget they exist. Somebody sets up N8n or Zapier or a self -hosted workflow runner. It starts as just notifications. Then it becomes kick off a deploy. Then it becomes call this internal endpoint. Then it becomes rotate this secret. Then it becomes close this incident.

And at some point, that tool has enough power to wreck your week. This is the same story as CI controllers. Jenkins controllers, GitHub runners, internal bot accounts, chat ops systems. They're not just tooling. They're privileged systems. So here's the practical takeaway. And I'm going to say it like I'd say it to my own team. If a tool can trigger actions in production, it is production -adjacent infrastructure.

Treat it like that. That means three things. First, exposure. If you have this thing reachable from the public internet, you need to be able to explain why. Because it was easier is not a reason. Put it behind SSO. Restrict inbound, and don't let webhooks turn into open doors. Second, credentials. This is where people get killed. Workflow tools tend to accumulate credentials, like lint accumulates in your dryer.

A token here, a key there, just make it work. And then you've got one compromise system that can call everything. So the move is per workflow credentials and least privilege. If workflow A posts to Slack, it should not have the token that also opens PRs and it definitely shouldn't have something that can touch cloud IAM. Third, ownership and patching. Who owns it? What's the patch cadence?

What's the backup and restore story? Is it monitored? Are logs shipped? Does it have an incident plan? If the answer is, uh, I don't know, you've got an outage waiting for a calendar date. And one more thing that matters in the real world. After a vulnerability like this, I assume credentials might be exposed until proven otherwise. Rotate the meaningful ones. GitHub tokens.

Cloud keys, any API tokens used by workflows. You don't need to rotate your entire company, but rotate the keys that would ruin your month if they leaked. The shortest version of story one is automation is a control plane and control planes need breaks. Story two, AWS is raising prices on EC2 capacity blocks for ML.

Capacity blocks are basically AWS saying, if you need GPU capacity at a specific time, you can book it in advance so you actually get it. It's aimed at ML training and other accelerator -heavy jobs where best effort doesn't cut it. The reporting this week calls out roughly a 15 % increase for certain P5e .48 Extra Large and P5en .48 Extra Large capacity block pricing. There's also mention...

That one region had an even bigger jump, and AWS's stance is basically, capacity blocks pricing is dynamic based on supply and demand. It's reviewed quarterly, and this is what they expect this quarter. And here's why I care even if you don't touch GPUs. Because this is a clean example of something ops people learn the hard way. Cost is part of reliability.

If your ML platform team has runs scheduled and suddenly the unit price shifts, the impacts aren't just the bill is bigger. The impacts are priorities get changed, capacity gets reduced, experiments get delayed, and now you're dealing with delivery pressure and business expectations. Also, it creates internal trust issues. Finance thinks you blew the budget. Engineering thinks finance doesn't get it.

Leadership asks why it wasn't predicted. And the answer is suppliers can change pricing. And if you don't have visibility and tripwires, you only learn when it's too late. So what should a normal platform team do? Start with awareness. Inventory, who uses capacity blocks? What instance families? Which regions? And how much of your spend is sensitive to a rate change? Not your entire AWS bill.

Specifically, the things that could jump and matter. Then, communicate early. If this price increase starts January 2026, you want that message in front of anyone who forecasts spend. Heads up, same workload may cost more starting January. We're adjusting forecasts. Then, set a tripwire. Something simple.

Alert when the effective rate for a known SKU changes above a threshold, or when spend jumps without a matching usage change. The point is to learn about it this week, not at the end of the month. And the bigger philosophical takeaway is this. The more you build on scarce resources, the more you're building on market behavior. GPUs, specialized networking, reserved capacity, all of it.

This is why mature platform teams build two lanes, the premium guaranteed lane for workloads that truly need it and the good enough flexible lane for everything else. Because you want options when the market shifts. Story two is basically the cloud is still a supplier and suppliers behave like suppliers. All right. Story three. This one is my favorite because it's a real platform move.

Netflix published How Temporal Powers Reliable Cloud Operations at Netflix. And the key line that grabbed everyone is that Temporal helped them reduce transient deployment failures from 4 % to 0 .0001%. The context is operating Spinnaker and all the long -running workflows involved in cloud operations and deployments. Now, I'm not here to fact -check the exact decimal.

I don't really care, because I've lived the problem they're solving, and you probably have too. A ton of operational work is basically a workflow. It has steps, it has dependencies, it has waiting. It has retries. It has do this, verify that, and if it fails, back up. And most teams implement these workflows as the flimsiest possible thing. A bash script, a Jenkins job, a GitHub action with a bunch of steps.

Or worse, a runbook that assumes the human will do all of the careful stuff correctly while the world is on fire. Durable execution is basically the adult version of that. It's saying a workflow engine should remember state. Handle retries correctly, persist progress, and be able to resume without you guessing which steps happen. Here's the steal this now version. Pick one operational process that you hate.

One that flakes or takes forever or depends on a hero. Something like rotating certs across services, rolling a new AMI, doing a blue -green cutover, draining and replacing nodes, promoting a release across regions, migrating a database with guardrails. Now ask, if this process fails halfway through, can we resume cleanly? Or... Do we restart and hope we don't double apply something? Do we know which step we're on?

Or do we just see job failed and now everyone is scrolling logs? Are the steps item potent? Or are we creating duplicate resources and then cleaning up later? If you answered no to any of that, you're already in the category Netflix is talking about, just at a different scale. And you don't need Temporal specifically to adopt the mindset. Temporal is one implementation.

The mindset is treat critical ops like software, not like rituals. Make the workflow explicit. Make it resumable. Make it observable. Make it safe to retry. And then... Build it so the average on -call engineer can run it without needing the one person who wrote the original script three years ago. The reason this is such a big deal in 2026 is because automation is accelerating.

Agents are going to write code and open PRs and trigger pipelines faster than humans can reason in real time. So if your underlying workflows are fragile, You are basically giving a faster engine to a car with bad brakes. That's not progress, that's just more dramatic failures. This is why Story 3 matters. It's about making ops workflows reliable enough that speed doesn't kill you. Alright, lightning round.

First, Kubernetes Dashboard. The Kubernetes Dashboard project is archived. And no longer maintained. And the SIG UI note basically says there aren't active contributors and maintainers to keep it going. They recommend Headlamp as an alternative. And the repo itself is now marked archived. This is one of those reminders that ops dependencies die.

If your org still relies on Kubernetes dashboard as part of your standard workflow, you should migrate now while you're calm and not later when your security team shows up with a clipboard. Next. Docker Hardened Images. Docker's pushing a story that hardened images are now free and open source under Apache 2 .0.

Positioned as a secure, minimal, production -ready set of images with stuff like SBOMs, Providence, and SVE transparency baked in. You don't have to buy the entire pitch, but the direction is right. Containers are supply chain. Base images are supply chain. If you're still building on random community images and hoping for the best, you are basically taking on risk you don't need. And finally, PipeDash.

It's a desktop app that aggregates CICD pipelines across providers so you're not bouncing between GitHub Actions, GitLab CI, Jenkins, etc. This is a small quality of life thing, but I love anything that reduces where do I even look during incident weeks. For the closer, here's the human thread across all of this. We are entering a phase where speed is cheap, automation is cheap.

Agents will make changes faster than humans can keep up. The hard part is designing systems where speed doesn't turn into security incidents, budget blowups, and we don't know what just happened outages. N8n is the story of speed without perimeter thinking. AWS Capacity Blocks is the story of speed without cost reality. Netflix with Temporal is the story of speed with the boring engineering to make it survivable.

And that's the mindset I'd take into 2026. Don't just automate. Make automation resumable. Make it observable. Make it least privileged. Make it safe to stop. Because if you build a faster machine without brakes, it's not going to fail gracefully. It's going to fail loudly. Alright, that's it for this episode of Ship It Weekly.

We covered the critical N8N vulnerability and why workflow automation tools are basically control planes now. We hit AWS raising capacity block pricing for GPU workflows and why cost is part of reliability. And we talked about Netflix using Temporal. And the bigger pattern of durability execution for cloud operations.

The lightning round was Kubernetes dashboard getting sunset, Docker hardened images going free and open source, and a pipeline dashboard aggregator. If you got something out of this, hit follow or subscribe wherever you are listening. Links and show notes are on shipitweekly .fm. And if you want to come on for a guest episode, hit the contact form on the site. I'm Brian, and I'll see you next week. Thank you.

Thank you. Hey, I'm Brian Teller. I work in DevOps and SRE and I run Teller's Tech. Ship It Weekly is where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. If something's just hype, we'll call it hype. If it changes how you operate, we'll talk about it. Most weeks, it's a quick news recap.

In between those, I drop interview episodes with folks across DevOps, SRE, and platform engineering. Today is one of those interviews. Today, I'm joined by Danny Teller. He's a DevOps architect at Tipalti, and we're going to talk about internal developer platforms and why he thinks we're headed towards more internal IDPs instead of just backstage everywhere. Danny, can you introduce yourself? Yeah, definitely.

Thank you very much, Brian. Well, thank you for having me, first of all. As I stated, I'm the DevOps architect for Tipalti. Tipalti is a... Rather a medium -sized business company that is in the fintech world. And most of the things that I handle is working with our developer teams and our architects to kind of steer the direction of the company from not a software perspective, but a DevOps perspective.

And that also includes platform engineering because we can't have just one of them anymore. So you got to go with both. Yep, for sure. So one of the things that made me want to reach out to you was about this talk about IDPs. And I'm curious your thoughts on...

Idps like what what are your thoughts on where do you think idps are headed for everyone else internal developer platforms I know idp is an acronym for for many different things uh in the devops identity it could be identity it could be yeah yeah what's your thoughts well it's obvious that in recent years there's been a rise of platforms in any company you can take backstage for example which was open sourced and

Everybody can take it and use it as they see fit And then in recent years, we had rising companies like, what are they called?

Northfolk, I think, or Northflank. And you have Port .io, which are offering a huge integration suite. And so on. And many companies that started to buy into platform engineering. There were already platform engineering teams across different companies, but they were doing something internal. Some of them transitioned from DevOps into DevX, which are more...

Program oriented rather than ops oriented and they started picking up things like backstage and introducing it to the organization to you know simplify the way developers work because one of the core things you have in platform engineering is reducing everything from developer perspective from operations perspective and developers lifetime to literally nothing so they have nothing to worry about but the golden

Standard can be like he types in something and into chat and he gets everything done behind the scenes that's kind of like that golden mantra, if you will.

So we have all that rising every day. And that's why things like Backstage are highly adopted and have a great community. That's why we have great products like Port .io and other ones and so forth. What do you think changed that makes internal tools more realistic now? I think with all the changes recently in the AI world, in Gen .AI. It kind of simplified how we view IDPs.

If you're talking about last two years, then we've been constrained to companies and open source solutions. So if you take Backstage, which is a great product when it works, which is a great sentence to say. So when it works, it's great. But when something breaks because it's so tightly knit together with Node .js and TypeScript and all the... Stuff it's written in, it makes it very difficult to maintain.

So unless you have developers that specialize in that and also understand infrastructure, then you can't actually maintain it. Every little thing breaks it. You always have to rebuild the image and it makes it very difficult to manage. Even if you look at Backstage, Portal Backstage, that recently launched from Spotify, the SaaS offering, this is what they do behind the scenes.

They explicitly said that whenever you make a change to Backstage, we rebuild your image. So they took what you're doing already and just call it a SaaS platform. And it's written just in their docs. It's a little funny.

I don't know if the docs are still written this way, but I do remember Spotify saying too that they recommend at least like six full -time developers managing backstage, which is just cost prohibitive for most companies. I mean, they just can't afford. But that level of staff to run an internal platform. Yeah, I mean, the DevEx team should not just be doing backstage.

They have to also develop more internal tools for the company. I mean, whatever DevEx engineers do is usually with creating, you know, like the pre -commit hooks, for example, like they were created. I mean, it's kind of a thing for platform engineering to have or DevEx developer experience to have. So this is what they do day in, day out. And then came stuff like IDPs.

And they made things a lot easier because you have a central control plane that can be integrated into virtually anything. And you can see a developer just working with one single pane of view instead of tens, which include metrics, traces, monitoring, deployments, pipelines, so on and so forth. So it's great. The idea is great. But then you're encountering a bigger problem. And that's when you start to manage it.

And because these teams are very... Oriented in writing it, that's great. But now that you have Gen AI, you can kind of cut that in half. Even your platform engineers, even your classic DevOps engineers, they can do it on their own because they are familiar with the tools like Backstage. They are familiar with tools like kubectl. They are familiar with Terraform, so on and so forth.

So even DevOps that are not trained as software engineers do know how these things work. So specking it out with tools like SpecKit or creating plans with Cursor or Cloud Code and Skills becomes very easy. Instead of, so you can just ask it, okay, go ahead and study how Backstage works. Let's create a Pythonic Backstage instead. Makes it also just as easy to do. Yeah, that makes sense.

So what do you think Backstage got right? I think Backstage as a concept, it's great. I think it's really, it's one place where you just throw many integrations. And it just works. And people have one centralized location for that. I mean, they got that right. The idea is great. Condensing also tech docs in there, which is my favorite feature in backstage. I love that feature.

And the tech radar that they have there, it's just great. It's a great user experience. When it works and you see it live and everything, you just want to buy into that. But when you start managing, that's when you lose it. So can you give me an example? Like, what did you build and what problem exactly were you solving with that build out? Yeah, so, well, we at first tried working with Backstage. We did.

We ran for a whole year. But given that we've had many projects in the same time, so managing Backstage kind of fell apart. And we had a couple of our three engineers trying to manage the backstage as a whole and doing their own thing in developer experience as well. And it just fell short. So you can maintain that for 350 developers. It's insane.

And that's not including the DevOps team, which was also trying to help at the time. So even though we integrated stuff into that, we tried to work with it, it just proved to be a maintenance nightmare. So it dropped off. And then you start looking into other products. You see like Port, like I mentioned, and other ones like Northflank and so forth. And great. So they can sit in your system.

They can integrate their SaaS platform. But when you start seeing how they work, when you start seeing if you can do it on your own, because the plugins are already there. They're already online anywhere. And even if you point an LLM to certain plugins and tell it to learn it and then recreate it, you can just do it with zero maintenance. It becomes super easy.

Barely an inconvenience yeah I so at my last company we actually ran roadie for a while which is another sas backstage like offering and I found that it was good at helping us get set up quickly but beyond that um and we were very immature to the whole idp process at the time this was a new experience for us we were just moving from a devops team into a platform like devex experience team but I did find that like I I

Don't really know what it really bought us at the end of the day and it may be a better product now this was a few years ago but I don't know what it really bought us above what backstage could offer or or any other like you mentioned another sass um in the adoption yeah definitely I mean you don't know you can't tell until you actually try it but for us anyway all these products they didn't they just fell short so

We decided to try something on our own because well At the end of the day, through IDPs, I'm supposed to be able to provision whole environments per se.

I'm supposed to be able to see every other environment that I have as a developer, even up to production and so on and so forth. So in the initial stage, we decided to drop all the other ones and focus on the real pain for our developers. And that was actually developer environments. It's something that's been brewing in our company for a very long time. We've been trying to do it for a very, very long time.

And it proved to be very difficult. Because of the nature of FinTech and all the stuff that you need to work with and maintain all the time, it just becomes a small developer environment becomes very, very large. So we tried to minify that as much as we could. And eventually we got to something that you can work with. It wasn't 100 % of the entire product, but 85 % is already good enough.

I mean, missing a couple of features, but it's already something a developer can use to work with. So we went with the classics. We decided to go with Argo CD, cross -plane, and all the infrastructure is going to manage it in the first stage. And that's going to start with that. You just commit a normal YAML and you get the entire infrastructure in your backend. Everything.

You can gain all the databases, all the stuff that you need to run the application smoothly. You get it out of the box. Everything is prepared using best practices from Platform Engineer. Golden images, golden charts, if you will. Networking, all clusters, standalone clusters, not namespaces. Even monolithic machines if we have to. So everything is bundled together.

So in about 30 minutes, you can spin up a whole semi -product environment. And the next stage after that is, well, you know, connecting an AI to that. Because if today I'm spinning an entire flow, everything the system has, I'm just saying, go ahead and use it.

So we're trying to play with various ways to see how we can ask an LLM like Claude or ChadGBT through Slag and say, listen I need this and this xyz flow go ahead and create that for me so instead of creating the entire flow which is say five to six machines including any uh kubernetes cluster and some old stuff legacy machines we can dumb it down to something really really small and the way to do that is playing with

Llm you have to make sure your model can actually access all these flows and to see them and compile exactly what it needs and then request that from from crossplane in our ground So this is something we're toying around with to see how we can do that.

But we have kind of success in that area because we've developed quite an idea for that. And it proves to be very useful for now. So it goes great, really. I don't mind sharing it. It's fairly straightforward. Is it just an MCP server that can reach out to Argo and kubectl? So pretty much behind the scenes, you have the Slack bot, which connects to Cloud or Bedrock or any other one. In turn, it does use an MCP.

So that MCP actually is a tool that calls to GitHub, and it kind of rags all the repositories back to itself using a small depth. Or actually, it just takes a specific file. That specific file is kind of an XML structured file that says, this is the repository. This is what it contains. This is the flow. This is the database needed. These are their names.

This is kind of what this repository represents and how it works. So when we use requests, I need payments flow, for example. Then it's going to pull all these repositories. They're actually relevant repositories because we have a RAG map of all of these repositories. It just pulls all these relevant files, piles that into a list of flows, repositories and flows.

And after that, it just communicates with using another MCP with Crossplane and Argo, submits a YAML that it needs, and then provisions it for the developer. Very cool and interesting. So going back to these developer environments that you set up, I'm assuming they're ephemeral in nature, like because you're using Crossplane to provision everything. Yes, we're using also a TTL controller as the annotation.

Basically, you get an environment from 8 to 5. That's it. And if you need to extend it, you can. Interesting. Are you using Terraform to provision the initial infrastructure or is everything through Crossplane? Everything is Crossplane. Okay. Interesting. And why did you choose that approach? Is it just to be closer to the actual application in Argo?

Yes, I wanted to use Argo as much as I could because it just makes sense. You already have the provisioners within Crossplane. So I don't need to toy with anything else. And eventually, if the developer in the first offering just submits a YAML file, which is, what, five lines long, then already they have something reduced instead of just provisioning through a pipeline and everything.

Just git commit and you get an environment. It's far simpler, far better, and they always get the same thing. That's very cool. So, okay, backing up again. Sorry, going back to the developer platforms and this new approach that you, you know, you went the SaaS way, it didn't work. You decided to do this internal thing. How did you get adoption from the overall engineering team? It's quite easy. Okay.

It's quite easy because it's a pain that we had to solve. So this is a major, sorry, even pain in the ass of developers since they couldn't have normal developer environments. So once you sell something that gives them exactly what they need. Which is environments, so it's kind of easy to roll them in. So they have one place. They're not even looking. All they get is this environment that they're connected to.

They have their own Argo CD set up in that environment. They can provision whatever they want. And the second phase, which will connect the LLM, will be even simpler for them. So every developer today wants to work with as less screens as possible. They just want to stick to their VS Code, their Visual Studio, JetBrains, whatever, and just stick their nose in there all day long.

They don't want to look at metrics and stuff. Just query this. I want this from QL to tell me if there's a problem in my app at 2 p .m. To 1 a .m. That's what they want.

So I think that it's easier to maintain that locally using a DevOps team, platform team, or even if you want to go even further in AI ops team, because I do believe that using things like agent core and bedrock is going to be the future of these things. You're not going to see more. More of graphs lying around or backstage lying around.

Because if you can get it through text, and that text can also generate an image for you, which is pretty damn good, by the way, for now, from nano banana, so on so forth. So you're kind of losing the entire idea of having platforms like backstage or other ones, all you need to worry about is integrations. That's interesting. That's an interesting idea. I'm curious, what was the biggest surprise with this approach?

Was it the tech? Was it the ownership model? Was it like change management? Was there anything that was a surprise, I guess? No, actually, there were no surprises. We planned it thoroughly because we knew what we wanted to achieve, how and when. And the biggest thing is what we're struggling with the most is what most people are struggling with. And that's fine -tuning the models to get exactly what you want.

So we're constantly working also with AWS engineers because we're also working with Bedrock all the time and we're leveraging that. We're kind of back and forth with them all the time for the support. And it is working. We are getting better and better at achieving what we want.

So, okay, let's say a listener was listening to this and they thought the internal integration aspect is really cool, but they've never set up an IDP before. They're deciding between...

That approach or or backstage what what signals would push them one way or another in your opinion I think it's maturity definitely maturity because I can also vouch for my for what he did and myself as well that we started off like everybody you go with things that are well known and instead of creating something on your own you want to test these waters first because maybe you'd be surprised that you're using

Backstage in such a small manner that maintaining it does give you the benefit of everything you need and you don't change it as much you maybe update a version here and there but you don't need all everything it can do or if you go and you find that you work with companies that give you the idp experience and that's also good enough for you and you can you can stick with that and it's great I mean if it works for

You then stay there but when you get to these points where everything they give you is just not enough that's when you start working by yourself That makes sense.

So what's one piece of advice that you would give like platform or DevEx teams that are wanting to start an IDP come 2026? I would actually leverage as much as AI as I can with that and also work with AI as best as I can. I mean, even the recent features of Cursor that they support skills from Claude or using Gemini CLI or knowing which model.

Is best for what kind of work after I get this through this hurdle and I understand what's going on and how to use it I will then go after idps I unders I will have the models understand and map out how they work and then start fixing my own ideas around that very cool all right let's shift gears a bit I wanted to talk about talk about two other things but uh that were non -idp related uh one was your mongo atlas I

Think it's called m atlas yeah yeah the mat list kind of curious what caused you to build this and was it because like terraform operator limits weren't enough or like what what was the reasoning like what were what pain point were you solving frustration okay yeah so the story starts with uh mongo when we started working mongo we had our terraform modules everything was up and running great but at some point the

Drift was just too large to maintain from um from Terraform perspective, from Terraform, the Mongo modules that we built ourselves, and what was actually within our Mongo Atlas cloud.

So that was a little difficult to manage. So we said, okay, we have all these resources, and doing Terraform import is not a nice way to do it. It's error prone. So we decided to let that go and see if we can do it in a different way. We tried exploring the Atlas CLI, but it gave us only a management perspective creation. Creation experience. It was very difficult to work with.

And then we decided to try the Kubernetes operator. And then we found two big issues. You can't own existing resources like you do in Terraform import. So if you create a similar YAML with a similar name resource and everything, it just says, no, it exists. You can't do anything with it. And the other part is it was semi -limited to what we needed to do.

So given all these issues and the lack of abilities to work back from existing infrastructure, it kind of led to the creation of a MATLAS. MATLAS kind of tries to combine the user experience from kubectl and also the abilities of Terraform. So in a nutshell, you can basically say, okay, I've got this organization in Mongo Atlas. I want to map all the infrastructure. All I have to do is hit a button.

And well, I'll hit a feature. It's called Discover. And it just pushes everything into a YAML file. And moreover, you can transform that YAML file using MathMath as well into a workable YAML. Ah, interesting. That's cool. Yeah. And since I don't rely on states at all, because I thought that it would be counterintuitive.

So if you change something because a lot of people may go on to atlas because of a problem or whatever and start changing settings because it does happen it's a database you don't know what's going to happen the next day so I left that part intentionally out so if you change something online you can just update that yaml again with the discover feature and have the same thing already as infrastructure as code if you

Apply something and then you can also do it in a surgical manner like a partial reconciliation so I don't break everything that's already existing in mongo atlas so people can find matless on github and they just search up okay awesome yeah definitely I mean it's kind of I've recently uh read from platform engineering labs that they have this tool called form a or form I and it kind of does it it's it's a tool

Written in pkl in app in apple language and the idea behind it is great it's kind of just trying to remove what terraform does and also do the partial reconciliation and the discovery and everything so when I read about that it reminded me of like I'm pretty much doing the same thing just in go and not in pkl interesting I have not heard of that tool interesting very cool it's fairly new it's like six months old I

Believe and then uh so lastly I wanted to just touch on this a little bit you are a a golden jacket uh holder I'm curious what that process was like going through all the certifications well it was fun and it was tedious at the same time But there was one aspect of it which really frustrated me a lot.

And I do hope that AWS will listen to that and do change it in the future because they do have the capacity to do that. One thing that really pissed me off about the exams is they're really outdated. That's the one thing that really hits me every time. For example, you have the DevOps engineer exam, which I did that about earlier this year.

When I already did that exam, the code commit service was already taken offline. Yet the exam had five to six questions around that. So why are you asking me questions about a service that no longer exists? Oh, that's frustrating. Yeah. Yeah. So, and the thing that also bugs me on that is they constantly say, yeah, go to skill builder and read the docs all the time. Read the docs. Okay.

The docs said the service is not existing anymore. So why should I learn about it? Yeah. Well, then you have situations like CodeStar going away and then changing. There's a lot of inconsistency. I agree with that. Yeah, and that's really the real frustrating part. But apart from that, it's fun to do.

I mean, it does give you a lot of perspective on AWS services and how it works, especially when you read the docs and try to understand further than what's actually asked about in the exams. And it's great. I mean, it's a great feeling when you finish all of that and, you know, kind of just putting it behind you. Well, congratulations. I mean, that's an awesome accomplishment.

What recommendations would you have for listeners that maybe want to head down that road of trying to pick up all the certifications? I would start from the very lowest ones. I mean, it's true what they say about the discounts. So start from the very first ones and get that 50 % discount for the next one. And it does train you for the other exams because you get to understand how these exams are structured.

Yeah, of course, you can go online and find these dumps or whatever people really have out there. You can do that. But once you really do these exams and even fail once or twice these exams, you understand the structure and it's quite easily understandable for the next time you go ahead and do that. Yeah, very true. Very true. All right. Anything you'd like to leave our listeners with? Any recommendations?

Yeah, actually, I would. Yeah. I mean. There's one thing that's been on my mind quite lately, and it's regarding for all platform engineers, developers, DevOps teams, literally quite everyone. And it feels that with all the rise and changes in Gen AI, a lot of people think that they're staying far too behind.

And I believe that because of these changes, we should strive to really get our knowledge up and running in our heads far faster than we usually should. And yeah, subscribe to these magazines, TLDR, so on and so forth, and read more articles and so on. So because it does give you the benefit of staying ahead of others. And you need to do that not because of job security, because it's just that interesting. Yeah.

Yeah, I think anyone that's in SRE, DevOps, platform engineering. You have to have a thirst for knowledge because you're always inundated with different tools, CI, CD processes, different ways of doing things. And if you're not staying on top of that, then you really are going to be behind. And the thing I always come back to is, you know, I do a big process or I'm rolling out a new AWS account, a new way of.

Provisioning basically a new poc that I'm doing what I did five years ago for that poc is not what I would do today I would architect it completely different than I would have just five years ago maybe even two years ago so yeah you you kind of need to always have that that thirst and hunger for learning and just staying on top of new things. You can't stay on top of everything, obviously.

I mean, that's also, you know, the reality is imposter syndrome does exist. You can't know everything and that's okay. That's why they're teams. That's why we're on teams and we're collaborators, you know, but just trying to be present, I think is really important. Yeah, most definitely. Awesome. Well, where can people find you online? Mostly I'm on LinkedIn. So that's the best way to reach out to me.

Some people already have, so. Awesome. So Danny Teller on LinkedIn, if you want to reach out. I'll also leave your MATLESS GitHub link in the show notes. And thanks for coming on, Danny. Really appreciate it. Thank you for having me. I had a great time here. Thanks, Brian. Thanks. All right. That's the interview with Danny. Quick reminder on the format.

Ship It Weekly is still the weekly news recap, and I'm dropping these guest convos in between. If you want to catch both, hit follow or subscribe wherever you are listening. And if this episode was useful, share it with a platform or DevEx friend and leave a quick rating or review. It's annoying how much that actually helps the show. We'll be back later this week with a regular news episode. We'll see you then. Thanks.

You can automate more than ever right now. Agents, pipelines, self -healing systems, all of it. But if you don't build brakes and blast radius controls at the same time, you're just speeding up the failure too. That's the theme for today. Hey, I'm Brian Teller. I work in DevOps and SRE and I run Teller's Tech.

This is Ship It Weekly, where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. If something's hype, I'll call it hype. If it changes how you operate, we'll talk about it. Happy New Year! It's January 2nd, so if you're listening on a walk, cleaning up a lab, or just avoiding thinking about Monday yet, welcome.

Quick housekeeping, the site is shipitweekly .fm. That's where the links and show notes live, plus the contact email. Also, I'm looking for interview guests. If you're building real infra, platform, SRE, DevEx, governance, migrations, any of it, and you want to come on for a chill conversation episode, hit the contact form. On shipitweekly .fm.

And one quick ask, if the show has been useful, follow or subscribe wherever you are listening. And if you've got 10 seconds, a rating or review really helps way more than it should. All right, three main stories for today. Cloudflare's Code Orange resilience plan and their whole fail small approach. It's basically the internet's biggest edge provider saying we're changing how we ship and how we contain blast radius.

Second, Pulumi making a very direct move at Terraform and HCL. This is about the IAC control plane. Governance, and what the next year looks like for teams trying to standardize without rewriting the world. Third, Meta's DRP, a root cause analysis platform at scale, not we bought a tool. More like we turned incident investigation into software. Then we'll do a lightning round with a few practical updates and tools.

And we'll close with a human story about the ironies of automation and why faster systems can actually increase the burden on humans if you don't design the oversight part. Let's get into it. So Cloudflare published a post about Code Orange and Fail Small. And the vibe is basically, we've had enough of big correlated failures. And I love that they're saying the quiet part out loud.

When you run global infrastructure, a lot of the stuff that hurts you isn't a single server dying. It's one config change. One policy tweak, one rollout mistake, and suddenly you have the same failure everywhere at once. That's the nightmare. Correlated failure. So fail small is the strategy to force problems to stay local. If you ship a bad change, it should not be able to take down everything.

It should hit one slice, one region, one cell, one shard, one whatever, and then you learn and stop it. That's the ideal. Now, the reason this matters for normal humans like us is because this is the same exact pattern inside companies, just scaled. Down. One shared Terraform module change breaks 50 repos. One CI permission tweak breaks every pipeline. One harmless IAM policy update knocks out a production deploy.

One bad external DNS change takes out half your ingress. You know the vibe. Cloudflare's post is basically a reminder that blast radius control is a feature you have to design and defend. It doesn't appear automatically. And it's also a reminder that reliability work often starts with how do we ship changes safely, not how do we add more dashboards. If you want something...

Actionable from this, here's the mental model I keep in my head. If you cannot confidently say this change will only affect a small part of traffic, then you need better segmentation or better rollout controls. That might mean canaries. It might mean progressive delivery. It might mean feature flags that can be disabled quickly.

It might mean splitting a giant shared module into smaller modules so one change doesn't cascade. But either way, the goal is the same. Make failure smaller than your ability to recover. And going into a new year, this is a good story to start with because everyone comes back from the holidays with energy and plans and big changes. And the fastest way to ruin January is to ship a big change with a big blast radius.

So that's story one. Cloudflare is doing the grown -up version of build guardrails for change. It's not glamorous, it's mandatory. All right, story two is a little more control plane politics. Pulumi dropped a blog post that's basically a direct play at Terraform and HCL shops. If you zoom out, this isn't just here's a feature.

It's them saying you can manage your Terraform and your HCL inside Pulumi's ecosystem, and you can migrate without rewriting everything, which is smart. Because the big friction for most platform teams is not that Terraform is bad. It's that standardization is hard and change is expensive. A lot of teams want stronger guardrails, better policy enforcement, better visibility. Better workflows.

They also do not want a two -year rewrite project where you rebuild everything in a new language. So Pulumi is trying to reduce the rewrite tax. They're basically saying, keep your existing Terraform, but move the management layer. Use Pulumi Cloud as the place you orchestrate runs, policy, approvals, drift. Governance.

And they also mention native HCL support inside Pulumi's IAC, so HCL isn't treated like a second -class citizen. This is why this story matters for your audience. It's not about who wins IAC. It's about control planes. In 2026, the platform conversation is increasingly about who owns the workflow and guardrails around infrastructure changes, not which syntax you prefer.

And it ties directly to what we talked about in episode eight with Cloudflare's IAC governance and shift left patterns. Same idea. You want infrastructure changes to be reviewed, tested, policy checked, and auditable before it touches production. Now, if you're a Terraform shop, I'm not telling you to run to Pulumi. But I am saying this is worth watching because it's a sign of where the market is going.

Teams want guardrails. They want policy. They want better workflows. They want less state pain. They want less secrets leakage. They want less, oops, I applied the wrong workspace. And the vendors are fighting for that control plane. Practical takeaway here.

If your IAC workflow is still people run applies locally, or we have a fragile CI pipeline that's half tribal knowledge, then you're going to feel this pressure all year. Whether you solve it with Terraform Cloud, Spacelift, Atlantis, GitHub Actions, Pulumi, or something homegrown, the point is the same. The workflow is the product. Alright, Story 3 is my favorite kind of SRE story.

We built a system to do the thing humans keep doing manually. Meta wrote about DRP, their Root Cause Analysis Platform. And the idea is actually pretty simple. But the execution is what makes it interesting. When an incident happens, teams do the same basic investigation patterns over and over. You check recent deploys. You check configuration changes. You look for correlated alerts. You compare time series.

You scan logs for known signatures. You look for dependency failures. You try to answer the same questions every time. And that's slow. It's manual. It depends on who's on call. It depends on tribal knowledge. So Meta built DRP to codify investigations into analyzers. Think of an analyzer as an automated investigator that knows how to pull signals, correlate timelines, and produce a hypothesis.

They describe it as a platform that runs these analysis at scale. A lot of teams, a lot of analysis, all the time. Even if you ignore the big headline numbers, the approach is the real story. SRE, maturity, isn't just more runbooks. It's turning repeated operational work into software. This is the same idea behind good incident tooling in general. You don't want hero debugging. You want systematic debugging.

You want the first 15 minutes of an incident to be consistent no matter who's on call. DRP is basically that idea productized internally. So what's the takeaway for normal shops? You don't need a meta -level RCA platform, but you can steal the pattern. Take the top five recurring incident types you see. The stuff that keeps happening. Then ask, what are the first three questions we always ask during those incidents?

Then automate those questions. That could be as simple as a Slack bot that pulls recent deploys for this service, last config changes, related incidents, top error rate deltas, and posts it automatically in the incident channel. Or a runbook that starts with real queries and links, not vague instructions like check the logs. Or a...

A simple script that generates a timeline, pulls the right dashboards, and collects the evidence. You don't need to solve everything. You just need to stop doing the same manual steps forever. This is a great new year story because it's a reminder that SRE work is supposed to reduce human toil, not just respond faster to pain. Alright, lightning round. First up, AWS ECR can now create repositories on a push.

This is a small feature that can save a lot of time in orgs that spin up a lot of services, environments, or ephemeral projects. Instead of manually creating the repo first, you can push and ECR can create it. But, as always, convenience is a governance problem in disguise. If you let anyone auto -create repos with any naming pattern, you'll end up with a junk drawer.

So you probably want guardrails around who can do it, what names are allowed, and what defaults get applied, like scanning, lifecycle rules, encryption, that sort of thing. Next, GitHub put out a Let's Talk About GitHub Actions post. I like these when they're less here's marketing and more here's how the platform is evolving.

It's worth skimming if you live in actions and you want to understand where they're investing. Especially if actions is your company's CI backbone. Because small improvements in reliability and performance turn into real saved time during incident weeks. Quick callback. Episode 6 covered the runner pricing pause. Same takeaway still holds. Actions isn't just CI. It's a control plane.

Next, Lauren Hochstein had a great SRE -ish post about incident data and statistical process control. The short version is, MTTR is usually not under statistical control, which means the average can be misleading. If you've ever had one weird, horrible incident blow up your average, you already know this. The point is not stop measuring.

The point is don't drive the car by staring at one metric that gets yanked around by outliers. Look at distributions, look at trends, and focus on repeatable improvements. Next, Drift Hound. This is a tool aimed at drift detection and visibility for terraform.

The reason it's worth mentioning is that Drift is one of those problems that gets worse as your infra footprint grows, and it quietly undermines IAC as a source of truth. Even if you don't use this tool specifically, it's a good reminder to revisit how you detect Drift and how you respond to it, because we'll notice eventually is not a strategy. And last, Superset.

This one is interesting because it's about running multiple coding agents locally in parallel, like a manager for AI agents concept. Useful, but also a good reminder that agentic tools amplify both speed and risk, which is a perfect segue into the human story. For the human closer, I want to pull from Uwe Fredersen's AI and the ironies of automation.

The core point is something we keep learning the hard way in operations. Automation doesn't remove responsibility. It moves responsibility. If a system gets faster, the human oversight problem gets harder, not easier. Because now the system can do more things faster, including the wrong things. And the human has less time to notice.

Less time to understand and less time to stop it if you've ever been on call during an automated rollout that goes sideways you know this feeling you are watching the machine do damage at a speed you can't match so the real job isn't just add automation it's design the control loop how do you detect that automation is going wrong how do you slow it down How do you contain the blast radius?

How do you make it observable so humans can understand what it's doing? And how can you make rollback as easy as rollout? This ties back to everything in this episode. Cloudflare's fail small is literally about making automation safer by limiting correlated impact. Pulumi's control plane move is about controlling the workflow.

Policies and safety rails around changes meta's drp is about automating investigation but in a way that makes humans faster and more consistent not replaced and blind and the agent tools in the lightning round are about the new frontier of this if you give an agent the ability to write code open prs run ci or touch infra you need to treat it like a junior engineer with a caffeine problem fast, helpful, and absolutely capable of doing something dumb if you give it too much freedom.

So going into the new year, here's the mindset I'd keep. Automate? Yes. But build the brakes at the same time. Alright, that's it for this episode of Ship It Weekly. We covered Cloudflare's Fail small push. Pulumi's move to manage all of your IAC, including Terraform and HCL. And Meta's DRP platform for automated incident investigations. The lightning round was AWS ECR auto -creating repos on push.

GitHub Actions platform direction. A better way to think about incident metrics, drift detection, and a quick look at multi -agent tooling. If you got something out of this, hit follow or subscribe wherever you are listening. And if you can, leave a quick rating or review. It helps a ton. Links and show notes are on shipitweekly .fm. And last reminder, I'm looking for interview guests.

If you want to come on and talk through real DevOps, SRE, platform engineering, cloud governance, migrations, incident scars, whatever, hit the contact form on the site. I'm Brian. Happy New Year, and I'll see you next week.

Hey, I'm Brian Teller. I work in DevOps and SRE and I run Teller's Tech. Ship It Weekly is where I filter the noise and pull out what actually matters when you're the one running infrastructure and owning reliability. If something's just hype, we'll call it hype. But if it changes how you operate, we'll talk about it. Most weeks, it's a quick news recap.

In between those, I drop interview episodes with folks across the DevOps world. This is one of those interviews. Today, I'm joined by Eric Pate. He's a cloud and DevOps engineer. He has a background in building and running enterprise web and ERP systems. And he has a really grounded approach to leveling up through hands -on projects. Eric? First off, Brian, thanks for having me. I appreciate it. Yeah, no worries.

I'm excited about this conversation because my DevOps journey. Definitely hasn't been a straight line. I come from a software engineering background. I have spent years building web applications. And I must say that at some point in my career, I realized I was fascinated about cloud technology and how things run in the cloud beyond just shipping code, you know. I'm looking forward to sharing what has worked for me.

What hasn't and hopefully a few lessons that can help someone else who is on a similar path as me. I appreciate it. That sounds awesome. It's interesting because I started as a developer, I guess, too. I worked...

Tech support but then I did some like early web development and kind of jumped into devops kind of a lot of the same way a different industry but um a lot of the same way you did so that's very exciting exactly brian so can you just tell us who you are and what you do day to day well like I said I'm a cloud and devops engineer with expected in AWS, Linux, CI, CD automation, containerization, infrastructure as code.

Basically coming from a software engineering background. So I principally develop web applications and then ship through cloud technology. And yes. Oh, that's very cool. And so what system, I guess most of the systems, are they AWS based like most of the cloud systems or? Yes, they are. So coming from my previous background. All right, so we develop PHP -based applications.

And previously, we host the applications on web servers, not cloud platforms, so to speak. But then with the advent of cloud technology and the thing in Android cloud, there has been the need to migrate from traditional deployment methods to cloud technology.

What made you realize when you were doing this full stack web development, what made you realize that writing code is only part of the picture and you wanted to dive into DevOps as well? Right. So let me say this, that I found myself more curious about modern methodologies and technologies around code deployment and how applications scale and remain reliable in production environments. Yes. Very cool.

Being a developer to touching DevOps stuff? Was it like networking, CICD, just AWS, Linux? Like what was the part that you found to be the hardest? Well, Brian, that's a great question, Brian. How does LevelUp for me, I would say, was familiarizing myself with key DevOps concepts and with the stack of technologies that come with it, I .e.

Source code management using Git, package management using Docker, building CICD pipelines.

Using tools such as Jenkins, GitHub Actions, container orchestration using Kubernetes, you know, mastering cloud services on platforms like AWS, Google Cloud, and also, you know, getting a grip on infrastructure as code, you know, using Terraform Ansible, and of course, implementing continuous monitoring and observability with tools, you know, such as Prometheus, Grafana, and things of the sort.

I must say that, you know, beyond that, tools also another hardest level level up for me was the shift shift in mindset all right the mindset shift I .e you know moving from let me say I know how this process works on my local host to how how that same process behaves in production at scale Now, that's a big jump.

You know, that's a big jump, which requires grit and a lot of hands -on experience, hands -on practice, and also a grip on the technology that you're working with. Yes, Brian. Yeah, those are two really good points. A, there's just a lot to know. When you talk DevOps, that is just such a wide breadth of tools and technologies. And then the other side of it, yeah, testing locally, great, it works.

Then you scale that up to 10 ,000 users, 100 ,000 users, a million, completely different complexities than you're going to have running that locally. And things are going to show up that you didn't even think of when you were down in the 10, 100 early user stage. Exactly, Brian. Yeah. So was there anything that clicked first for you or were there things that took longer than expected as far as like learning like AWS?

Because like speaking of like DevOps being wide, AWS has a ton of different technologies and a ton of different.

Ways of doing the same thing so is there anything like in particular that maybe clicked earlier or or was harder to learn over time that's a great question again brian I would say this that um what really clicked for me was realizing that um devops sits at what what what I will call the intersection of you know software engineering and systems thinking and business impact what you're doing ultimately and largely impacts on business decisions.

So I enjoyed solving problems like, why is this deployment failing? Or why does this service degrade and that's a load? And how do you, or how do we automate this so it never breaks again? You know, that shift in mindset, just from writing features to owning the process, owning the system, owning reliability and delivery.

You know actually pulled me in yes so so that's it for me now brian what let me add this that what didn't click for me like you said earlier all right was the sheer breadth and depth of new tools and technologies I had to learn you know you you you are not dated with lots of of of tools and technologies to learn technologies in respect of cloud in this respect of networking You need to know the ins and outs of Linux, the ICD pipelines, observability, you know, it can feel overwhelming, really overwhelming.

But early on, I tried to try to learn everything at once. And that I would say was my mistake. All right. You need to progress slowly. You need to trust the process, follow along slowly. So progress really accelerated when I focused on the fundamentals and learned. By building, you know, and breaking things on my home lab. Exactly. Yeah, that's a good point.

So yeah, just focusing on one area and then building on that. Exactly, right. So looking back at your background, you've worked on enterprise grade systems for big organizations. What changes when it's not a side project, kind of like we talked about earlier, running locally in dev, what changes between that and like mission critical, like enterprise grade systems? Like what are some things maybe you have to worry about?

Um that you wouldn't in a local development environment all right so brian my my first instance in deploying a mission critical application was a voting was an online voting system okay for for a school cool and yes and you can imagine that the system works fine on your machine And then the moment you have to go live or go or deploy on production, all right, the results you are getting is entirely different.

Users are trying to access the system and they cannot. And you know, elections are contentious matters and that you cannot afford to let your users lose trust in your system. And so for me, that was a great learning curve for me. You really have to be prepared for such instances that you need to have some virtual environments where you can literally test what you have built and you want to deploy.

So for some mission -critical systems, I have been privileged to work with teams that have deployed online voting systems. We have deployed some enterprise resource planning platform for institutions like the food. An agri -organization of the United Nations, the Ghana office to be specific. So that platform encapsulates everything to make their business processes work efficiently.

So there are modules on procurement, modules on human resource management, modules on financial management and things like that. These are mission -critical systems which you can't afford to let them fail. Yeah. So for me, they have been a great learning experience for me practicing on or working on these systems. Yes, Brian. So what did that era teach you that actually helped you in DevOps now?

Like reliability mindset or like working with stakeholders or like change management? What was the most valuable, I guess, part of that? Brian, you literally mentioned everything that was a learning experience for me, right? From system reliability to change management.

It was a great learning experience for me that for users, once your system is unreliable, it's not available, it's not giving them the expected results, they begin to lose trust in the system. And then also the change management process is very critical and very important. There has to be user acceptance. Be user accepted.

You need to tag them slowly and make sure that they learn or they get comfortable with the system. And so lots of sensitization, stakeholder engagements, lots of consultations. All right, you run campaigns, you run some communication campaigns here and there to ensure that there is some... There is user acceptance, all right?

There's user acceptance because no matter how good your system is, if users do not accept the system, the system is 100 % bound to fail. Yeah, that's very true. Very true. Okay, so switching gears, you posted about your home lab on the go, which I believe was a Mac mini. I'm just curious, why that approach? Why a Mac mini? Mac mini, all right.

So let me just say this, that I purchased, This Mac Mini just some time ago. And I said to myself, well, why don't I just turn this into my home lab? Dedicated specifically for practicing, you know, and for honing my tech skills. So I own a laptop and I also have this home lab dedicated specifically to... Practicing my, my DevOps skills. And so with this, this platform, I am not afraid to break things down.

All right. For sure. Yeah. To feel in the, make all the mistakes. All right. To, to, um, what should I say? To make all the mistakes that, that you need to make while, while, while learning as well. Yeah. So this home lab is, is dedicated. To help me practice my DevOps skills and also to hone my tech skills as well. Very cool. So are you doing like Docker, pipelines, monitoring, like break -fix? Exactly.

So just last night, so I decided to containerize a static version of my website. All right. So I Dockerized the website, all right, built the image on that platform. All right. I pushed that Docker image to Docker Hub, right? And then I went on also to run the Docker run command, right? So which literally converted the image to a container.

And then I ran the application from the web browser, you know, to local host port 8080. And that gave... So I used that home lab, all right, to practice writing of Docker files, all right, CICD pipelines, constructing my Docker Compose files, you know, building my Jenkins files.

And so you would be seeing a post for me very shortly detailing the process that went through and containerizing the static version of my website. Very cool. So basically that home lab helps me practice without guardrails. All right. Make the mistakes. All right. Do my debugging in there. Build Docker images. Run Docker images. Run Jenkins files. You know, build my Jenkins files. Run CICD pipelines.

Push my image to Docker Hub. Pull from Docker Hub. Things like that. Very cool. Well, I'll look forward to that post. And I think, I think that just goes to show you that you don't need fancy hardware either for a home lab.

Just take what you take, whatever you have or, or what you've used previously, or, you know, if you have an old Mac mini or whatever, you don't need, you don't need like a big rack and stack server set up. Like you don't need a bunch of fancy machines. Exactly, Brian. Exactly, Brian. So shifting gears a little bit from the home lab, and we talked about this at the top too a little bit, but I'm just curious.

So we talked about how the tools are endless, right? You have Terraform, Kubernetes, Jenkins, Prometheus, Grafana, Ansible. How do you choose, how do you personally go by choosing what to learn next? All right. So this has been my structure. All right. So I begin with source code management. All right. So the basics for me is very critical. You need to be familiar with the basics.

You need to have a stronghold on the basics, which is Linux. All right. So things to do with Linux commands, mastering bash scripting are very important and very, very, very, very critical because at the core of DevOps is Linux. You cannot be a DevOps engineer. And not no Linux. Linux is very critical. So I focus on the fundamentals, which is Linux.

And then I move on to source code management tools, you know, like Git. And then I progress into package management using Docker, right? And then I progress to CICD pipelines using Jenkins. So the idea is how do I use automation, all right, to build? The Docker image and then push it to a repository. All right. Okay.

And then from the Docker, from the CICD pipelines and practice, all right, so I move on to the cloud platform. All right. So that now after pushing to the Docker repository, all right, it has to be pulled onto your cloud platform. Now we have cloud services also that can serve us as your... Your Docker repository. All right. So the various cloud platforms have services that can serve as, you know, Docker repository.

And then I move on to learning infrastructure as code. All right. But in between that one, so from Docker to the cloud platform, my process would be to learn Kubernetes next. And then after learning Kubernetes, you learn Terraform. Okay.

Or infrastructure as code so basically terraform ansible and then you move on to to to implementing continuous monitoring and observability with you know tools like prometheus and grafana gotcha so so so for me that has been my structure well some people have their structure.

But for me, at the heart of DevOps has been to learn and show that you get the fundamentals right and show that you have a good grip on Linux and then you progress from there. And then learn the entire, through the process, learn the entire pipeline and follow it through. It sounds like, right? Exactly. Exactly, Brian. Exactly, Brian. So what are your thoughts on certifications versus like...

Doing your own projects what do you think is more valuable well for certifications let I would say this that's a very dicey dicey matter yeah it is that's a dicey matter right well here is it hands down I'll see real projects would always make the big difference for anyone right certifications and courses will help build structure but running real you know, setting up CICD pipelines, deploying to cloud infrastructure, breaking things, monitoring them, fixing things, you know, that's where everything clicks.

That's where everything clicks, yes. So certifications help, all right? Like I said earlier, certifications and courses would help you build structure. But, you know... Hands down, I would say real projects would always make the big difference. Real projects and hands -on projects will make the real difference. Because you have to fail, right? You have to learn to fail and be okay with that.

You have to learn to fail. You have to learn to fail. You have to learn to fail. For me, there have been several times where I have just tried to organize a Docker Compose file. As basic as getting it to run successfully. Brian, and it's been a challenge. And so I put that file down, go to sleep, come back the following morning, and it starts to end. And surprisingly, things begin to click for me. All right.

And then when you realize that you made mistakes, oh, I made the mistake here, I made the mistake there, I made this mistake. Just two weeks back, all right, one Docker Compose file that I wrote just wasn't working. Only for me to realize there was a dash, a dash I had introduced somewhere that shouldn't have been there. And it's been a great learning experience for me.

So I would say that hands down, real practice and real projects would always make a difference. Yeah, I agree with that. I think, yeah, I think certifications, boot camps have their place, but as a foundational, a structure school, same thing, but you need that hands -on learning. At some point you need to actually dive in. Be comfortable with failing because that's okay. Like, you know, it's okay.

You're going to stumble a little bit and that's okay. So wrapping up a bit, what's one piece of advice you would give someone trying to break into DevOps in 2026? Where should they start? What should they do? All right. So what I would say is this, that let me say this, that I would just say, focus on depth and not hype. Focus on depth and not hype. So I...

I have found out that employers, you know, care more about how you think and solve problems, right? How you think and solve problems than knowing every tool there is, all right? How you think and how you solve problems, all right? Because at the heart of that all, that is what DevOps really is. Simplifying things, automating processes, and then shipping code faster and efficiently.

All right and so how you think now my my my other advice would be this that consistency would always beat intensity all right that consistency would always beat intensity you don't need to learn everything in three months you need to show up steadily all right you you need to show up you need to have a routine if you tell yourself that you would dedicate two hours each day to learning devops do that You need to show steady progress.

You need to build. You need to document. You need to reflect. You need to repeat. And it's okay to fail. It's okay to fail. Do this repeatedly. And I would say also that slow down and be consistent. You don't need to master everything overnight. Just show up. Build something small and keep going. And also one thing I would say also would be this. Learn to share your journey, all right?

Learn to write, learn to post, learn to talk about what you are learning because it's only through that process that it reinforces your knowledge. It creates visibility as well. And then opportunities often come from when people see the way you think and not just what's on your resume. The way you think, the way you solve problems, the way you handle challenges. That would be for me, Brian.

Yeah, I agree with that a lot. All right, so where should people follow you and what should they check out? All right, Brian, so people can follow me on LinkedIn and then also my personal website, which is ericpatty .com, ericpatty .com. I'm also on Facebook and also on Instagram. And so I'm primarily on LinkedIn and I have a website, ericpatty .com. Yes, Brian. Awesome. Appreciate it.

Well, thank you, Eric Pate, for coming on and being on Ship It Weekly. I really appreciate it. Thank you, Brian. It's been great. Thank you. All right. That's the interview with Eric. Quick reminder on the format. Ship It Weekly is still the weekly news recap. And I'm dropping these guest convos in between. If you want to catch both, hit follow or subscribe wherever you are listening.

And if this episode was useful, share it with a coworker or your on -call buddy and leave a quick rating or review. It's annoying how much that stuff actually helps the show. We'll be back next week with the regular news episode. We'll see you then. Thanks for listening.