Ship It Conversations: Backstage vs Internal IDPs, and Why DevEx Muscle Matters (with Danny Teller)

Transcript

Thank you. Hey, I'm Brian Teller. I work in DevOps

and SRE and I run Teller's Tech. Ship It Weekly

is where I filter the noise and pull out what

actually matters when you're the one running

infrastructure and owning reliability. If something's

just hype, we'll call it hype. If it changes

how you operate, we'll talk about it. Most weeks,

it's a quick news recap. In between those, I

drop interview episodes with folks across DevOps,

SRE, and platform engineering. Today is one of

those interviews. Today, I'm joined by Danny

Teller. He's a DevOps architect at Tipalti, and

we're going to talk about internal developer

platforms and why he thinks we're headed towards

more internal IDPs instead of just backstage

everywhere. Danny, can you introduce yourself?

Yeah, definitely. Thank you very much, Brian.

Well, thank you for having me, first of all.

As I stated, I'm the DevOps architect for Tipalti.

Tipalti is a... rather a medium -sized business

company that is in the fintech world. And most

of the things that I handle is working with our

developer teams and our architects to kind of

steer the direction of the company from not a

software perspective, but a DevOps perspective.

And that also includes platform engineering because

we can't have just one of them anymore. So you

got to go with both. Yep, for sure. So one of

the things that made me want to reach out to

you was about this talk about IDPs. And I'm curious

your thoughts on... idps like what what are your

thoughts on where do you think idps are headed

for everyone else internal developer platforms

i know idp is an acronym for for many different

things uh in the devops identity it could be

identity it could be yeah yeah what's your thoughts

well it's obvious that in recent years there's

been a rise of platforms in any company you can

take backstage for example which was open sourced

and everybody can take it and use it as they

see fit And then in recent years, we had rising

companies like, what are they called? Northfolk,

I think, or Northflank. And you have Port .io,

which are offering a huge integration suite.

And so on. And many companies that started to

buy into platform engineering. There were already

platform engineering teams across different companies,

but they were doing something internal. Some

of them transitioned from DevOps into DevX, which

are more... program oriented rather than ops

oriented and they started picking up things like

backstage and introducing it to the organization

to you know simplify the way developers work

because one of the core things you have in platform

engineering is reducing everything from developer

perspective from operations perspective and developers

lifetime to literally nothing so they have nothing

to worry about but the golden standard can be

like he types in something and into chat and

he gets everything done behind the scenes that's

kind of like that golden mantra, if you will.

So we have all that rising every day. And that's

why things like Backstage are highly adopted

and have a great community. That's why we have

great products like Port .io and other ones and

so forth. What do you think changed that makes

internal tools more realistic now? I think with

all the changes recently in the AI world, in

Gen .AI. It kind of simplified how we view IDPs.

If you're talking about last two years, then

we've been constrained to companies and open

source solutions. So if you take Backstage, which

is a great product when it works, which is a

great sentence to say. So when it works, it's

great. But when something breaks because it's

so tightly knit together with Node .js and TypeScript

and all the... stuff it's written in, it makes

it very difficult to maintain. So unless you

have developers that specialize in that and also

understand infrastructure, then you can't actually

maintain it. Every little thing breaks it. You

always have to rebuild the image and it makes

it very difficult to manage. Even if you look

at Backstage, Portal Backstage, that recently

launched from Spotify, the SaaS offering, this

is what they do behind the scenes. They explicitly

said that whenever you make a change to Backstage,

we rebuild your image. So they took what you're

doing already and just call it a SaaS platform.

And it's written just in their docs. It's a little

funny. I don't know if the docs are still written

this way, but I do remember Spotify saying too

that they recommend at least like six full -time

developers managing backstage, which is just

cost prohibitive for most companies. I mean,

they just can't afford. But that level of staff

to run an internal platform. Yeah, I mean, the

DevEx team should not just be doing backstage.

They have to also develop more internal tools

for the company. I mean, whatever DevEx engineers

do is usually with creating, you know, like the

pre -commit hooks, for example, like they were

created. I mean, it's kind of a thing for platform

engineering to have or DevEx developer experience

to have. So this is what they do day in, day

out. And then came stuff like IDPs. And they

made things a lot easier because you have a central

control plane that can be integrated into virtually

anything. And you can see a developer just working

with one single pane of view instead of tens,

which include metrics, traces, monitoring, deployments,

pipelines, so on and so forth. So it's great.

The idea is great. But then you're encountering

a bigger problem. And that's when you start to

manage it. And because these teams are very...

Oriented in writing it, that's great. But now

that you have Gen AI, you can kind of cut that

in half. Even your platform engineers, even your

classic DevOps engineers, they can do it on their

own because they are familiar with the tools

like Backstage. They are familiar with tools

like kubectl. They are familiar with Terraform,

so on and so forth. So even DevOps that are not

trained as software engineers do know how these

things work. So specking it out with tools like

SpecKit or creating plans with Cursor or Cloud

Code and Skills becomes very easy. Instead of,

so you can just ask it, okay, go ahead and study

how Backstage works. Let's create a Pythonic

Backstage instead. Makes it also just as easy

to do. Yeah, that makes sense. So what do you

think Backstage got right? I think Backstage

as a concept, it's great. I think it's really,

it's one place where you just throw many integrations.

And it just works. And people have one centralized

location for that. I mean, they got that right.

The idea is great. Condensing also tech docs

in there, which is my favorite feature in backstage.

I love that feature. And the tech radar that

they have there, it's just great. It's a great

user experience. When it works and you see it

live and everything, you just want to buy into

that. But when you start managing, that's when

you lose it. So can you give me an example? Like,

what did you build and what problem exactly were

you solving with that build out? Yeah, so, well,

we at first tried working with Backstage. We

did. We ran for a whole year. But given that

we've had many projects in the same time, so

managing Backstage kind of fell apart. And we

had a couple of our three engineers trying to

manage the backstage as a whole and doing their

own thing in developer experience as well. And

it just fell short. So you can maintain that

for 350 developers. It's insane. And that's not

including the DevOps team, which was also trying

to help at the time. So even though we integrated

stuff into that, we tried to work with it, it

just proved to be a maintenance nightmare. So

it dropped off. And then you start looking into

other products. You see like Port, like I mentioned,

and other ones like Northflank and so forth.

And great. So they can sit in your system. They

can integrate their SaaS platform. But when you

start seeing how they work, when you start seeing

if you can do it on your own, because the plugins

are already there. They're already online anywhere.

And even if you point an LLM to certain plugins

and tell it to learn it and then recreate it,

you can just do it with zero maintenance. It

becomes super easy. barely an inconvenience yeah

i so at my last company we actually ran roadie

for a while which is another sas backstage like

offering and i found that it was good at helping

us get set up quickly but beyond that um and

we were very immature to the whole idp process

at the time this was a new experience for us

we were just moving from a devops team into a

platform like devex experience team but i did

find that like i i don't really know what it

really bought us at the end of the day and it

may be a better product now this was a few years

ago but i don't know what it really bought us

above what backstage could offer or or any other

like you mentioned another sass um in the adoption

yeah definitely i mean you don't know you can't

tell until you actually try it but for us anyway

all these products they didn't they just fell

short so we decided to try something on our own

because well At the end of the day, through IDPs,

I'm supposed to be able to provision whole environments

per se. I'm supposed to be able to see every

other environment that I have as a developer,

even up to production and so on and so forth.

So in the initial stage, we decided to drop all

the other ones and focus on the real pain for

our developers. And that was actually developer

environments. It's something that's been brewing

in our company for a very long time. We've been

trying to do it for a very, very long time. And

it proved to be very difficult. Because of the

nature of FinTech and all the stuff that you

need to work with and maintain all the time,

it just becomes a small developer environment

becomes very, very large. So we tried to minify

that as much as we could. And eventually we got

to something that you can work with. It wasn't

100 % of the entire product, but 85 % is already

good enough. I mean, missing a couple of features,

but it's already something a developer can use

to work with. So we went with the classics. We

decided to go with Argo CD, cross -plane, and

all the infrastructure is going to manage it

in the first stage. And that's going to start

with that. You just commit a normal YAML and

you get the entire infrastructure in your backend.

Everything. You can gain all the databases, all

the stuff that you need to run the application

smoothly. You get it out of the box. Everything

is prepared using best practices from Platform

Engineer. Golden images, golden charts, if you

will. Networking, all clusters, standalone clusters,

not namespaces. even monolithic machines if we

have to. So everything is bundled together. So

in about 30 minutes, you can spin up a whole

semi -product environment. And the next stage

after that is, well, you know, connecting an

AI to that. Because if today I'm spinning an

entire flow, everything the system has, I'm just

saying, go ahead and use it. So we're trying

to play with various ways to see how we can ask

an LLM like Claude or ChadGBT through Slag and

say, listen i need this and this xyz flow go

ahead and create that for me so instead of creating

the entire flow which is say five to six machines

including any uh kubernetes cluster and some

old stuff legacy machines we can dumb it down

to something really really small and the way

to do that is playing with llm you have to make

sure your model can actually access all these

flows and to see them and compile exactly what

it needs and then request that from from crossplane

in our ground So this is something we're toying

around with to see how we can do that. But we

have kind of success in that area because we've

developed quite an idea for that. And it proves

to be very useful for now. So it goes great,

really. I don't mind sharing it. It's fairly

straightforward. Is it just an MCP server that

can reach out to Argo and kubectl? So pretty

much behind the scenes, you have the Slack bot,

which connects to Cloud or Bedrock or any other

one. In turn, it does use an MCP. So that MCP

actually is a tool that calls to GitHub, and

it kind of rags all the repositories back to

itself using a small depth. Or actually, it just

takes a specific file. That specific file is

kind of an XML structured file that says, this

is the repository. This is what it contains.

This is the flow. This is the database needed.

These are their names. This is kind of what this

repository represents and how it works. So when

we use requests, I need payments flow, for example.

Then it's going to pull all these repositories.

They're actually relevant repositories because

we have a RAG map of all of these repositories.

It just pulls all these relevant files, piles

that into a list of flows, repositories and flows.

And after that, it just communicates with using

another MCP with Crossplane and Argo, submits

a YAML that it needs, and then provisions it

for the developer. Very cool and interesting.

So going back to these developer environments

that you set up, I'm assuming they're ephemeral

in nature, like because you're using Crossplane

to provision everything. Yes, we're using also

a TTL controller as the annotation. Basically,

you get an environment from 8 to 5. That's it.

And if you need to extend it, you can. Interesting.

Are you using Terraform to provision the initial

infrastructure or is everything through Crossplane?

Everything is Crossplane. Okay. Interesting.

And why did you choose that approach? Is it just

to be closer to the actual application in Argo?

Yes, I wanted to use Argo as much as I could

because it just makes sense. you already have

the provisioners within Crossplane. So I don't

need to toy with anything else. And eventually,

if the developer in the first offering just submits

a YAML file, which is, what, five lines long,

then already they have something reduced instead

of just provisioning through a pipeline and everything.

Just git commit and you get an environment. It's

far simpler, far better, and they always get

the same thing. That's very cool. So, okay, backing

up again. Sorry, going back to the developer

platforms and this new approach that you, you

know, you went the SaaS way, it didn't work.

You decided to do this internal thing. How did

you get adoption from the overall engineering

team? It's quite easy. Okay. It's quite easy

because it's a pain that we had to solve. So

this is a major, sorry, even pain in the ass

of developers since they couldn't have normal

developer environments. So once you sell something

that gives them exactly what they need. which

is environments, so it's kind of easy to roll

them in. So they have one place. They're not

even looking. All they get is this environment

that they're connected to. They have their own

Argo CD set up in that environment. They can

provision whatever they want. And the second

phase, which will connect the LLM, will be even

simpler for them. So every developer today wants

to work with as less screens as possible. They

just want to stick to their VS Code, their Visual

Studio, JetBrains, whatever, and just stick their

nose in there all day long. They don't want to

look at metrics and stuff. Just query this. I

want this from QL to tell me if there's a problem

in my app at 2 p .m. to 1 a .m. That's what they

want. So I think that it's easier to maintain

that locally using a DevOps team, platform team,

or even if you want to go even further in AI

ops team, because I do believe that using things

like agent core and bedrock is going to be the

future of these things. You're not going to see

more. more of graphs lying around or backstage

lying around. Because if you can get it through

text, and that text can also generate an image

for you, which is pretty damn good, by the way,

for now, from nano banana, so on so forth. So

you're kind of losing the entire idea of having

platforms like backstage or other ones, all you

need to worry about is integrations. That's interesting.

That's an interesting idea. I'm curious, what

was the biggest surprise with this approach?

Was it the tech? Was it the ownership model?

Was it like change management? Was there anything

that was a surprise, I guess? No, actually, there

were no surprises. We planned it thoroughly because

we knew what we wanted to achieve, how and when.

And the biggest thing is what we're struggling

with the most is what most people are struggling

with. And that's fine -tuning the models to get

exactly what you want. So we're constantly working

also with AWS engineers because we're also working

with Bedrock all the time and we're leveraging

that. We're kind of back and forth with them

all the time for the support. And it is working.

We are getting better and better at achieving

what we want. So, okay, let's say a listener

was listening to this and they thought the internal

integration aspect is really cool, but they've

never set up an IDP before. They're deciding

between... that approach or or backstage what

what signals would push them one way or another

in your opinion i think it's maturity definitely

maturity because i can also vouch for my for

what he did and myself as well that we started

off like everybody you go with things that are

well known and instead of creating something

on your own you want to test these waters first

because maybe you'd be surprised that you're

using backstage in such a small manner that maintaining

it does give you the benefit of everything you

need and you don't change it as much you maybe

update a version here and there but you don't

need all everything it can do or if you go and

you find that you work with companies that give

you the idp experience and that's also good enough

for you and you can you can stick with that and

it's great i mean if it works for you then stay

there but when you get to these points where

everything they give you is just not enough that's

when you start working by yourself That makes

sense. So what's one piece of advice that you

would give like platform or DevEx teams that

are wanting to start an IDP come 2026? I would

actually leverage as much as AI as I can with

that and also work with AI as best as I can.

I mean, even the recent features of Cursor that

they support skills from Claude or using Gemini

CLI or knowing which model. is best for what

kind of work after i get this through this hurdle

and i understand what's going on and how to use

it i will then go after idps i unders i will

have the models understand and map out how they

work and then start fixing my own ideas around

that very cool all right let's shift gears a

bit i wanted to talk about talk about two other

things but uh that were non -idp related uh one

was your mongo atlas i think it's called m atlas

yeah yeah the mat list kind of curious what caused

you to build this and was it because like terraform

operator limits weren't enough or like what what

was the reasoning like what were what pain point

were you solving frustration okay yeah so the

story starts with uh mongo when we started working

mongo we had our terraform modules everything

was up and running great but at some point the

drift was just too large to maintain from um

from Terraform perspective, from Terraform, the

Mongo modules that we built ourselves, and what

was actually within our Mongo Atlas cloud. So

that was a little difficult to manage. So we

said, okay, we have all these resources, and

doing Terraform import is not a nice way to do

it. It's error prone. So we decided to let that

go and see if we can do it in a different way.

We tried exploring the Atlas CLI, but it gave

us only a management perspective creation. creation

experience. It was very difficult to work with.

And then we decided to try the Kubernetes operator.

And then we found two big issues. You can't own

existing resources like you do in Terraform import.

So if you create a similar YAML with a similar

name resource and everything, it just says, no,

it exists. You can't do anything with it. And

the other part is it was semi -limited to what

we needed to do. So given all these issues and

the lack of abilities to work back from existing

infrastructure, it kind of led to the creation

of a MATLAS. MATLAS kind of tries to combine

the user experience from kubectl and also the

abilities of Terraform. So in a nutshell, you

can basically say, okay, I've got this organization

in Mongo Atlas. I want to map all the infrastructure.

All I have to do is hit a button. And well, I'll

hit a feature. It's called Discover. And it just

pushes everything into a YAML file. And moreover,

you can transform that YAML file using MathMath

as well into a workable YAML. Ah, interesting.

That's cool. Yeah. And since I don't rely on

states at all, because I thought that it would

be counterintuitive. so if you change something

because a lot of people may go on to atlas because

of a problem or whatever and start changing settings

because it does happen it's a database you don't

know what's going to happen the next day so i

left that part intentionally out so if you change

something online you can just update that yaml

again with the discover feature and have the

same thing already as infrastructure as code

if you apply something and then you can also

do it in a surgical manner like a partial reconciliation

so i don't break everything that's already existing

in mongo atlas so people can find matless on

github and they just search up okay awesome yeah

definitely i mean it's kind of i've recently

uh read from platform engineering labs that they

have this tool called form a or form i and it

kind of does it it's it's a tool written in pkl

in app in apple language and the idea behind

it is great it's kind of just trying to remove

what terraform does and also do the partial reconciliation

and the discovery and everything so when i read

about that it reminded me of like i'm pretty

much doing the same thing just in go and not

in pkl interesting i have not heard of that tool

interesting very cool it's fairly new it's like

six months old i believe and then uh so lastly

i wanted to just touch on this a little bit you

are a a golden jacket uh holder i'm curious what

that process was like going through all the certifications

well it was fun and it was tedious at the same

time But there was one aspect of it which really

frustrated me a lot. And I do hope that AWS will

listen to that and do change it in the future

because they do have the capacity to do that.

One thing that really pissed me off about the

exams is they're really outdated. That's the

one thing that really hits me every time. For

example, you have the DevOps engineer exam, which

I did that about earlier this year. When I already

did that exam, the code commit service was already

taken offline. Yet the exam had five to six questions

around that. So why are you asking me questions

about a service that no longer exists? Oh, that's

frustrating. Yeah. Yeah. So, and the thing that

also bugs me on that is they constantly say,

yeah, go to skill builder and read the docs all

the time. Read the docs. Okay. The docs said

the service is not existing anymore. So why should

I learn about it? Yeah. Well, then you have situations

like CodeStar going away and then changing. There's

a lot of inconsistency. I agree with that. Yeah,

and that's really the real frustrating part.

But apart from that, it's fun to do. I mean,

it does give you a lot of perspective on AWS

services and how it works, especially when you

read the docs and try to understand further than

what's actually asked about in the exams. And

it's great. I mean, it's a great feeling when

you finish all of that and, you know, kind of

just putting it behind you. Well, congratulations.

I mean, that's an awesome accomplishment. What

recommendations would you have for listeners

that maybe want to head down that road of trying

to pick up all the certifications? I would start

from the very lowest ones. I mean, it's true

what they say about the discounts. So start from

the very first ones and get that 50 % discount

for the next one. And it does train you for the

other exams because you get to understand how

these exams are structured. Yeah, of course,

you can go online and find these dumps or whatever

people really have out there. You can do that.

But once you really do these exams and even fail

once or twice these exams, you understand the

structure and it's quite easily understandable

for the next time you go ahead and do that. Yeah,

very true. Very true. All right. Anything you'd

like to leave our listeners with? Any recommendations?

Yeah, actually, I would. Yeah. I mean. There's

one thing that's been on my mind quite lately,

and it's regarding for all platform engineers,

developers, DevOps teams, literally quite everyone.

And it feels that with all the rise and changes

in Gen AI, a lot of people think that they're

staying far too behind. And I believe that because

of these changes, we should strive to really

get our knowledge up and running in our heads

far faster than we usually should. And yeah,

subscribe to these magazines, TLDR, so on and

so forth, and read more articles and so on. So

because it does give you the benefit of staying

ahead of others. And you need to do that not

because of job security, because it's just that

interesting. Yeah. Yeah, I think anyone that's

in SRE, DevOps, platform engineering. You have

to have a thirst for knowledge because you're

always inundated with different tools, CI, CD

processes, different ways of doing things. And

if you're not staying on top of that, then you

really are going to be behind. And the thing

I always come back to is, you know, I do a big

process or I'm rolling out a new AWS account,

a new way of. provisioning basically a new poc

that i'm doing what i did five years ago for

that poc is not what i would do today i would

architect it completely different than i would

have just five years ago maybe even two years

ago so yeah you you kind of need to always have

that that thirst and hunger for learning and

just staying on top of new things. You can't

stay on top of everything, obviously. I mean,

that's also, you know, the reality is imposter

syndrome does exist. You can't know everything

and that's okay. That's why they're teams. That's

why we're on teams and we're collaborators, you

know, but just trying to be present, I think

is really important. Yeah, most definitely. Awesome.

Well, where can people find you online? Mostly

I'm on LinkedIn. So that's the best way to reach

out to me. Some people already have, so. Awesome.

So Danny Teller on LinkedIn, if you want to reach

out. I'll also leave your MATLESS GitHub link

in the show notes. And thanks for coming on,

Danny. Really appreciate it. Thank you for having

me. I had a great time here. Thanks, Brian. Thanks.

All right. That's the interview with Danny. Quick

reminder on the format. Ship It Weekly is still

the weekly news recap, and I'm dropping these

guest convos in between. If you want to catch

both, hit follow or subscribe wherever you are

listening. And if this episode was useful, share

it with a platform or DevEx friend and leave

a quick rating or review. It's annoying how much

that actually helps the show. We'll be back later

this week with a regular news episode. We'll

see you then. Thanks.

Ship It Conversations: Backstage vs Internal IDPs, and Why DevEx Muscle Matters (with Danny Teller)

Chapters

Transcript

Catch This Episode

Host Commentary

Show Notes

More from Ship It Weekly

EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership

Ship It Conversations: Gareth Kersey on IaCConf 2026, AI, and Corey Quinn’s Terraform Keynote

Ship It Conversations: David Tuite on Backstage, Internal Developer Portals, and the Shift to AI Agents

containerd CRI Vulnerabilities, Datadog PostgreSQL HA on Kubernetes, AWS DevOps Agent with Datadog MCP Server, EKS Control Plane Egress, and Why Users Feel the Wait

Get the next episode in your inbox