Your ETL Skills Aren't Obsolete. But the Way You Use Them Soon Will Be.
Agentic AI can now write the pipeline. It still can't tell you why a legacy system writes a placeholder record under one undocumented condition, or whether a query respects a business rule that never made it into any schema. A look at what's actually becoming obsolete in data engineering — and what isn't.
AIENGINEERING
Ask a room full of ETL developers whether AI is going to make their skills obsolete, and you'll get a comforting, well-rehearsed answer almost immediately: AI is just another tool. We've been here before. Data engineering survived the move from handwritten MapReduce jobs to Spark. It survived the shift from raw SQL scripts to modular dbt models. This is simply the next tool in the belt.
I understand the instinct. But I think that reassurance is hiding something important, and I want to name it precisely instead of dancing around it.
AI is commoditizing ETL implementation faster than it is commoditizing ETL engineering.
Those are not the same thing, and the gap between them is where this whole conversation lives. Writing Python to connect to a source system, stage records into a dataframe, apply transformation logic, and land the result in a warehouse has always been legitimate, respected engineering work. It's also, increasingly, work an agent can do — not perfectly, not without oversight, but reliably enough that the value of doing it by hand is quietly eroding.
So I don't think the right question is "will the ETL developer disappear?" The right question is: does building the pipeline remain the thing that makes that developer valuable?
I don't think it will for much longer. And I think the engineers most at risk aren't the ones with weak skills — they're the ones who keep sharpening "ETL" while the market has already started pricing in "ETL engineering" as the separate, more durable thing.
Yes, Traditional ETL Mechanics Really Are Losing Value
Let's not soften this, because the argument is stronger if we don't.
For decades, the ETL lifecycle has followed a predictable shape: establish a source connection, parse the JSON, write the transformation logic, configure the DAG, load the target, profile the incoming data, and get paged at 2 a.m. when someone upstream changes a schema without telling you. A developer's value was, in large part, a function of how well they navigated that procedural complexity by hand.
Agentic AI is changing the economics of that work. Instead of writing every line of procedural logic, engineers can increasingly specify intent — the constraint, the available tools, the desired outcome — and let an agent handle more of the execution.
This isn't speculative; it's visible in the direction the major platforms are moving. Microsoft's Fabric Data Agents are now generally available and extend into Data Factory itself, letting agents author, run, and diagnose pipelines from natural-language requests rather than manually configured steps. IBM has a comparable capability in watsonx, currently in technical preview, that converts natural-language requirements directly into data flows an agent can run and monitor. Coding agents like Claude Code and OpenAI's Codex are increasingly able to scaffold dbt models, refactor SQL across dialects, and diagnose pipeline failures with minimal hand-holding. I'd encourage you to look at each vendor's current documentation rather than take my word for exact capabilities — this space moves quarter to quarter, and today's preview is next quarter's default. What doesn't age fast is the direction: mechanical pipeline construction is becoming something you increasingly describe rather than hand-write.
I'll resist the temptation to stack this with statistics. It's tempting, because the industry surveys on this are genuinely striking — how much of an engineer's week has historically gone to building and firefighting pipelines rather than higher-value work, how fast AI coding tool adoption has grown, how little of that adoption has translated into proportional output gains. But precise percentages from secondary sources are exactly the kind of thing that's easy to get wrong in the retelling, and this argument doesn't actually need six statistics to hold up. It needs one honest observation: the raw act of producing pipeline code is becoming a much less scarce skill than it used to be, in a growing number of organizations. That's a narrower claim than "coding is dead," and it's also the one I can actually stand behind.
If your professional moat is "I can write the code required to move data from one place to another," that moat is getting shallower.
But automating the construction of a pipeline is not the same thing as automating the engineering judgment behind it. And that distinction is easiest to see in a specific example rather than an abstract one.
A few years ago, I watched a transformation that looked, on paper, completely mechanical: reconcile order records between two systems and flag discrepancies. Any agent today could probably generate that join in seconds. What took the engineer real time — and what actually mattered — wasn't the SQL. It was knowing that one of those systems occasionally wrote a placeholder record for orders cancelled mid-processing, a quirk left over from a platform migration years earlier, and that treating those placeholders as "real" discrepancies would have quietly corrupted a fraud-detection metric downstream. Nothing in the schema told you that. Nothing in the data dictionary told you that. Someone had to know it, or go find someone who did. That's the part of the job that doesn't show up in a pull request.
The Job Isn't Disappearing. It's Moving Up the Stack.
I keep coming back to a simple image. The traditional ETL developer has functioned a lot like a specialized vehicle builder who also drives the truck — assembling every mechanical linkage, then personally navigating the route, mile by mile, pothole by pothole.
The emerging version of this role looks different. The engineer isn't in the driver's seat anymore. The engineer is designing the road network: where it goes, how it's paved, where the guardrails sit, what the speed limits are, and — critically — when a vehicle shouldn't be allowed onto the road at all.
Under the traditional model, the flow looked like this: business requirement → engineer writes pipeline code → output. Under the emerging model, it looks more like: engineer defines intent, context, and constraints → agent selects tools and executes → engineer evaluates the result and intervenes where judgment is required.
That reshuffling changes how engineers actually spend their days. Less time goes into boilerplate API wrappers, hardcoded schema mappings, and firefighting the same predictable formatting errors for the hundredth time. More time goes into defining what "good" looks like in terms a system can act on, building the context an agent needs to make a sound decision, deciding what it's allowed to touch, and auditing whether its reasoning actually held up — the kind of judgment the placeholder-record example above depended on entirely.
That's exactly where the human advantage lives, and I don't think it's going anywhere. A model can synthesize a complex query in seconds. What it can't reliably do is know whether that query respects a piece of business logic that never made it into any schema. The engineer remains accountable for why a pipeline exists, not just whether it runs.
The Competencies Worth Acquiring Now
If you're a strong Python or SQL developer feeling uneasy about "context engineering" and "agent orchestration," I'd push back gently on the framing. This isn't primarily a programming gap. It's a set of adjacent competencies — things worth deliberately practicing over the next year, not things you're already behind on.
Intent and context engineering. This is bigger than writing a good prompt. It's the discipline of translating a fuzzy business outcome into structured instructions and reference material a system can actually use — memory files, reference schemas, semantic definitions. The practical version: next time you'd normally just write the transform yourself, try writing down the context a competent new hire would need to get it right, and see how much of that context currently lives only in your head.
Agent orchestration and tool design. Increasingly, engineers decide what tools an agent can touch, what actions it's permitted to take, and when a human has to sign off before anything moves. The Model Context Protocol (MCP) is one concrete standard for connecting agents to databases and APIs through a consistent interface — but the protocol matters less than the underlying skill of designing safe, well-scoped access. Practice that by starting narrow: give an agent read access before write access, and earn trust incrementally rather than by default.
Semantic architecture. If an agent is going to reason over your organization's data, the meaning of that data has to be explicit somewhere it can find it — semantic layers, metadata, data contracts, clearly defined metrics. Without that scaffolding, agentic tools will confidently join the wrong tables and report conflicting numbers with total conviction. This is genuinely a place where cleaning up your semantic layer this quarter pays off twice: once for your human analysts, and again for every agent that touches the same data later.
Observability and evaluation. The central question shifts from "why did this line of code fail?" to "why did the agent choose this action, and should it have?" That means tracing tool-call chains, auditing generated query plans, and setting clear thresholds for when a human needs to step in — before you've handed over enough autonomy that you find out the hard way.
The Disciplines Worth Refusing to Surrender
The previous section was about what to build. This one is about what not to give up — the engineering disciplines that don't get to disappear just because an agent can now write the code faster than you can. If anything, more autonomy makes these more important, not less.
Governance stays deterministic. An agent should never be the one deciding whether PII is allowed to move across a regional boundary. Access controls, lineage, and compliance enforcement are exactly the kind of thing that needs a hard rule, not a judgment call, no matter how good the model's reasoning looks in a demo.
Data modeling discipline doesn't go away. The fact that an LLM can parse messy, undocumented tables is not a license to keep building messy, undocumented tables. Clean models and enforced contracts are the foundation that makes an agent's output trustworthy instead of merely plausible.
Failure-mode thinking matters more, not less. "Self-healing" cannot quietly mean "the agent invented a workaround so the pipeline would pass validation." If row counts suddenly drop overnight, the right response usually isn't to patch the transformation until the numbers look right again. It's to stop and ask why — the same instinct that caught the placeholder-record problem earlier, just applied automatically instead of by hand.
Source-system empathy remains irreplaceable. An agent can inspect a schema in milliseconds. It has no way of knowing why a legacy system populates a field with a nonsensical default under one specific, undocumented condition — the kind of thing you only know because someone told you, or because you debugged it yourself years ago. That institutional context doesn't transfer to a model. It transfers between people, deliberately, or it gets lost.
Read together, these two lists are really one instruction: get better at the judgment work an agent can't do, and refuse to let the judgment work an agent shouldn't do quietly slide over to it because it's technically capable of attempting it.
What Leaders Owe Their Teams Right Now
For engineering leaders, the honest problem is that you're being asked to prepare people for a future you don't fully understand yet either. Conventional upskilling assumes a stable target — learn tool X, get certified in Y. Agentic AI doesn't hold still long enough for that. What counts as leading-edge practice today can be obsolete, or simply absorbed into a vendor's default product, before a formal training program finishes rolling out.
So don't try to build the perfect multi-year curriculum. Do two things instead:
Create structured room to experiment. Give engineers a real, low-stakes environment to rebuild an existing pipeline using agentic tools, try their own orchestration approach, and compare it against the conventional version. The value isn't proving agents are better — it's learning concretely where they help and where they quietly fail, on your own systems, before it matters.
Change what you measure. Lines of code and pull requests merged are vanity metrics here, and they can actively reward the wrong behavior. Track lead time for changes, rework rate, and downtime hours instead — outcomes, not output.
That's it. You don't need a newsletter subscription list from me; you need permission structures that let your team find out what actually works before the next vendor release changes the answer again.
Don't Protect the Pipeline. Protect the Engineer.
Some of the traditional, procedural skills of ETL development really are losing value. I think that's okay.
Writing hundreds of lines of boilerplate code just to move and reshape data was never actually the highest-value thing a data engineer contributed. It was the implementation work that stood between a business problem and a working solution — necessary, but not the point. The placeholder-record problem was never really about the SQL. It never is.
Agentic AI is removing more of that implementation friction. Which means the opportunity in front of engineers isn't to out-produce a model at writing Python. It's to get better at the things these systems still need from a human: context, architecture, judgment, boundaries, and accountability.
Back to the road. The future data engineer will spend a lot less time personally driving every vehicle down the highway. But someone still has to decide where the road goes, how it's built, where the guardrails sit, what the speed limits are — and when a vehicle doesn't get to leave the lot at all.
Don't mourn the disappearance of pipeline toil. Use what it frees up to become a better engineer.