Data Engineers

Impact

Coding assistants draft transformation logic, SQL, orchestration configuration and unit tests from a plain-language description, and the major data platforms have added natural-language interfaces that generate queries and pipelines directly. Managed connectors handle most source ingestion, and AI features now suggest schema changes, detect data-quality anomalies and explain lineage. The day-to-day shifts from writing and debugging pipeline code towards designing the data model, setting standards, reviewing generated code and controlling cost and reliability across a growing estate.

Risk

Core build tasks are automating now; value shifts to architecture, data contracts, governance and cost control.

The score sits in the very high band: occupation-level measures place the role in the highest official exposure tier, observed usage of generative AI by data and database professionals is among the highest of any occupation, and adoption across the sector is very high. Writing transformation code, SQL, tests, documentation and connector configuration is being automated today. What stays human is deciding what the data should mean, negotiating contracts with producers and consumers, designing for reliability and privacy, and judging the trade-offs between cost, latency and correctness. In the 0-3 year window, expect smaller teams delivering more pipelines, with hiring tilted towards engineers who can own a platform rather than write individual jobs.

Sector readiness

Deep Integration Across Data Platforms

The sector is further along than almost any other: coding assistants are standard in data teams, and Databricks, Snowflake, dbt and Fivetran have all shipped AI features for generating, documenting and monitoring pipelines. Technology companies and financial services lead; large enterprises with legacy warehouses are adopting more slowly but are moving in the same direction.

1. 01

   Review more than you write. Coding assistants will produce most of your SQL and transformation code. Your value is in specifying it precisely and catching what is wrong, so sharpen review habits and test design.

2. 02

   Own the data model. The hardest decisions are about what the data means and how it joins. Generated code inherits whatever model you give it, so make modelling your core craft.

3. 03

   Make cost a first-class skill. AI-generated queries are often wasteful. Engineers who understand compute pricing and optimisation are the ones who save organisations real money.

4. 04

   Build the contracts. Formal agreements on schemas, freshness and quality between producers and consumers are what make automated pipelines safe. Lead on them.

5. 05

   Move towards AI data infrastructure. Pipelines that feed retrieval systems, evaluation sets and fine-tuning are where new data engineering work is appearing.

6. 06

   Document with the tools, then verify. AI generates lineage and documentation well, but it can be confidently wrong. Make verification part of your definition of done.

1. 01

   AI coding assistants. Tools that generate SQL, transformation logic, tests and orchestration code from natural language have removed much of the routine build effort.

2. 02

   Natural-language platform features. Databricks, Snowflake and dbt now let users generate pipelines, queries and documentation inside the platform itself.

3. 03

   Managed ingestion. Pre-built connectors and schema-drift handling mean source ingestion rarely needs custom code.

4. 04

   Automated data-quality monitoring. Anomaly detection and lineage tools find problems that engineers used to discover by hand or from complaints.

5. 05

   Very high observed usage. Recorded generative-AI usage for database and data roles is among the highest of any occupation, which pulls the score into the top band.

6. 06

   Demand from AI workloads. Organisations need reliable data pipelines to feed AI systems, which sustains demand even as individual tasks automate.

Technology and SaaS companies

The most exposed setting, where assistants and modern platforms are universal and small teams are expected to deliver large estates.

Financial services

High adoption but heavier governance, so human review, lineage and auditability keep engineers central to the process.

Healthcare and public sector

Legacy systems and strict data rules slow adoption; integration and compliance work remain largely manual.

Retail and consumer

Rapid adoption of managed platforms and AI features, with engineers moving towards real-time and personalisation infrastructure.

1. 01

   Data modelling and architecture. Deciding how data is structured and joined is the judgement that generated code depends on; study dimensional and domain-driven approaches.

2. 02

   Code review and test design. With most code generated, the ability to specify, test and critically review becomes the core engineering skill.

3. 03

   Cost and performance optimisation. Understand compute pricing, partitioning and query plans so you can make generated pipelines efficient.

4. 04

   Data governance and privacy. Contracts, lineage, access control and regulatory compliance are where human accountability stays.

5. 05

   AI infrastructure. Learn retrieval pipelines, vector stores and evaluation datasets to work on the data behind AI systems.

6. 06

   Platform engineering. Treat the data platform as a product with reliability targets and self-service for users, which is where senior roles are going.

Generated transformation modelsExample 1

How

An engineer describes a required mart in plain language, has the assistant draft dbt models and tests, then reviews the joins and adds edge-case tests before merging.

Gain

Build time falls sharply while review quality determines correctness.

Natural-language debuggingExample 2

How

When a pipeline fails overnight, the engineer pastes logs and the query into the platform assistant to get a diagnosis and proposed fix.

Gain

Incidents are resolved faster with less time spent reading stack traces.

Automated documentation and lineageExample 3

How

A team uses platform AI to generate column descriptions and lineage graphs, with engineers verifying and correcting them.

Gain

Documentation stays current without consuming engineering time.

Database AdministratorsMore exposed · exposure 65

AI impact

Tuning, backup and routine maintenance are increasingly handled by cloud platforms and AI, leaving less scope than pipeline design.

Work moves to

Cloud migration, security and performance governance.

Machine Learning EngineersDifferent skills, growing · exposure 58

AI impact

AI assists with code but demand for people who build and operate models keeps growing.

Work moves to

Model deployment, evaluation and production reliability.

Chief Data Officers (CDOs)Complementary, less exposed · exposure 46

AI impact

Strategy, governance and organisational accountability are less exposed than the hands-on engineering.

Work moves to

Data strategy, policy and executive leadership.

Our report is one reading of the evidence. This section is the other dataset: what people who do or know this work make of it. Nobody has scored this role yet. Sign in to add yours.