Data engineer: how it differs from an analyst

The difference in one line

An analyst answers questions from data. A data engineer makes sure the data exists.

An analyst takes a table and calculates. An engineer builds where that table comes from: gathering data from a dozen sources, normalising it, loading it into a warehouse, making sure it refreshes on time and doesn't break.

What the work involves

Building data pipelines. Regular processes: pull from a source, transform, load into storage. A mid-sized company has hundreds of them.

Watching quality. Data arrives corrupted, incomplete, late. The task is to notice before someone builds a report on it.

Designing the warehouse. How to arrange data so it can be queried conveniently and quickly.

Investigating breakages. When a report shows nonsense, people come to the engineer.

Optimising. A calculation that runs six hours and doesn't fit into the night has to be made faster.

What you need to know

Database queries at a high level. Not "I can select" but confidently: window functions, execution plans, optimisation.

Python. The main tool for processing and automation.

How storage systems work. The differences between databases for daily operation and for analysis, principles of data organisation.

Orchestration tools. Systems that run pipelines on a schedule and manage dependencies.

Stream processing. When data is handled as it arrives rather than once a day.

Infrastructure. Containers, cloud storage, operational basics.

The barrier is markedly higher than for an analyst: you need data work, programming and infrastructure understanding together.

Where people come from

From analysis. The commonest route. An analyst tired of waiting for data who started building it themselves.

From development. Also common: they can already program and pick up the data specifics.

From database administration. Less often, but it happens.

From nothing. Almost never. Entry-level roles are few, and they go to people with at least adjacent experience.

Should you aim here first

Usually not. It's more sensible to enter through analysis or development, work a year, and move across deliberately — understanding what you dislike about your current role and why data engineering fixes it.

There's a practical reason too: a data engineer who was never an analyst builds warehouses that analysts find awkward. Experience from the other side helps enormously.

Trying it before moving

Build your own small pipeline:

• Take a source that updates: open data, a public service interface.
• Write a process that regularly fetches what's new.
• Normalise it and load it into a database.
• Set up a schedule and an alert on failure.
• Build a report on top.

Such a project shows exactly what's wanted at entry, and it can be done in a couple of weeks of evenings.

And doing it alongside someone who'll build reports on your data gives you immediate feedback on whether what you built is usable. That's the central skill of the profession.