How to become a Data Engineer

A complete, honest roadmap for the skills data engineers actually use, from programming and SQL through data modelling, warehousing, pipelines, distributed processing, streaming, cloud platforms, orchestration, quality, and the data work behind AI. It runs top to bottom, foundational to advanced, so you always know what comes next. Free to read, no signup required.

1. 01

   Step 01Programming & software foundations

   Data engineering is software engineering applied to data. Before any platform, get comfortable writing real code and working like an engineer.

   • Python: The default language for data work. Learn data structures, control flow, error handling, and working with files and APIs.
   • Software engineering habits: Version control, testing, code review, and clean code. Pipelines are software, and they rot without these.
   • The command line & Linux: Processes, permissions, environment variables, and reading logs, because most data infrastructure runs on Linux.
   • How the web moves data: HTTP, APIs, and JSON, since a lot of a pipeline is pulling data from and pushing it to services.

   Data engineers who can only click through a UI hit a ceiling fast. The ones who can write, test, and debug real code are the ones who build systems that hold up.

   PythonGit & testingLinux CLIAPIs & JSONClean code
2. 02

   Step 02Databases & SQL, deeply

   SQL is the single most-used skill in data engineering. You will read and write it every day, so get genuinely good at it.

   • Core SQL: SELECT, joins, aggregation, grouping, and filtering to answer real questions from data.
   • Advanced SQL: Window functions, CTEs, subqueries, and set operations for the transformations pipelines actually need.
   • How databases work: Indexes, query plans, transactions, and why a query is slow, so you can make it fast.
   • Relational and beyond: Postgres and MySQL as the defaults, plus an awareness of NoSQL stores and when each fits.

   Core SQLWindow functionsCTEsIndexes & query plansTransactionsPostgres / MySQL
3. 03

   Step 03Data modelling & warehousing

   How you structure data decides whether analysis is fast and correct or slow and wrong. Modelling is a core data-engineering craft.

   • OLTP vs OLAP: The difference between systems built for transactions and systems built for analysis, and why they’re modelled differently.
   • Normalization vs dimensional: Normalised models for operational data, and dimensional models (star schemas, facts and dimensions) for analytics.
   • The data warehouse: What a warehouse is, how it differs from an operational database, and why analytics workloads live there.
   • Lakes and lakehouses: Data lakes for raw and semi-structured data, and the lakehouse pattern that blends lake and warehouse.

   A clean dimensional model saves every downstream analyst hours and prevents a class of wrong numbers. Time spent on modelling is rarely wasted.

   OLTP vs OLAPStar schemaFacts & dimensionsWarehouseData lakeLakehouse
4. 04

   Step 04ETL, ELT & building pipelines

   The heart of the job is moving data reliably from sources to where it’s useful. This is what a data engineer is paid to build.

   • Extract, transform, load: The stages of a pipeline, and the modern ELT shift where you load raw data first and transform it in the warehouse.
   • Batch pipelines: Scheduled jobs that process data in chunks, the workhorse pattern for most analytics.
   • Idempotency & reprocessing: Designing jobs so rerunning them is safe, and so you can backfill history without creating duplicates.
   • Incremental loads: Processing only what changed instead of everything, which is what keeps pipelines affordable at scale.

   ETL vs ELTBatch processingIdempotencyBackfillsIncremental loads
5. 05

   Step 05Big data & distributed processing

   Once data outgrows a single machine, you process it across many. Understanding distributed computing is what separates a data engineer from a scripter.

   • Why distributed: What breaks when data no longer fits on one machine, and how splitting work across a cluster solves it.
   • Partitioning & shuffles: How data is split across nodes, and why moving it between them (the shuffle) is the expensive part.
   • The processing engines: Distributed engines like Spark, plus the older MapReduce ideas they build on, for large-scale transformation.
   • File formats: Columnar formats like Parquet and how partitioning and compression make large scans fast and cheap.

   Distributed computePartitioningShufflesSparkParquetCompression
6. 06

   Step 06Streaming & real-time data

   Not all data can wait for the nightly batch. Streaming handles events as they happen, and it’s an increasingly common requirement.

   • Batch vs streaming: When periodic batch jobs are enough, and when a use case genuinely needs data processed within seconds.
   • Event streaming: Message brokers and logs like Kafka that carry events between systems reliably and in order.
   • Stream processing: Transforming and aggregating data in motion, including windows and handling late-arriving events.
   • Delivery guarantees: At-least-once, at-most-once, and exactly-once, and why the guarantee you choose shapes the whole design.

   Streaming is powerful but adds real operational complexity. Reach for it when the business genuinely needs fresh data, not because it sounds impressive.

   Batch vs streamingKafkaEvent logsWindowingExactly-once
7. 07

   Step 07Cloud data platforms

   Almost all data engineering now happens in the cloud. Learn the building blocks rather than memorising one vendor’s buttons.

   • Object storage: Cheap, durable storage (S3 and its peers) that acts as the foundation of most data lakes.
   • Cloud warehouses: Managed analytics warehouses like Snowflake, BigQuery, and Redshift that separate storage from compute.
   • Managed services: How cloud platforms provide pipelines, streaming, and orchestration so you run less infrastructure yourself.
   • Cost awareness: Cloud data costs scale with what you scan and store, so designing for cost is part of the engineering.

   Object storageSnowflake / BigQuery / RedshiftStorage vs computeManaged servicesCost control
8. 08

   Step 08Orchestration & workflow management

   Real data platforms run hundreds of interdependent jobs. Orchestration is how you schedule them, order them, and recover when one fails.

   • DAGs: Modelling pipelines as directed graphs of tasks with clear dependencies, so the right things run in the right order.
   • Scheduling & triggers: Running jobs on a schedule or in response to events, and handling dependencies between them.
   • Retries & recovery: Automatic retries, alerting on failure, and re-running just the failed part rather than everything.
   • Orchestrators: Tools like Airflow and its peers that manage all of this, plus the concepts that apply whichever you use.

   DAGsSchedulingDependenciesRetriesAirflow & peers
9. 09

   Step 09Data quality, testing & observability

   A pipeline that runs but produces wrong data is worse than one that fails loudly. Trust in the data is the real product.

   • Testing data: Validating schemas, ranges, uniqueness, and freshness, and catching bad data before it reaches a dashboard.
   • Data contracts: Agreements with upstream teams about the shape and meaning of data, so a source change doesn’t silently break you.
   • Lineage: Tracing where a number came from and what feeds it, which is essential for debugging and trust.
   • Observability: Monitoring freshness, volume, and quality so you find out about a problem before a stakeholder does.

   The fastest way to lose a data team’s credibility is a wrong number in an executive report. Quality checks are not optional polish; they’re the job.

   Data validationData contractsLineageFreshness monitoringAlerting
10. 10

    Step 10Reliability, scaling & governance

    As a platform grows, keeping it reliable, affordable, and compliant becomes the harder half of the job.

    • Reliability: SLAs on freshness and uptime, graceful failure, and pipelines that recover on their own where possible.
    • Performance & cost: Partitioning, pruning, and right-sizing compute so pipelines stay fast without an unbounded bill.
    • Security & privacy: Access control, encryption, and handling personal data carefully, because you often hold the whole company’s data.
    • Governance: Cataloguing, ownership, and compliance so data is discoverable and used responsibly.

    SLAsPerformance tuningCost controlAccess controlPIIGovernance
11. 11

    Step 11Data engineering for AI & ML

    AI and ML run on data, and data engineers increasingly build the plumbing behind them. This is a fast-growing part of the role.

    • Data for machine learning: Preparing clean, reliable training and feature data, since model quality is capped by data quality.
    • Feature stores: Serving consistent features to both training and production so models see the same data in both.
    • Data for LLM apps: Building the retrieval pipelines behind AI features: chunking, embedding, and loading data into vector stores.
    • Using AI tools: AI assistants speed up writing SQL, transformations, and tests, but you own reviewing what ships.

    The fundamentals in this roadmap matter more as AI grows, not less. Reliable pipelines and clean data are exactly what every AI system depends on.

    ML data prepFeature storesVector storesRetrieval pipelinesAI tooling