# Data Engineering
All Data Engineering notes →1 — What is Data Engineering?
How data engineering evolved into its own discipline, how the role differs from analytics engineering and data science, and the foundational distinctions (batch vs. streaming, OLTP vs. OLAP) that shape the rest of this book.
2 — Data Lifecycle
The end-to-end journey data takes from generation through collection, ingestion, storage, processing, serving, consumption, governance, and eventual archival.
3 — Data Engineering Principles
The cross-cutting engineering principles — scalability, reliability, maintainability, data quality, idempotency, fault tolerance, cost, security, and observability — that every pipeline in this book is judged against.
1 — Relational Data Modeling
Entity-relationship modeling, normalization and denormalization trade-offs, and the keys, constraints, and referential integrity rules that keep relational schemas consistent.
2 — Analytical Data Modeling
Dimensional modeling for analytics — facts and dimensions, star and snowflake schemas, Data Vault, wide tables, slowly changing dimensions, and surrogate keys.
3 — Time-Series and Event Modeling
Modeling immutable, time-ordered data — event data, append-only logs, change data capture, and temporal tables.
1 — Files and Storage Formats
The file formats data engineers choose between — CSV, JSON, Avro, Parquet, ORC — plus the compression, encoding, partitioning, and bucketing decisions that determine how efficiently they can be queried.
2 — Storage Engines
How storage engines are actually built — row vs. column stores, LSM trees vs. B+ trees, and the object storage, HDFS, and lake storage layers data platforms sit on.
3 — Data Lake, Warehouse & Lakehouse
Data lakes, warehouses, marts, and the lakehouse architectures (Delta Lake, Apache Iceberg, Apache Hudi) that merge them, organized through the medallion (bronze/silver/gold) pattern.
1 — Batch Ingestion
Batch ingestion patterns — ETL vs. ELT, bulk vs. incremental loads, CDC-driven loads, and snapshot loading strategies.
2 — Streaming Ingestion
Event streaming and message queue platforms — Kafka, Pulsar, Kinesis, Pub/Sub — and the ordering and delivery-guarantee semantics that make streaming ingestion hard to get right.
3 — Change Data Capture
How CDC actually works under the hood — write-ahead logs, Debezium, database log-based capture, schema evolution, and common CDC architectural patterns.
1 — Distributed Computing Fundamentals
The distributed-systems fundamentals underneath every big-data engine — parallel processing, distributed execution, cluster computing, scheduling, and resource management.
2 — Apache Spark
Apache Spark end to end — architecture, RDDs, DataFrames and Datasets, the Catalyst optimizer and Tungsten execution engine, shuffle and partitioning behavior, broadcast joins, and adaptive query execution.
3 — Batch Processing Frameworks
The batch processing framework landscape beyond Spark — Hadoop MapReduce, Tez, Flink's batch mode, and Apache Beam's unified batch/stream model.
4 — Stream Processing
Stream processing engines — Spark Streaming, Structured Streaming, Flink, Kafka Streams, Beam — and the watermarks, windowing, state management, and event-time-vs-processing-time semantics they all have to solve.
1 — SQL Foundations
Foundational SQL — SELECT, JOIN, GROUP BY, HAVING, UNION, CASE expressions, and EXISTS — as the baseline every later SQL chapter builds on.
2 — Advanced SQL
Advanced analytical SQL — window functions, ranking, running totals, recursive queries and recursive CTEs, pivoting, and common table expressions.
3 — Query Optimization
How a query planner turns SQL into an execution plan — reading execution plans, index usage, predicate pushdown, partition pruning, join strategy selection, and cost-based optimization.
1 — Workflow Fundamentals
The fundamentals every orchestrator builds on — DAGs, scheduling, task dependencies, retries, and backfills.
2 — Apache Airflow
Apache Airflow in depth — DAG design, operators and sensors, the TaskFlow API, dynamic DAG generation, scheduling, and monitoring.
3 — Modern Orchestrators
The modern orchestrator landscape beyond Airflow — Dagster, Prefect, Argo Workflows, Temporal, and Azure Data Factory.
1 — Data Validation
Validating data as it moves — constraints, assertions, Great Expectations, Deequ, and schema validation.
2 — Data Testing
Testing pipelines like software — unit tests, integration tests, full pipeline tests, and contract testing between producers and consumers.
3 — Metadata Management
The metadata layer that makes data discoverable and trustworthy — data catalogs, lineage tracking, schema registries, and data discovery tooling.
1 — Building a Data Platform
The architectural decisions behind building a data platform — core platform components, data mesh vs. data fabric, and centralized vs. federated ownership models.
2 — Storage Architecture
Tiered storage architecture — hot, warm, and cold tiers — and the lifecycle policies that move data between them automatically.
3 — Compute Architecture
Compute architecture for data platforms — running workloads on Kubernetes, autoscaling, serverless compute, and the cost management trade-offs between them.
1 — AWS Data Stack
The AWS data stack — S3, Glue, EMR, Athena, Redshift, Kinesis, and Lambda — and how they compose into an end-to-end pipeline.
2 — Azure Data Stack
The Azure data stack — ADLS, Synapse, Event Hub, Data Factory, Databricks, and Microsoft Fabric — and how they compose into an end-to-end pipeline.
3 — Google Cloud Data Stack
The Google Cloud data stack — BigQuery, Dataflow, Dataproc, Pub/Sub, Composer, and Cloud Storage — and how they compose into an end-to-end pipeline.
1 — Monitoring Pipelines
Monitoring data pipelines with metrics, logs, and traces, and defining pipeline health through SLIs and SLOs.
2 — Alerting
Alerting on the failure modes specific to data pipelines — freshness, completeness, volume anomalies, latency, and outright failures.
3 — Data Reliability
Data reliability engineering — data contracts, lineage as a debugging tool, incident management, and root cause analysis for pipeline failures.
1 — Security Fundamentals
Security fundamentals for data platforms — IAM, RBAC, encryption at rest and in transit, secrets handling, and key management.
2 — Governance
Data governance — metadata-driven policy, regulatory compliance including GDPR, data retention rules, and audit logging.
3 — Privacy Engineering
Privacy engineering techniques for protecting sensitive data — masking, tokenization, anonymization, and differential privacy.
1 — Performance Optimization
Performance optimization for distributed pipelines — parallelism, partitioning strategy, data skew, shuffle optimization, and caching.
2 — Cost Optimization
Cost optimization for data platforms — storage and compute cost drivers, compression, autoscaling, and spot instance strategies.
3 — Capacity Planning
Capacity planning for data systems — throughput estimation, scaling strategy, benchmarking, and load testing.
1 — Batch Processing System Design
Open-ended batch processing system design — log analytics platforms, ETL platforms, and reporting pipelines.
2 — Streaming System Design
Open-ended streaming system design — clickstream analytics, fraud detection, IoT platforms, and real-time metrics systems.
3 — Data Lakehouse Design
Designing a lakehouse end to end — bronze/silver/gold layering, incremental pipeline design, and cross-team data sharing.
4 — ML Data Platform Design
Designing the data platform underneath ML systems — feature stores, offline and online stores, and the pipelines that feed model training and serving.
1 — SQL Interview Problems
SQL interview problems by difficulty — easy, medium, and hard — with a dedicated focus on window function problems.
2 — Spark Interview Questions
Spark interview questions covering architecture, optimization techniques, debugging approaches, and performance tuning.
3 — Data Engineering System Design Interviews
How to run an open-ended data engineering system design interview — framing trade-offs, capacity estimation, and bottleneck analysis.
4 — Behavioral Interviews
Behavioral interview preparation framed around ownership, reliability, incident response, and leadership principles.
1 — Build an End-to-End Data Platform
A capstone build integrating Kafka, Spark, Airflow, Delta Lake or Iceberg, and Grafana/Prometheus into one end-to-end data platform.
2 — Build a Streaming Analytics Platform
A capstone build of a streaming analytics platform — a clickstream pipeline, real-time dashboards, alerting, and observability.
3 — Build a Lakehouse on Kubernetes
A capstone build of a lakehouse on Kubernetes — the Spark Operator, Airflow, MinIO, Trino, and Iceberg working together.
4 — Staff-Level Architecture Case Studies
Staff-level architecture case studies from Netflix, Uber, Airbnb, LinkedIn, Meta, and Google's data platforms.
1 — Data Engineering Cheat Sheets
A consolidated quick-reference index across all the cheat sheets and checklists in this Part.
10 — 100 MAANG Data Engineering Interview Questions
A consolidated list of 100 data engineering interview questions asked at MAANG-tier companies.
2 — SQL Cheat Sheet
A quick-reference index of common SQL syntax, functions, and query patterns.
3 — Spark Optimization Checklist
A checklist of Spark performance and cost optimization techniques to run through before shipping a job.
4 — Kafka Cheat Sheet
A quick-reference index of Kafka concepts, CLI commands, and configuration patterns.
5 — Airflow Best Practices
A checklist of Airflow DAG design and operational best practices.
6 — Data Modeling Patterns
A reference catalog of recurring data modeling patterns across relational, dimensional, and event-based schemas.
7 — Lakehouse Comparison (Delta vs Iceberg vs Hudi)
A side-by-side comparison of Delta Lake, Apache Iceberg, and Apache Hudi across features, ecosystem, and trade-offs.
8 — Cloud Data Services Comparison (AWS vs Azure vs GCP)
A side-by-side comparison of AWS, Azure, and Google Cloud's data services by category.
9 — Common Interview Pitfalls
The most common mistakes candidates make in data engineering interviews, and how to avoid them.
Data Engineering
A book-shaped table of contents for data engineering: foundations and lifecycle, data modeling, storage systems, ingestion and CDC, distributed processing (Spark/Flink), SQL mastery, workflow orchestration, data quality, platform and cloud architecture, pipeline observability, security and governance, performance engineering, system design, and MAANG interview preparation through capstone builds — cross-linking the existing observability book instead of duplicating it.