Data Engineering

If this were a book, this page is the table of contents. Each Part below is a chapter; each chapter links out to the concepts, designs, and platform notes that already exist elsewhere in this wiki instead of duplicating them. Unwritten chapters are listed as Planned rows, not empty files.

Parts

00 — Foundations of Data Engineering

The mental models and vocabulary that frame everything else in this book — what data engineering actually is, the end-to-end lifecycle data moves through, and the cross-cutting principles (scalability, reliability, cost, security) every later Part assumes.

• 1 — What is Data Engineering? — (stub)
• 2 — Data Lifecycle — (stub)
• 3 — Data Engineering Principles — (stub)

01 — Data Modeling

How data is shaped for different consumption patterns — relational modeling for OLTP systems, dimensional modeling for analytics, and event/temporal modeling for immutable, time-ordered data. See also Log Aggregation for the schema-on-read vs. schema-on-write tradeoff applied to the logs-specific case.

• 1 — Relational Data Modeling — (stub)
• 2 — Analytical Data Modeling — (stub)
• 3 — Time-Series and Event Modeling — (stub)

02 — Storage Systems

The physical and logical layers data lives in — file formats, storage engines (row vs. column, LSM tree vs. B+ tree), and the lake/warehouse/lakehouse architectures built on top of them.

• 1 — Files and Storage Formats — (stub) — the partitioning and bucketing covered here is distinct from the query-time fan-out sharding in Scatter-Gather & Sharding; this chapter is about how data sits on disk, not how a single query is split across workers.
• 2 — Storage Engines — (stub)
• 3 — Data Lake, Warehouse & Lakehouse — (stub) — see Observability Data Lake for a worked cold/warm/hot tiering instance of this pattern.

03 — Data Ingestion

Getting data into the platform — batch loads, streaming ingestion, and change data capture as the bridge between operational databases and analytical systems.

• 1 — Batch Ingestion — (stub)
• 2 — Streaming Ingestion — (stub) — see also Distributed Message Queue for a full Kafka-like system design case study.
• 3 — Change Data Capture — (stub) — see the Outbox Pattern for the standard mechanism that makes CDC emission reliable without dual writes.

04 — Distributed Data Processing

Compute over ingested data at scale — distributed systems fundamentals, Apache Spark in depth, batch processing frameworks, and stream processing engines.

• 1 — Distributed Computing Fundamentals — (stub)
• 2 — Apache Spark — (stub)
• 3 — Batch Processing Frameworks — (stub)
• 4 — Stream Processing — (stub) — see also Stream Processing System for a full Flink-like system design case study, and Q12: Mixed Exactly-Once Billing Tenant for a worked exactly-once delivery scenario.

05 — SQL Mastery

The query layer every data engineer is judged on in interviews — foundational SQL, advanced analytical SQL (windows, recursion), and how a query planner turns SQL into an execution plan.

• 1 — SQL Foundations — (stub)
• 2 — Advanced SQL — (stub)
• 3 — Query Optimization — (stub)

06 — Workflow Orchestration

Coordinating pipelines as dependency graphs — scheduling fundamentals, Apache Airflow in depth, and the modern orchestrator landscape (Dagster, Prefect, Temporal).

• 1 — Workflow Fundamentals — (stub)
• 2 — Apache Airflow — (stub)
• 3 — Modern Orchestrators — (stub)

07 — Data Quality

Trusting the data once it’s moving — validation and assertion frameworks, testing pipelines like software, and the metadata layer (catalog, lineage, schema registry) that makes data discoverable.

• 1 — Data Validation — (stub)
• 2 — Data Testing — (stub)
• 3 — Metadata Management — (stub)

08 — Data Platform Architecture

Zooming out from individual pipelines to the platform that hosts them — architecture patterns (data mesh vs. centralized), storage architecture, and compute architecture.

• 1 — Building a Data Platform — (stub)
• 2 — Storage Architecture — (stub)
• 3 — Compute Architecture — (stub)

09 — Cloud Data Engineering

The managed-service equivalents of every Part above, per hyperscaler — AWS, Azure, and Google Cloud’s respective data stacks.

• 1 — AWS Data Stack — (stub)
• 2 — Azure Data Stack — (stub)
• 3 — Google Cloud Data Stack — (stub)

10 — Observability for Data Pipelines

Applying observability practice to pipelines specifically — monitoring, alerting, and reliability engineering for data systems. This is observability applied to datasets, not services; see Observability for the systems-telemetry sense of the word and the underlying metrics/logs/traces/SLO fundamentals this Part builds on.

• 1 — Monitoring Pipelines — (stub)
• 2 — Alerting — (stub)
• 3 — Data Reliability — (stub)

11 — Security & Governance

Protecting and governing data at rest and in motion — IAM/RBAC/encryption fundamentals, governance and compliance, and privacy engineering techniques (masking, anonymization, differential privacy).

• 1 — Security Fundamentals — (stub)
• 2 — Governance — (stub) — grounded in the m-data-platform charter’s schemas/, lineage/, and governance/ sub-areas in the parent monorepo.
• 3 — Privacy Engineering — (stub)

12 — Performance Engineering

Making pipelines fast and cheap — performance optimization, cost optimization, and capacity planning.

• 1 — Performance Optimization — (stub)
• 2 — Cost Optimization — (stub)
• 3 — Capacity Planning — (stub)

13 — Data Engineering System Design

Applying everything above to open-ended system design problems — batch and streaming system design, lakehouse design, and ML data platform design (feature stores). Follows the same five-step format used throughout System Design: requirements, high-level design, deep dive, self-observability, trade-offs at 10x scale.

• 1 — Batch Processing System Design — (stub)
• 2 — Streaming System Design — (stub)
• 3 — Data Lakehouse Design — (stub)
• 4 — ML Data Platform Design — (stub)

14 — MAANG Interview Preparation

Interview-specific drilling — SQL problems by difficulty, Spark interview questions, open-ended data engineering system design interviews, and behavioral interviews framed around ownership and reliability.

• 1 — SQL Interview Problems — (stub)
• 2 — Spark Interview Questions — (stub)
• 3 — Data Engineering System Design Interviews — (stub)
• 4 — Behavioral Interviews — (stub)

15 — Capstone Projects

End-to-end builds that integrate the whole book — a full data platform, a streaming analytics platform, a lakehouse on Kubernetes, and staff-level architecture case studies from Netflix, Uber, Airbnb, LinkedIn, Meta, and Google.

• 1 — Build an End-to-End Data Platform — (stub)
• 2 — Build a Streaming Analytics Platform — (stub)
• 3 — Build a Lakehouse on Kubernetes — (stub)
• 4 — Staff-Level Architecture Case Studies — (stub)

16 — Appendices

Quick-reference material — cheat sheets, checklists, and comparison tables — for use alongside the chapters above.

• 1 — Data Engineering Cheat Sheets — (stub)
• 2 — SQL Cheat Sheet — (stub)
• 3 — Spark Optimization Checklist — (stub)
• 4 — Kafka Cheat Sheet — (stub)
• 5 — Airflow Best Practices — (stub)
• 6 — Data Modeling Patterns — (stub)
• 7 — Lakehouse Comparison (Delta vs Iceberg vs Hudi) — (stub)
• 8 — Cloud Data Services Comparison (AWS vs Azure vs GCP) — (stub)
• 9 — Common Interview Pitfalls — (stub)
• 10 — 100 MAANG Data Engineering Interview Questions — (stub)

Metadata

Author	Amit Singh
Scope	data-engineering