7 — Metrics Storage (TSDB)

Chunk/block encoding, the write-ahead log, compaction and the write amplification it trades for query speed, and why a cardinality spike is a storage-engine problem, not just a cost line item.

Data Engineering

A book-shaped table of contents for data engineering: foundations and lifecycle, data modeling, storage systems, ingestion and CDC, distributed processing (Spark/Flink), SQL mastery, workflow orchestration, data quality, platform and cloud architecture, pipeline observability, security and governance, performance engineering, system design, and MAANG interview preparation through capstone builds — cross-linking the existing observability book instead of duplicating it.

1 — Files and Storage Formats

The file formats data engineers choose between — CSV, JSON, Avro, Parquet, ORC — plus the compression, encoding, partitioning, and bucketing decisions that determine how efficiently they can be queried.

3 — Data Lake, Warehouse & Lakehouse

Data lakes, warehouses, marts, and the lakehouse architectures (Delta Lake, Apache Iceberg, Apache Hudi) that merge them, organized through the medallion (bronze/silver/gold) pattern.

Data Engineering

A book-shaped table of contents for data engineering: foundations and lifecycle, data modeling, storage systems, ingestion and CDC, distributed processing (Spark/Flink), SQL mastery, workflow orchestration, data quality, platform and cloud architecture, pipeline observability, security and governance, performance engineering, system design, and MAANG interview preparation through capstone builds — cross-linking the existing observability book instead of duplicating it.

1 — What is Data Engineering?

How data engineering evolved into its own discipline, how the role differs from analytics engineering and data science, and the foundational distinctions (batch vs. streaming, OLTP vs. OLAP) that shape the rest of this book.