14 — Transactional Outbox

Atomically write to the database and publish a message by using a single local transaction. The outbox table is polled or tailed by CDC to publish reliably. Solves the dual-write problem.

Data Engineering

A book-shaped table of contents for data engineering: foundations and lifecycle, data modeling, storage systems, ingestion and CDC, distributed processing (Spark/Flink), SQL mastery, workflow orchestration, data quality, platform and cloud architecture, pipeline observability, security and governance, performance engineering, system design, and MAANG interview preparation through capstone builds — cross-linking the existing observability book instead of duplicating it.

1 — Batch Ingestion

Batch ingestion patterns — ETL vs. ELT, bulk vs. incremental loads, CDC-driven loads, and snapshot loading strategies.

2 — Streaming Ingestion

Event streaming and message queue platforms — Kafka, Pulsar, Kinesis, Pub/Sub — and the ordering and delivery-guarantee semantics that make streaming ingestion hard to get right.

Data Engineering

A book-shaped table of contents for data engineering: foundations and lifecycle, data modeling, storage systems, ingestion and CDC, distributed processing (Spark/Flink), SQL mastery, workflow orchestration, data quality, platform and cloud architecture, pipeline observability, security and governance, performance engineering, system design, and MAANG interview preparation through capstone builds — cross-linking the existing observability book instead of duplicating it.

1 — What is Data Engineering?

How data engineering evolved into its own discipline, how the role differs from analytics engineering and data science, and the foundational distinctions (batch vs. streaming, OLTP vs. OLAP) that shape the rest of this book.