# Prometheus
All Prometheus notes →2 — Time Series Fundamentals
The time series data model behind Prometheus — metric names, labels, timestamps, samples, and how PromQL classifies the data it operates on.
3 — Prometheus in the Observability Ecosystem
Where Prometheus sits in the CNCF landscape — its pull-based cloud-native origins, its companion projects, and where this book does (and doesn't yet) connect it to the wider stack.
1 — Prometheus Components
The functional pieces inside a Prometheus server — scrape manager, TSDB, rule engine, query engine — and the real commands used to install and run one on a VM, under systemd, or in Docker.
2 — Pull Model Deep Dive
Why Prometheus chose a pull-based scrape model over pushing metrics, what that trades away, and how the Pushgateway papers over the one workload — short-lived batch jobs — where pull genuinely doesn't fit.
3 — Data Flow
A short connective walk through Prometheus end to end — from an instrumented app exposing a metric, through scraping and storage, to a PromQL query surfaced as an alert or a dashboard panel — with each stage pointing to the chapter that owns it.
1 — Metrics Deep Dive
The four Prometheus metric types — Counter, Gauge, Histogram, Summary — walked through hands-on across a Linux batch app + Node Exporter and a Windows web app + Windows Exporter, plus the histogram_quantile bucket-math caveat and the Histogram-vs-Summary tradeoff.
2 — Labels and Cardinality
Label mechanics and series identity in Prometheus — how labels turn one metric name into many time series, the storage/performance cardinality math, and target relabeling vs. metric relabeling with real relabel_configs YAML.
2 — Exporters
What a Prometheus exporter is, installing Node Exporter as a systemd service, and monitoring the container runtime itself via Docker Engine metrics and cAdvisor.
1 — Discovery Mechanisms
How Prometheus finds scrape targets — static configs, file-based service discovery with watched JSON files, and DNS service discovery via SRV/A records — plus validating and reloading configuration safely.
1 — PromQL Fundamentals
The four PromQL data types, label matchers and selectors, and how to run PromQL outside the Prometheus UI via the HTTP API.
2 — PromQL Functions
Math, date/time, type-conversion, and sorting functions; the rate() vs irate() decision; and the histogram_quantile() function-call mechanics.
3 — Aggregation Operators
The PromQL aggregation operator table, the by clause, the without clause, and worked collapsing examples across single and multiple labels.
4 — Vector Matching
How PromQL matches labels between two instant vectors — ignoring/on, one-to-one vs many-to-one/one-to-many with group_left/group_right — plus arithmetic, comparison, and logical operators.
5 — Advanced PromQL
The complete recording-rule syntax reference (rule files, worked example, level:metric_name:operations naming), the offset and @ modifiers, and subqueries.
1 — Recording Rules
Why recording rules exist in an alerting pipeline: pre-computing expensive or frequently-evaluated expressions so alert rules stay cheap, and how that ties to rule-group evaluation cadence.
2 — Alerting Rules
The alert state lifecycle — inactive, pending, firing — built from Prometheus's scrape and evaluation clocks, plus a line-by-line walk through a real alert rule's for:, labels:, and annotation templating.
3 — Alertmanager
How Alertmanager groups and deduplicates alerts in practice — group_wait, group_interval, and repeat_interval — with routing/receiver depth and open gaps called out honestly.
2 — Long-Term Storage
Why single-node Prometheus has no built-in long-term-storage or HA story, and how remote_write receivers like Thanos, Cortex, Mimir, and VictoriaMetrics fill that gap at a horizontally-scaled, multi-tenant layer.
2 — Security
Securing the exporter-to-Prometheus link with TLS and basic auth — self-signed certs, bcrypt password hashing, tls_server_config, and end-to-end curl verification, plus an honest look at what this setup doesn't cover.
2 — Hands-On Labs
A sequenced, hands-on path through the practical material already covered elsewhere in this book, arranged as a lab progression for PCA readiness.
3 — Deep Dive Discussions
Interview-framed answers to the 'why' questions candidates get asked about Prometheus — why pull, why not SQL, why labels — honestly scoped to what this book actually has source material for.
1 — PromQL Cheat Sheet
A grouped, copy-paste reference of node_exporter PromQL queries for CPU, memory, disk, network, swap, inode, and TCP socket questions.
4 — Exporter Catalog
A lookup table of exporters covered in this book — key metrics, metric types, and what each metric tells you — for the two exporters with real worked examples in the source material.
5 — Prometheus Configuration Reference
A field-by-field reference for prometheus.yml — global settings, scrape_configs options, and worked examples pulled from real multi-job configurations.
6 — Common Anti-Patterns
Three concrete Prometheus anti-patterns seen in this book's source material — high-cardinality labels, invalid/reserved metric naming, and misaligned histogram_quantile buckets — with why each one breaks and where to read the full explanation.
1 — Why Monitoring Exists
Evolution of monitoring, observability vs. monitoring, the USE/RED methods, the four golden signals, and SLIs/SLOs/SLAs as the vocabulary the rest of this book assumes.
3 — TSDB Internals
The head block/WAL/compaction model underneath Prometheus's on-disk TSDB — why a cardinality spike is a storage-engine incident, not just a cost line item.
1 — Client Libraries
Instrumenting an application directly with a Prometheus client library (Go, Java, Python, .NET, Node.js, Rust) rather than relying on exporters or auto-instrumentation.
3 — Custom Instrumentation
Writing your own metrics inside application code — naming conventions, label design, and the business/performance/error/latency metric categories worth instrumenting deliberately.
2 — Kubernetes Discovery
Prometheus's native Kubernetes service discovery role (pod/service/endpoints/ingress) and how the Prometheus Operator's ServiceMonitor/PodMonitor/ScrapeConfig CRDs turn that into a declarative, per-namespace scrape contract.
3 — Cloud Discovery
Cloud-provider and registry-based service discovery — AWS EC2, Azure VM/Container Apps, GCP, Consul, and Eureka — for scraping targets that don't live in a single static inventory.
1 — Scaling Prometheus
Federation, functional and horizontal sharding, HA server pairs, and remote_write as Prometheus's own answers to 'this one server can't hold it all anymore.'
3 — Performance Tuning
Tuning Prometheus's own resource footprint and query performance — memory, CPU, WAL replay time, compaction cadence, scrape interval, and retention as levers, with cardinality as the dominant cost driver.
4 — High Availability
Running Prometheus HA pairs, the duplicate-sample problem that creates, deduplication strategies, and failover/DR posture for a monitoring system that is itself a dependency.
1 — Kubernetes Best Practices
Running Prometheus on Kubernetes via Helm and the Prometheus Operator's kube-prometheus-stack — RBAC scope, TLS between components, and NetworkPolicy isolation for the monitoring namespace.
3 — Troubleshooting
Diagnosing missing metrics, duplicate series, high-cardinality blowups, slow queries, WAL corruption, and memory pressure/OOMKills in a running Prometheus deployment.
1 — PCA Exam Objectives
The Prometheus Certified Associate exam blueprint, a study strategy for covering it, and the pitfalls that trip up otherwise-competent candidates.
3 — Practice Exams
Practice questions at increasing difficulty (beginner, intermediate, scenario-based) plus full-length mock exams for PCA readiness.
1 — Prometheus System Design
Designing Prometheus-based monitoring at scale — multi-region topology, HA design, cost optimization, and capacity planning as a system-design interview prompt.
2 — Interview Questions
A seniority-tiered Prometheus/monitoring interview question bank, from beginner fundamentals through staff/architect-level system design and trade-off framing.
4 — Real Production Architectures
Worked case studies of real Prometheus deployment shapes — Kubernetes-native, multi-cluster, hybrid-cloud, multi-tenant, large-enterprise, and SaaS monitoring platform architectures.
2 — Recording Rule Cookbook
A cookbook of distinct, ready-to-paste recording rule patterns — currently the only worked examples in this book's source material are already the teaching example in Advanced PromQL, so there isn't yet a second, genuinely different set of recipes.
3 — Alert Rule Cookbook
A cookbook of ready-to-use Prometheus alerting-rule YAML patterns — currently only one worked example exists in this book's source material (the SLO-breach alert in Alerting Rules, Part 06), not yet enough distinct patterns to call this a genuine cookbook.
7 — PCA Exam Cheat Sheet
A condensed one-page PCA exam reference — pending the exam objectives chapter this depends on.
8 — Interview Cheat Sheet
A condensed one-page interview-prep reference — pending the interview-question bank this depends on.
Prometheus
A book-shaped table of contents for Prometheus: monitoring foundations through architecture, data model, instrumentation, service discovery, PromQL, alerting, production operation, PCA certification, and MAANG interview prep — cross-linking existing notes instead of duplicating them.