Design a Metrics Monitoring & Alerting System (Prometheus / Datadog)

Every company you'll interview at runs something like Prometheus, Datadog, New Relic or Grafana Mimir, and every on-call engineer has been woken up by it. "Design a metrics monitoring and alerting system" is a popular senior-level question because it combines a high-throughput write path, a specialised storage engine, a query engine and a correctness-sensitive alerting pipeline. It also has a classic trap (cardinality) that separates people who have operated these systems from people who have only read about them.

The Metrics Data Model

Start by defining what you store, because every later decision depends on it:

http_requests_total{service="checkout", method="POST", status="500", region="ap-south-1"}
    @ 1760100015  →  18342

• Metric name + labels identify a series. The example above is one series; the same metric with status="200" is another.
• Samples are (timestamp, float64 value) pairs appended to a series, typically every 10–60 seconds.

A detail interviewers like: percentiles cannot be averaged. The p99 of a service isn't the mean of each pod's p99. Histograms with shared bucket boundaries can be summed across pods and then turned into a quantile, which is why server-side aggregation prefers histograms over summaries.

Requirements

Functional

• Collect metrics from hosts, containers, applications and managed services (pull and push).
• Store them and serve queries for dashboards: time range, label filters, aggregations (sum by, rate, quantiles).
• Define alert rules; evaluate them continuously; notify via PagerDuty/Opsgenie, Slack, email, SMS.
• Silences (planned maintenance), grouping, escalation policies.
• Retention: raw data for weeks, downsampled for a year or more.

Non-functional

• Ingest millions of samples per second; ingestion must not fall behind during incidents (when it matters most).
• Dashboard queries over recent data in under a second.
• Alerting must be more available than the systems it watches, with bounded detection delay (a minute or two).
• Some data loss is tolerable (a missed sample is fine); losing alerts is not.
• Multi-tenant: one team's cardinality mistake must not take down everyone's monitoring.

Capacity Estimates

• 100,000 hosts/pods, each exposing ~1,000 series (node metrics + app metrics) → 100M active series.
• Scrape interval 15 s → 100M / 15 ≈ 6.7M samples/s ingested.
• Samples/day: 6.7M × 86,400 ≈ 576 billion.
• At ~1.4 bytes/sample compressed (Gorilla-style), ≈ 800 GB/day; 15 days of raw data ≈ 12 TB, ×3 replication ≈ 36 TB.
• Uncompressed (16 bytes per sample: 8 B timestamp + 8 B value) it would be ~9 TB/day. Compression is a ~10x lever.
• In-memory head: recent samples for 100M series. If each series needs a few KB of memory for its active chunk, labels and index entries, that's hundreds of GB of RAM across the ingester fleet; series count drives memory far more than sample rate.
• Rollups: 5-minute resolution is 1/20 of 15 s raw; 1-hour is 1/240. A year of 1-hour rollups (with 4 aggregates each) is still small.

If you want to practise this style of arithmetic, see back-of-the-envelope estimation.

Pull vs Push Collection

The pragmatic design: run a collection agent on every node (or as a sidecar/DaemonSet) that scrapes local targets and accepts pushes from local processes, adds host labels, batches and compresses, and pushes to the central ingestion tier. That's effectively what the Datadog Agent, Grafana Agent/Alloy and the OpenTelemetry Collector do, and it gives you pull semantics locally with push semantics across the network. Say this in the interview; it shows you know the debate is mostly settled by hybrids.

Architecture

This is roughly the shape of horizontally scalable Prometheus-compatible systems like Cortex, Thanos (sidecar variant) and Grafana Mimir: stateless distributors, a sharded and replicated ingester ring holding recent data, immutable blocks in object storage for history, and a query layer that merges both.

• Distributors hash each series by (tenant, metric name, sorted labels) onto a consistent hash ring and write to 3 ingesters (quorum of 2 acks). Consistent hashing lets you add ingesters without reshuffling everything; the same technique as in Dynamo-style key-value stores.
• Optional Kafka buffer between distributors and ingesters absorbs ingestion spikes and allows replay after ingester failures, at the cost of another system to operate (see Kafka design).
• Agents buffer to local disk when the central system is unreachable, so a network blip doesn't create gaps.

Time-Series Storage: How a TSDB Works

Write path inside an ingester

1. WAL first: each incoming batch is appended to a write-ahead log on local disk, so a crash can rebuild the in-memory state.
2. Head block in memory: each series has an open "chunk" of recent samples (e.g. up to ~120 samples), compressed as it's appended.
3. Cut blocks: every 2 hours, the head is written out as an immutable block: compressed chunks + an index + metadata. Blocks are uploaded to object storage.
4. Compaction: a background compactor merges small blocks into larger ones (2h → 12h → 24h…), deduplicates replicated samples, applies deletions and produces downsampled blocks.

Compression: why ~1.4 bytes per sample is possible

Facebook's Gorilla paper (VLDB 2015) introduced the scheme most TSDBs now use:

• Timestamps: delta-of-delta. Scrapes at 1760100000, 1760100015, 1760100030 have deltas 15, 15 → delta-of-delta 0, encoded in a single bit. Jitter of a few ms costs a few extra bits.
• Values: XOR with the previous value. A gauge that stays at 42.0 XORs to 0 (1 bit). Slowly changing values share sign, exponent and high mantissa bits, so the XOR has long runs of leading and trailing zeros; only the meaningful middle bits are stored.

The paper reported an average of about 1.37 bytes per sample on Facebook's data, versus 16 bytes uncompressed. That 10x is why a month of metrics fits on a handful of machines.

The inverted index

Queries select series by labels: http_requests_total{service="checkout", status=~"5.."}. The index maps each label=value pair to a sorted postings list of series IDs, exactly like a search engine:

__name__="http_requests_total"  →  [3, 7, 9, 12, 15, 40, ...]
service="checkout"              →  [7, 9, 15, 88, ...]
status="500"                    →  [9, 15, 102, ...]
intersection                    →  [9, 15]

Intersecting sorted lists is linear in their lengths. This is the structure that cardinality destroys: a label with 10 million distinct values creates 10 million postings entries and 10 million series.

Why not Cassandra or a SQL table?

You can build a TSDB on a wide-column store (OpenTSDB on HBase and KairosDB on Cassandra did), with row key = series + time bucket. It works, but you pay per-cell overhead far above 1.4 bytes, and label queries need a separate index. Purpose-built TSDBs (Prometheus TSDB, VictoriaMetrics, M3DB, InfluxDB's engine) exist because the access pattern is narrow and compressible. In an interview, name the option, then explain why the specialised format wins.

Query Path

• Query frontend: splits a 7-day query into per-day sub-queries, runs them in parallel, and caches results for completed (immutable) days. Dashboards reload the same panels constantly, so this cache has a very high hit rate.
• Queriers fetch recent data (last ~2 hours) from ingesters (all replicas, deduplicated) and older data from blocks in object storage via a store gateway that caches block indexes and chunks.
• Choose resolution automatically: a 30-day graph doesn't need 15-second points; use 5-minute rollups. A dashboard 1,500 pixels wide can only show ~1,500 points anyway.
• Recording rules precompute expensive aggregations (sum by (service) (rate(http_requests_total[5m]))) every minute and store them as new series, so dashboards and alerts query small, cheap series.
• Query limits: max series touched, max samples, timeout per tenant, so one bad query ({__name__=~".+"} over 30 days) can't take down the read path.

Downsampling and Retention

Engineers debug with raw data from the last few days; capacity planners look at trends over months. Store each at the resolution it needs:

Rollups must be mergeable

If you store only the average per 5-minute window, you can't compute a correct 1-hour average later (windows may have different sample counts), and you've lost the spike that triggered the incident. Store count, sum, min and max per window (and for counters, the increase). From those you can derive average, extremes and rates at any coarser resolution:

from dataclasses import dataclass

@dataclass
class Rollup:
    count: int = 0
    total: float = 0.0
    min: float = float("inf")
    max: float = float("-inf")

    def add(self, v):
        self.count += 1
        self.total += v
        self.min = min(self.min, v)
        self.max = max(self.max, v)

    def merge(self, o):
        self.count += o.count
        self.total += o.total
        self.min = min(self.min, o.min)
        self.max = max(self.max, o.max)

    @property
    def avg(self):
        return self.total / self.count

def downsample(samples, step):
    out = {}
    for ts, v in samples:
        out.setdefault(ts - ts % step, Rollup()).add(v)
    return dict(sorted(out.items()))

raw = [(0, 10), (15, 12), (30, 11), (45, 90), (60, 10), (75, 10)]
m1 = downsample(raw, 60)
for bucket, r in m1.items():
    print(bucket, r.count, r.avg, r.min, r.max)
# 0  4 30.75 10 90     <- the spike to 90 survives in max
# 60 2 10.0  10 10

whole = Rollup()
for r in m1.values():
    whole.merge(r)
print(whole.avg)                              # 23.83 (correct: 143 / 6)
print((m1[0].avg + m1[60].avg) / 2)           # 20.375 (wrong: average of averages)

The dry run shows both lessons: the max preserves the spike to 90 that an average would hide, and merging sums and counts gives the true mean (23.83) while averaging averages gives a wrong one (20.375), because the two windows hold different numbers of samples.

For histograms, downsampling sums bucket counts, which keeps quantile estimation possible at coarse resolution. Retention is enforced by the compactor deleting blocks past their tier's age; with object storage lifecycle policies, older tiers move to cheaper storage classes automatically (see object storage design).

Cardinality Explosion: The Classic Failure

A developer adds one label to make debugging easier:

http_requests_total{service="checkout", status="500", user_id="u_8812391"}

With 5 million active users, that metric goes from tens of series to tens of millions. Each new series costs memory in the head block, index entries and a new chunk. Ingesters run out of memory, restart, replay WALs that are now huge, and the monitoring system goes down during exactly the incident you needed it for.

Do the math

Series for one metric = product of distinct values of each label. service (50) × endpoint (40) × status (10) × pod (2,000 across all services, but each service's pods only report their own endpoints) is manageable. Add user_id (5M) and it's catastrophic. Unbounded values (user IDs, request IDs, raw URLs with IDs, email addresses, timestamps, error messages) never belong in metric labels.

Defences

1. Per-tenant and per-metric series limits enforced at the distributor: reject new series beyond the limit with a clear error, rather than accept and crash.
2. Relabeling/drop rules at the agent to strip known-bad labels, or aggregate them away before sending.
3. Cardinality dashboards: top metrics by series count, series churn per deploy, labels with the most values. Make the cost visible to the team that created it.
4. Guidance in client libraries and code review: label values must come from a bounded set; normalise URLs to route templates (/orders/{id}, not /orders/91823).
5. Use the right tool: per-user or per-request detail belongs in logs or traces (high cardinality is their job), linked from metrics via exemplars. See distributed logging design.
6. Watch churn: Kubernetes pod names change on every deploy, so pod labels create new series continuously. Series that stop receiving samples still occupy the index until their block ages out; high churn inflates the index even if active series look fine.

Alert Rule Evaluation

An alert rule is a query plus a condition and a duration:

groups:
- name: checkout-slo
  interval: 30s
  rules:
  - alert: CheckoutHighErrorRate
    expr: |
      sum(rate(http_requests_total{service="checkout", status=~"5.."}[5m]))
        /
      sum(rate(http_requests_total{service="checkout"}[5m])) > 0.05
    for: 2m
    labels:
      severity: page
      team: payments
    annotations:
      summary: "Checkout 5xx ratio above 5% for 2 minutes"
      runbook: "https://runbooks.internal/checkout-5xx"

The evaluator runs every rule group on its interval. For each result series, it tracks a state machine: inactive → pending (condition true) → firing (true continuously for the for duration) → resolved (condition false). The for clause prevents a single noisy evaluation from paging someone.

class AlertRule:
    def __init__(self, name, threshold, for_s):
        self.name, self.threshold, self.for_s = name, threshold, for_s
        self.state, self.active_since = "inactive", None

    def evaluate(self, value, now):
        if value > self.threshold:
            if self.state == "inactive":
                self.state, self.active_since = "pending", now
            if self.state == "pending" and now - self.active_since >= self.for_s:
                self.state = "firing"
                return ("FIRE", self.name, value)
        else:
            if self.state == "firing":
                self.state, self.active_since = "inactive", None
                return ("RESOLVE", self.name, value)
            self.state, self.active_since = "inactive", None
        return None

rule = AlertRule("CheckoutHighErrorRate", 0.05, for_s=120)
for t, v in [(0, 0.01), (60, 0.07), (120, 0.09), (180, 0.08), (240, 0.02)]:
    print(t, v, rule.evaluate(v, t), rule.state)
# 0   0.01 None inactive
# 60  0.07 None pending        <- condition true, timer starts
# 120 0.09 None pending        <- only 60 s so far
# 180 0.08 ('FIRE', ...) firing    <- true for 120 s
# 240 0.02 ('RESOLVE', ...) inactive

Scaling rule evaluation

• Shard rule groups across evaluator instances by hashing the group name; each group is evaluated by exactly one instance (with a standby or a replicated pair whose duplicate alerts Alertmanager dedupes).
• Evaluate against recording rules rather than raw high-cardinality series.
• Persist alert state (pending-since timestamps) so an evaluator restart doesn't reset every for timer and delay pages.
• Handle missing data explicitly: "no data" can mean healthy-and-idle or dead-and-silent. Use absent() alerts for critical metrics and up == 0 for scrape targets.

What to alert on

Page on symptoms users feel (error rate, latency SLO burn, checkout success rate), not on every cause (CPU at 85% on one pod). SLO burn-rate alerts, popularised by Google's SRE workbook, fire when the error budget is being consumed fast enough to matter, using a short and a long window together to be both fast and resistant to blips. Cause-based signals go to dashboards and tickets, not pagers.

Alert Routing and Noise Control

Firing alerts flow to an Alertmanager-style service that turns raw alerts into a small number of useful notifications:

Notification delivery must be reliable: persist outgoing notifications, retry with backoff, and use more than one channel for pages (push + SMS + phone call). Run Alertmanager as a clustered set that gossips silences and notification state, so losing one instance neither drops nor duplicates pages.

Monitoring the Monitor

The monitoring system is most needed when everything else is failing, so its failure modes deserve explicit design:

• Dead man's switch: an alert that always fires (vector(1)) and is sent to an external service. If that external service stops receiving it, it pages you: the alerting pipeline itself is broken.
• Independent meta-monitoring: a small, separate Prometheus (different cluster, ideally different region or provider) that watches the main system's ingestion rate, rule evaluation lag and notification failures.
• Isolation from production dependencies: the monitoring stack shouldn't depend on the same database, DNS, or Kubernetes cluster it monitors, or an outage there blinds you.
• Graceful degradation: under overload, shed dashboard queries before rule evaluation, and rule evaluation for low-severity rules before paging rules. Ingestion backpressure should drop low-priority tenants' samples before high-priority ones.
• Clock and staleness: samples with far-future or far-past timestamps are rejected; stale series are marked so graphs don't draw flat lines through gaps.

Interview Angle

A 45-minute plan

Follow-ups to prepare

• "How do you compute p99 latency across 500 pods?" Sum histogram buckets across pods, then histogram_quantile; never average per-pod p99s.
• "An ingester dies. Do we lose data?" Replication factor 3 with quorum writes; the WAL on the dead node is replayed when it returns; queries deduplicate replicas.
• "How do you support 13 months of retention cheaply?" Object storage + downsampled tiers + compaction; only recent data in memory/SSD.
• "How would you detect anomalies without fixed thresholds?" Compare against the same time last week, or use seasonal baselines computed by recording rules; keep them for dashboards and tickets, since they're noisy for paging.
• "How is this different from logging?" Metrics are numeric, pre-aggregated and cheap per sample with bounded cardinality; logs are high-cardinality events. The aggregation side of metrics is related to ad click aggregation, where the same windowed-rollup ideas apply.

Common mistakes

• Storing metrics in MySQL rows or one Kafka message per sample with no aggregation story.
• Ignoring cardinality, or adding user IDs as labels.
• Downsampling to averages only.
• Paging on every threshold, with no grouping, inhibition or silences.
• A monitoring system that shares fate with the system it monitors, and nothing watching the monitor.

Get the data model and cardinality right, explain why the TSDB compresses so well, and treat alerting as its own reliability-critical pipeline. That combination covers what interviewers at infrastructure-heavy companies are looking for in this question.

Frequently Asked Questions

Pranjal Jain

Ex-Microsoft SDE · IIT Kanpur · Founder of Prepflix. Helps engineers crack startup and product company interviews across India.