Prometheus Cardinality: The Silent Killer of Your Monitoring Stack

In three years of running production Prometheus stacks, the single most common cause of instability I have seen is label cardinality explosion. It starts subtly — query latency increases, the server uses more RAM, scrapes start timing out — and then one morning Prometheus OOMs and your monitoring is dark right when you need it most.

What is cardinality?

In Prometheus, every unique combination of metric name and label values creates a separate time series. Cardinality is the total count of active time series in your TSDB.

A metric like this:

http_requests_total{method="GET", path="/api/users", status="200"}

...is one time series. But if you add a user_id label with 50,000 unique users:

http_requests_total{method="GET", path="/api/users", status="200", user_id="abc123"}

...you now have 50,000 time series for that one path alone. Multiply by methods, status codes, and other endpoints and you are looking at millions of series.

Measuring your current cardinality

Prometheus exposes cardinality stats via its API. Start here:

curl -s http://localhost:9090/api/v1/status/tsdb | \
  jq '.data.headStats'

The numSeries field is your total active series count. Above 5M, expect degraded query performance. Above 20M, you are heading toward instability.

To find which metrics are the worst offenders:

curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=10' | \
  jq '.data.seriesCountByMetricName[:10]'

Common culprits

1. High-cardinality HTTP labels

Never use raw URL paths as label values. /api/users/12345/posts/67890 is a unique series for every user and post combination. Normalise paths before recording:

// Bad
prometheus.Labels{"path": r.URL.Path}

// Good — use a route pattern from your router
prometheus.Labels{"path": "/api/users/:id/posts/:postId"}

2. User or request IDs in labels

This is the most common mistake I see in code reviews. IDs belong in log lines and trace spans — not in metric labels.

3. Unbounded error messages

// Bad: error strings are unbounded
prometheus.Labels{"error": err.Error()}

// Good: categorise errors
prometheus.Labels{"error_type": classifyError(err)}

Fixing existing high-cardinality metrics

If you have an existing metric with exploding cardinality, your options are:

1. Drop the offending label via a metric_relabel_configs rule in your scrape config:

metric_relabel_configs:
  - source_labels: [__name__, user_id]
    regex: "http_requests_total;.+"
    action: labeldrop
    target_label: user_id

2. Replace the label value with a bucketed or normalised version using replacement.

3. Record a new aggregated metric via a recording rule and deprecate the original.

Setting cardinality alerts

Add this alert to catch cardinality spikes early:

- alert: HighTSDBCardinality
  expr: prometheus_tsdb_head_series > 2000000
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus cardinality is growing — investigate new label values"

Summary

Cardinality is a resource that must be actively managed, not an afterthought. The key rules:

• Never use IDs (user, request, session) as label values.
• Always normalise URL paths before recording metrics.
• Monitor prometheus_tsdb_head_series and alert at a threshold that gives you headroom to react.
• Audit top-cardinality metrics weekly during rapid growth phases.

A Prometheus instance with disciplined labelling can run on a single node for years. One with cardinality rot will need replacing in months.