Observability

1.1One mean and three percentiles

Let's build a stand-in for the checkout traffic and see what the dashboard's number does with it. The script below makes up 10,000 request times, or latencies (the time from a request arriving to its response going out). Ninety-nine percent of them are around 20 milliseconds, and the other 1% are slow, between 1.5 and 2.5 seconds. It then summarises them four ways: the mean (the average), the median, the p99 and the p99.9.

The last three are percentiles. Sort all the requests from fastest to slowest and walk along the list. The median, also written p50, is the request halfway along, so half of the requests are faster than it. The p99 is the request 99% of the way along, so only one request in a hundred is slower. The p99.9 is 99.9% of the way along, with one request in a thousand slower.

The random generator starts from a fixed seed, so you'll get exactly the numbers below when you run it.

import random, statistics
 
rng = random.Random(11)
lat = [rng.gauss(20, 3) if rng.random() < 0.99 else rng.uniform(1500, 2500) for _ in range(10_000)]   # ms
lat.sort()
print(f"requests : {len(lat):,}   (99% take about 20 ms, 1% take 1.5-2.5 s)")
print(f"mean     : {statistics.mean(lat):7.1f} ms")
print(f"median   : {lat[len(lat) // 2]:7.1f} ms")
print(f"p99      : {lat[int(0.99 * len(lat))]:7.1f} ms")
print(f"p99.9    : {lat[int(0.999 * len(lat))]:7.1f} ms")

requests : 10,000   (99% take about 20 ms, 1% take 1.5-2.5 s)
mean     :    40.3 ms
median   :    20.1 ms
p99      :  1546.2 ms
p99.9    :  2408.3 ms

Compare the mean with your prediction. The median, 20.1 ms, describes the typical request accurately. The mean, 40.3 ms, is double that and matches nobody's experience, and it's the number our dashboard was showing. The p99 of 1,546 ms sits right at the start of the slow group, because the slow group is exactly 1% of the requests, and the p99.9 of 2,408 ms is deep inside it. Those two describe the requests that made users leave. A hospital that reports its average patient temperature has the same problem: the average can be normal while one patient in a hundred has a dangerous fever.

1.2What we would have to record

Our script could report all of this because it had the full list of 10,000 times in memory. A real service doesn't keep that list, and the dashboard only ever sees whatever summary someone decided to compute. To find our customer's request we need two different things. First, the overall shape of the latencies, kept cheaply and continuously, so that a slow 1% shows up at all. Second, the detail of individual requests, so that when we find a slow one we can see what happened to it.

The simplest way to get both is also the first thing most people try: at the end of every request, write down a line describing it. Let's do that and see what it costs.

2.1One structured line per request

A line that a service writes to describe an event is called a log. The easiest log line to write is a sentence, such as order 4412 ok, and the hardest to use: to find slow orders in a pile of sentences you end up searching with regular expressions at 3 a.m. So instead we write one JSON object per event, with the same field names every time. JSON is text with named fields, which every language can read and write, and a log written this way is called a structured log. Here is the line for our customer's request (it's wrapped across lines here so it fits the page; in the log it's one line):

{"ts":"2026-09-27T09:41:07.118Z","level":"info","msg":"request done",
 "service":"checkout","route":"/api/v1/orders","method":"POST","status":201,
 "duration_ms":3712,"trace_id":"4bf92f3577b34da6a3ce929d0e0e4736",
 "span_id":"00f067aa0ba902b7","customer_id":"c_81723"}

Every field answers a question we'll want to ask later. route, method and status say what kind of request it was and how it ended, duration_ms says how long it took (3,712 ms, the almost four seconds our customer waited), and customer_id says whose it was. The two IDs, trace_id and span_id, are there to join this line to other records. The trace ID is a unique name for this request that the services handling it pass along to each other, and section 7 explains how. For now, treat it as a label that will let us find the same request in other places.

With this line written for every request, the complaint is answerable. Search the last hour of logs for customer_id equal to c_81723, and the request is there, with its duration, status and IDs.

2.2What it costs

That line is 262 bytes when written on one line. A thousand requests a second, all day, comes to this:

Nearly all of those 22.6 GB describe fast, healthy requests that nobody will ever look at. And there's a second cost in reading them back. Asking "what fraction of requests in the last hour were slower than a second?" means scanning 3.6 million lines. Logs answer "what happened to this request?" well, and answer "how are we doing overall?" expensively. Two questions follow: how do log systems find a line without reading everything, and how do we keep the volume down?

2.3Searching logs: two ways to index

To avoid reading every line, a log store builds an index, a lookup table made ahead of time so a search can jump straight to the matches. How the store indexes decides both what you can ask it and what it costs:

An inverted index works like the index at the back of a book. For each word it keeps the list of lines that contain it, so finding every line containing c_81723 means reading one list instead of the whole log. The price is that every word of every line has to be added to some list as it arrives. Loki makes the opposite trade. It groups lines into streams, one per combination of a few tags such as the service name, and indexes only the tags. Its overview says it "does not index the contents of the logs, but only indexes metadata about your logs as a set of labels for each log stream." Writing is cheap, and a query that can't narrow itself down by those tags has to read a lot of text. Loki calls the tags labels, and they follow the same rules as the metric labels in section 5, where you'll see why.

2.4Keeping the volume down

The other lever is writing fewer lines. Each technique below gives up some detail to save storage, and each has a way to go wrong:

One more hazard comes from writing the line itself. A log call on the request path formats the text, takes a lock on the output, and writes it. At high request rates that synchronous write becomes the latency, and a slow disk, or a full pipe to the log shipper (the agent that carries log lines from the machine to the log store), blocks every request thread until it drains.

Even with all of these, a log is still one record per event, so its cost rises with traffic. What we want for the question "how are we doing overall?" is something whose cost doesn't depend on traffic at all. Counting does that.

3.1A series is a name plus labels

One counter for all requests isn't much use, because we'd want to know about failures separately from successes. So the service keeps one counter for each kind of request, told apart by labels: pairs such as method="POST" and status="201" attached to the metric name. Each distinct combination of a name and its labels, together with its history, is called a time series. A single reading on it, a timestamp and a number, is a sample. Here are three series of one metric, each with its first two samples:

http_requests_total{method="POST", route="/api/v1/orders", status="201"} → 1027 @ t1, 1051 @ t2, …
http_requests_total{method="POST", route="/api/v1/orders", status="500"} → 3 @ t1, 3 @ t2, …
http_requests_total{method="GET",  route="/api/v1/orders", status="200"} → 88 @ t1, 90 @ t2, …

Inside Prometheus the name is just one more label, called __name__. A series is its full label set, sorted, and a sample is a (timestamp, float64) pair on it. Our customer's request is one of the increments of the first series.

The metric types, of which there are four, exist only in the client library that the service uses. On the wire and in the database they are all series of float64 samples. The last column names the query functions you use on each type; rate() comes up in 3.3 and the histogram ones in section 6:

3.2Pull: the scrape

The service holds its current counter values, so something has to carry them to Prometheus and build the history. Prometheus does this by pulling. Each service exposes its current values as plain text over HTTP, usually at the path /metrics, and Prometheus fetches that page on a fixed interval, often 15 seconds. Each fetch is a scrape. The page looks like this:

# HELP http_requests_total Requests handled.
# TYPE http_requests_total counter
http_requests_total{method="POST",route="/api/v1/orders",status="201"} 1027
http_requests_total{method="POST",route="/api/v1/orders",status="500"} 3

Here is what happens to the first series over two scrapes, and then when the service goes down:

Watch the right-hand box. The page on the left only ever holds the latest count, and Prometheus's series on the right is where the history builds up, one sample per scrape. Between the fetch and the store, Prometheus parses each line, applies metric_relabel_configs (rules that can drop series or labels, which section 5 uses), and attaches the labels job and instance that say which target the sample came from. It also records up for every scrape and scrape_duration_seconds for how long it took.

?Why pull instead of having services push?

Because pull gives Prometheus the list of what should exist. If a target is down, the scrape fails and up becomes 0, and you can alert on that. With push, a dead service just goes quiet, and silence looks the same as no traffic. Pull also puts the rate under the server's control, so a misbehaving client can't flood it. The price is that short-lived jobs and targets Prometheus can't connect to, such as machines on a private network, need help: the Pushgateway, a small server that batch jobs push to and Prometheus scrapes, or an agent that scrapes locally and forwards the samples with remote_write.

3.3rate() and counter resets

Prometheus now holds a growing list of samples per series, but a raw counter value such as 1,051 is almost never what you want to see on a graph. What you want is how fast it's climbing, and rate(x[5m]) computes that. It's written in PromQL, Prometheus's query language. The [5m] part selects the samples from the last five minutes, and rate turns them into a per-second increase, just as we did by hand in the scene above (24 orders in 15 seconds is 1.6 per second).

?Why doesn't a restart break the graph?

Because rate() treats any decrease as a reset. If a counter goes 1000, 1040, 12, 50, it assumes the process restarted and counted up from zero, and adds the 12 to the increase. That's why you must never use rate() on a gauge: every drop looks like a reset. rate() also extrapolates to the edges of the window, so increase() over a window often returns a non-integer for an integer counter. And it needs at least two samples in the window, so the range should be at least four scrape intervals, which lets it survive one failed scrape.

We now have cheap counts with history. A raw sample is 16 bytes, an int64 timestamp and a float64 value, and a service with a few thousand series scraped every 15 seconds adds samples all day. Next we'll look at how Prometheus stores them without that adding up.

4.1Where a sample goes

A new sample first goes to the place that's fastest to write and read: memory. Prometheus keeps the newest data in an in-memory part called the head. Memory vanishes in a crash, so every sample is also appended to a write-ahead log (WAL) on disk, the same idea that filesystems and databases use: write down what you're about to rely on first, so that after a crash you can replay it. Within the head, the samples of one series are packed together and compressed into a chunk, which is cut at about 120 samples, half an hour of data at a 15-second scrape interval. Finally, old data is written out as a block: a two-hour slice of the history, stored as chunks plus an index, which is never changed once written.

Follow one sample, the 1,051 from the scrape above, through all four:

The head is the only part that has to be kept in RAM, so the number of active series decides how much memory Prometheus needs. The three-hour threshold is 1.5 times the two-hour block range. The scene also shows why the log exists: it's the only thing between a crash and losing everything the head held.

4.2Gorilla compression

Inside a chunk, each sample costs far less than 16 bytes, because monitoring data is predictable. The Gorilla paper (Pelkonen et al., VLDB 2015) observed two things about it and compressed each:

• Timestamps are regular. Scrapes happen every 15 seconds, so the gap between one sample's timestamp and the next is nearly constant. Prometheus stores times in milliseconds, so scrapes at 0, 15,000 and 30,000 ms have gaps of 15,000 and 15,000. The difference between gaps, the delta of the delta, is zero, and a zero can be stored in one bit. Gorilla stored "about 96% of all time stamps" in a single bit.
• Values change slowly. Take a value and the one before it and combine their bits with XOR, which gives a 0 wherever the bits agree. If the value didn't change, the result is all zeros, and when it changed a little most of the bits are still zero, so you store only the meaningful bits in the middle. In Gorilla's data, "roughly 51% of all values are compressed to a single bit" because they hadn't changed.

Together that took Facebook's data "down from 16 bytes to an average of 1.37 bytes" per point. Here's Prometheus's version of the timestamp half:

func (a *xorAppender) Append(t int64, v float64) {
	var tDelta uint64
	num := binary.BigEndian.Uint16(a.b.bytes())
	switch num {
	case 0:   /* first sample: full timestamp (varint) and full 64-bit value */
	case 1:   /* second: timestamp delta, then XOR'd value */
	default:
		tDelta = uint64(t - a.t)
		dod := int64(tDelta - a.tDelta)
 
		// Gorilla has a max resolution of seconds, Prometheus milliseconds.
		// Thus we use higher value range steps with larger bit size.
		switch {
		case dod == 0:
			a.b.writeBit(zero)
		case bitRange(dod, 14):
			a.b.writeByte(0b10<<6 | (uint8(dod>>8) & (1<<6 - 1))) // 0b10 size code combined with 6 bits of dod.
			a.b.writeByte(uint8(dod))                             // Bottom 8 bits of dod.
		case bitRange(dod, 17):
			a.b.writeBits(0b110, 3)
			a.b.writeBits(uint64(dod), 17)
		case bitRange(dod, 20):
			a.b.writeBits(0b1110, 4)
			a.b.writeBits(uint64(dod), 20)
		default:
			a.b.writeBits(0b1111, 4)
			a.b.writeBits(uint64(dod), 64)
		}
 
		a.writeVDelta(v)
	}
	/* ... */
}

Read the switch from the top. When dod is zero, the whole timestamp costs one bit. Otherwise the code picks the smallest of four sizes that holds the difference, and writes a short prefix saying which one. A perfectly regular scrape costs one bit of timestamp per sample. A scrape that arrives a few milliseconds late costs 16 bits, because the smallest non-zero case is 14 bits of difference plus a 2-bit prefix.

4.3What a sample costs

Prometheus's storage docs say it stores "an average of only 1-2 bytes per sample". That average hides a wide range, and the range follows from the two observations above. Take 2,000 series of 480 samples each (two hours at 15 s) in four shapes, build blocks from them with promtool tsdb create-blocks-from openmetrics, and divide the chunk bytes by the number of samples:

These are Prometheus v3.9.1 promtool figures. The index adds roughly 75–90 bytes per series per block on top.

Look at the middle two rows. The counter and its timing are the same; the only change is that scrapes arrive a little early or late. Timestamp jitter nearly doubles the size, because it moves every timestamp out of the one-bit case. With ±50 ms of jitter the delta of the delta is almost never zero, so each sample pays 16 bits of timestamp instead of 1, and that is roughly the 1.8-byte difference between the two rows. A random float is worse again, because its XOR with the previous value has few zero bits to drop.

Disk and memory therefore scale with samples, and samples scale with series times scrape frequency. Halving the scrape interval doubles the bill for every series. The capacity formula from the docs is retention_seconds × samples_per_second × bytes_per_sample, with bytes per sample closer to 2 than 1 for real counters.

Now we can finish the comparison we started in section 2. One checkout service at 1,000 requests a second, logs against metrics:

The counters cost the same at ten requests a second as at a million, while the logs grow with every request. That ratio is why you alert on and graph metrics, and why logs and traces are sampled, filtered, or kept only for a short time.

4.4Finding series by label

Storing samples cheaply is half the job. A query names series by label, as in http_requests_total{status="500"}, and Prometheus has to find the matching series among millions without looking at all of them. It uses the same structure as the log search engines in section 2.3, an inverted index. For each label pair it keeps a sorted list of the IDs of the series that have it, called a postings list:

// MemPostings holds postings list for series ID per label pair. They may be written
// to out of order.
// EnsureOrder() must be called once before any reads are done. This allows for quick
// unordered batch fills on startup.
type MemPostings struct {
	mtx sync.RWMutex
 
	// m holds the postings lists for each label-value pair, indexed first by label name, and then by label value.
	/* ... */
	m map[string]map[string][]storage.SeriesRef
	/* ... */
}

The type is a map from label name to label value to a list of series references, so for each label pair you get the series that have it. A selector with several matchers intersects their lists. Here is a query that asks for the 500-status error rate of one job, step by step:

An equality matcher such as status="500" is one map lookup. A regular expression such as route=~"/api/.*" can't be looked up, because Prometheus has to test every value of the route label against the pattern and then merge all the matching postings lists. With a few values that costs nothing. With 100,000 values of one label, it becomes the query.

Everything in this section, memory for the head, bytes in chunks, index size and query time, scales with the number of series. So what decides how many series we get?

5.1Labels multiply

The number of series for one metric is, at worst, the product of the number of values of each label. Our counter has three series because it has three combinations of method and status that occur. Watch what happens when someone decides it would be useful to see which customer each request belongs to:

For one metric the worst case is the product over all its labels, and the next question gives a feel for how fast that grows.

The Prometheus naming guide says it directly: "Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values."

5.2What a series costs in memory

Every series in the head costs RAM: its in-memory record (a memSeries struct), its label set, its entries in the postings index, and the chunk being filled. To put a number on it, point Prometheus 3.9.1 at a static /metrics page with N series of a five-label counter, scrape it every 5 seconds for 90 seconds, and read the server's resident memory (RSS, the RAM the process holds):

RSS includes the headroom Go's garbage collector keeps, and you have to provision for that too. The cost per added series is the same at both sizes, about 2.9 KB, so memory grows in a straight line with series count.

?Why does cardinality hurt even when the traffic doesn't change?

Because the cost is per series, and series churn. Every pod restart in Kubernetes (a pod is one running copy of a service) creates a new pod label value, so a new set of series. The old ones stay in the head until it's compacted, up to about three hours. A rollout that replaces 40 pods briefly doubles the series count of every metric that carries a pod label.

5.3Finding and fixing it

When memory climbs or queries slow down, the first job is to find which metric or label is responsible. Three queries find most cardinality problems. The first lists the metric names with the most series, the second counts how many values one label has, and the third watches whether the head is growing:

# Which metric names have the most series?
topk(10, count by (__name__) ({__name__=~".+"}))
 
# Which label on one metric has the most values?
count(count by (customer_id) (http_requests_total))
 
# Is the head growing? (Prometheus's own metrics)
prometheus_tsdb_head_series
rate(prometheus_tsdb_head_series_created_total[5m])

Offline, promtool tsdb analyze <data-dir> lists the labels with the most values and the highest churn in a block. Once you know the culprit, there are five fixes, and they differ in what they keep:

Counters labelled by route and status now give us cheap, bounded counts. They still can't tell us how long requests took, and latency was the problem we started with.

6.1Histograms aggregate, summaries don't

The idea is to sort each request into a bucket by its latency, such as "up to 5 ms", "up to 10 ms", "up to 25 ms", and count how many land in each. A metric that does this is a histogram. In Prometheus each bucket is a counter that counts every observation at or below its upper bound, which is labelled le (for "less than or equal"). For example, http_request_duration_seconds_bucket{le="0.5"} 1204 means 1,204 requests so far took half a second or less. Each bucket is a series, so a histogram is a handful of series per label set, plus _sum and _count.

The alternative is a summary, which computes quantiles (a quantile is a percentile written as a fraction, so the p99 is the 0.99 quantile) inside each process and exports them. That's accurate for that process, but you can't combine them across servers: the p99 of ten servers isn't the average of their p99s, or any other function of them. Counts do combine. Buckets from many servers can be added together, and the percentile computed from the total:

histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Here rate() turns each bucket counter into a per-second rate, sum by (le) adds the same bucket across all replicas, and histogram_quantile() estimates the p99 of the total. That's why the Prometheus histogram guide recommends histograms whenever you need to aggregate, which for a service with more than one replica is always.

6.2How histogram_quantile() estimates

A histogram only knows how many observations fell in each bucket, not where in the bucket they were. To estimate a quantile, histogram_quantile() finds the bucket containing the target rank and assumes the observations are spread evenly inside it:

	observations := buckets[len(buckets)-1].Count
	/* ... */
	rank := q * observations
	b := sort.Search(len(buckets)-1, func(i int) bool { return buckets[i].Count >= rank })
 
	if b == len(buckets)-1 {
		return buckets[len(buckets)-2].UpperBound, forcedMonotonic, fixedPrecision
	}
	/* ... */
	var (
		bucketStart float64
		bucketEnd   = buckets[b].UpperBound
		count       = buckets[b].Count
	)
	if b > 0 {
		bucketStart = buckets[b-1].UpperBound
		count -= buckets[b-1].Count
		rank -= buckets[b-1].Count
	}
	return bucketStart + (bucketEnd-bucketStart)*(rank/count), forcedMonotonic, fixedPrecision

Take the code in two parts. The sort.Search finds the first bucket whose running count reaches the rank. The last line then interpolates: it starts at the bucket's lower bound and moves a fraction of the way to its upper bound, where the fraction is how far the rank is through the bucket's own count. If the p99 falls 60% of the way through the 100–250 ms bucket, the answer is 100 + 0.6 × 150 = 190 ms, whatever the real values were. Two consequences are visible. The estimate is a straight line inside one bucket. And if the rank lands in the last bucket, the one with no upper limit (+Inf), the answer is the highest finite bound, so a p99 can never report more than your largest bucket.

?How wrong can it be?

With wide buckets, probably more wrong than you'd guess. Draw 200,000 latencies from three distributions, bucket them with the default buckets of the official client libraries (5 ms, 10, 25, 50, 100, 250, 500 ms, 1 s, 2.5, 5, 10 s), and compare what histogram_quantile() says (reimplemented from the code above) with the true percentile:

The p50 of the bimodal case is off by 36% too. The first row's p99 falls in the 100–250 ms bucket, and interpolating linearly across a bucket that wide lands the estimate far from the real value. The fix is buckets that are narrow where your target is. A service level objective (SLO) is a target such as "99% of requests under 300 ms". If that's yours, put a bucket boundary at exactly 300 ms, and the question "what fraction was under 300 ms?" has an exact answer, even when the p99 estimate doesn't.

6.3Native histograms

Classic histograms make you choose bucket boundaries up front, and each bucket is a separate series. Native histograms store a whole histogram as one sample, with exponentially spaced buckets chosen automatically from a resolution parameter, and only the non-empty buckets stored.

In Prometheus 3.8 they became a stable feature, enabled per scrape config with scrape_native_histograms: true, and by 3.9.1 the old native-histograms feature flag is no longer among the listed feature flags. The client library has to expose them too.

Now the dashboard can show a p99 that jumped from 200 ms to 4 seconds. It still can't tell us which requests were slow, or what they were waiting for.

7.1Spans and trace context

A span is one timed operation: a request handled, a query run, a call made. It has a start, a duration, a parent, and attributes, key-value details such as the route or the customer ID. The spans of one request form a tree, because the gateway's span has the checkout call as a child, which has the inventory call as a child. That tree is a trace, and all of its spans share one trace ID.

For that to work across services, each call has to carry the trace ID and the caller's span ID to the next service, so the next service can attach its span in the right place. The standard way is the W3C Trace Context traceparent header, an extra line in the HTTP request:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                └ flags: 01 = sampled
             │  │                                └ parent span ID (8 bytes)
             │  └ trace ID (16 bytes)
             └ version

The flags field says whether this request is being recorded in full, called being sampled, and 7.2 explains who decides that and why. Here is our checkout request going through three services:

Look at the last two messages. Each service reports only its own spans, to a collector, a separate program that receives spans and passes them on. The whole trace exists only after the collector or a backend behind it joins the spans by trace ID, so whatever does the joining needs all the spans of a trace. That comes back in 7.3.

?Why does one uninstrumented service break the whole trace?

Because propagation is hop by hop. A service is instrumented when it has tracing code that reads the incoming traceparent and writes it on outgoing calls. If a service in the middle isn't instrumented, the services after it start new traces. You get two half-traces with no link between them. Proxies, queues and thread pools are the usual culprits: the context has to be carried across every asynchronous hand-off, too.

7.2Too many traces: head sampling

Recording a span for every operation in every service, and exporting all of them, is expensive, in the same way that logging every request was in section 2. Most systems record only a share of requests, and choosing the share is called sampling. The simplest way is head sampling: the decision is made at the very start of the request, at the root span, and passed down in the traceparent flags so that every service agrees. It's cheap, because an unsampled request records nothing anywhere.

Dapper, Google's tracing system, started this way. Its 2010 paper describes a first production version averaging one sampled trace for every 1,024 candidates.

The weakness is that the decision is made before anything interesting has happened. At 1 in 1,024, a failure that affects one request in 10,000 is captured about once every ten million requests. Our customer's four-second request had a 1-in-1,024 chance of being kept, and the sampler didn't know it was slow when it decided. We'd like to decide after the request has finished.

7.3Deciding afterwards: tail sampling

Tail sampling decides after the trace is complete: keep every trace with an error, every trace slower than a threshold, and a small share of the rest. The most common tracing toolkit is OpenTelemetry, an open standard with libraries for every major language plus a collector program of its own. The OpenTelemetry Collector has a tail sampling processor that does this, with policies such as status_code, latency, probabilistic and rate_limiting. Here is what it does with three checkout requests, one of them ours:

The two sampling methods trade against each other:

?Why does tail sampling need two layers of collectors?

Because a decision about a trace needs every span of it, and spans arrive from many services at different collectors. The processor's README says all spans of a trace "MUST be received by the same collector instance". The standard deployment is a first layer running the load-balancing exporter, which routes by trace ID, in front of a second layer running tail sampling. The second layer buffers every trace for decision_wait (30 s by default) and holds up to num_traces (50,000 by default) in memory.

7.4Inside the OpenTelemetry SDK

Sampling decisions and exporting happen inside each service, in a library. OpenTelemetry splits that library into an API (what other libraries call to create spans, and a no-op if nothing's installed) and an SDK (what the application installs to sample, batch and export them). A span's path out of the process:

The design decision that matters most is what happens when that queue is full. In opentelemetry-go, by default, the span is dropped and counted:

func (bsp *batchSpanProcessor) enqueueDrop(ctx context.Context, sd ReadOnlySpan) bool {
	if !sd.SpanContext().IsSampled() {
		return false
	}
 
	select {
	case bsp.queue <- sd:
		return true
	default:
		atomic.AddUint32(&bsp.dropped, 1)
		/* ... self-observability counter, error type queueFull ... */
	}
	return false
}

The select tries to put the span on the queue. If the queue has room it succeeds, and if not, the default branch runs: it adds one to a dropped-spans counter and gives up without waiting.

?Why drop spans instead of waiting?

Because telemetry must never take the service down with it. If the exporter is slow or the collector is unreachable, blocking would make every request wait on the tracing pipeline. The SDK spec defines the defaults (queue 2,048, batch 512, delay 5,000 ms, timeout 30,000 ms) and says plainly that once the queue is full, "spans are dropped." There's a WithBlocking() option for the rare case where losing spans is worse than latency.

We now have three signals that each hold part of the story about one request. The last question is how to get from one to the next.

8.1From a spike to the customer's request

Each jump needs something the two signals share. The link from metrics to traces is an exemplar: a trace ID attached to one observation in a histogram bucket.

http_request_duration_seconds_bucket{le="0.5"} 1204 # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736"} 0.43

The part after the # is an extension of the scrape format called OpenMetrics: one example observation of 0.43 seconds, and the trace it came from. A dashboard can show it as a dot on the latency graph that links straight to that request's trace. In Prometheus 3.9.1, storing exemplars still needs --enable-feature=exemplar-storage. The link from traces to logs is the trace_id field on every log line, and the link from logs back to metrics is using the same names for the same things: route, status, service.

Now go back to our customer and see what each signal did:

Notice that we never scanned 22.6 GB of logs. The counts were cheap enough to keep for every request, so they showed the spike. The exemplar turned the spike into one trace ID, tail sampling had kept that trace because it was slow, and the final log search was a lookup by one ID.

8.2What to measure first

You can't instrument everything at once, so two checklists cover most services. RED, for anything that serves requests: Rate, Errors, Duration. USE, from Brendan Gregg, for anything that is a resource: Utilisation, Saturation, Errors.

Alert on the RED metrics users feel, through SLO burn rates (chapter 40 explains these), and use the USE metrics to explain the alert once you're looking.

9.1Commands for the questions this chapter raised

Each question has a query or a tool that answers it on a running system.

# Is every target being scraped? (section 3.2)
up == 0
 
# How many series are there, and is the count growing? (section 5)
prometheus_tsdb_head_series
rate(prometheus_tsdb_head_series_created_total[5m])
 
# Which metric names have the most series? Which label has the most values? (section 5.3)
topk(10, count by (__name__) ({__name__=~".+"}))
count(count by (customer_id) (http_requests_total))
 
# What is the p99 across all replicas? (section 6.1)
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# Which labels have the most values and the highest churn in stored data? (section 5.3)
promtool tsdb analyze <data-dir>
 
# Are exemplars being stored for the dashboard dots? (section 8.1)
prometheus --enable-feature=exemplar-storage ...

9.2Rules that hold up

1. Alert and graph on metrics; use logs and traces for detail. The same service differs by about 4,000 times in volume between logs and metrics.
2. Bound every label. A label's values should be something you could list: routes, status classes, tiers. Never a raw path or an ID.
3. Use histograms, not summaries, and put a bucket boundary at your SLO threshold.
4. Use at least four scrape intervals as the rate() range, and never use rate() on a gauge.
5. Put trace_id on every log line, and use a ParentBased sampler in every service so a trace is kept whole or dropped whole.
6. Let telemetry drop under load; don't block on it. Count what's dropped and watch the counter.

9.3What you trade for what

9.4Symptom, cause, fix