Prometheus: Metrics-Based Monitoring for Modern Infrastructure

How Prometheus collects data

Most monitoring systems use a push model: the application sends metrics to a central server. Prometheus inverts this. It polls targets on a schedule. Each target exposes a plain-text HTTP endpoint at /metrics. Prometheus scrapes that endpoint every few seconds and stores what it finds. The target does not need to know where Prometheus is running.

The pull model has practical advantages. You can scrape the same target from multiple Prometheus instances for redundancy. You can tell from a failed scrape that a target is down, which a push model cannot do if the target simply stops sending. Targets do not need Prometheus credentials. Access flows one way: Prometheus reaches out to targets, not the other way around.

Metric types

Prometheus defines four core metric types. Each one is designed for a different kind of measurement. Picking the right type matters because it affects how you query the data and what questions you can reasonably answer.

Labels and time series

Every metric in Prometheus is identified by a name and a set of key-value pairs called labels. Each unique combination of metric name and label values is a separate time series. Labels are what make a single metric name useful across many dimensions.

A counter called http_requests_total on its own tells you the total count of HTTP requests. Add a status label and you can split that count by response code. Add a handler label and you can see request counts per endpoint. The power is in being able to aggregate across labels when you want the big picture, then narrow down to specific label values when something looks wrong.

Labels are cheap to add but not free to maintain. Each unique label value combination creates another time series. A label with unlimited unique values, like a user ID or a session token, will create as many time series as there are unique values. That number of unique values blows up storage. The practical guidance is: use labels for dimensions you actually query across and keep the set of possible values for each label small.

PromQL

PromQL is the query language for Prometheus. It is built around the idea of operating on time series as first-class values. You select series by name and labels, apply functions to them, aggregate across label dimensions and compute things like rates of change and percentiles. The queries that look intimidating at first tend to follow a small number of patterns once you understand what the functions are doing.

The rate() function is one of the first things worth understanding properly. Counters are monotonically increasing numbers. The raw value is not very useful on its own. What you usually want to know is how fast the counter is increasing, which is exactly what rate() gives you. It calculates the per-second rate of increase over a specified time window and handles counter resets gracefully when a process restarts.

Alerting

Prometheus evaluates alerting rules on a regular interval. A rule is a PromQL expression with a threshold. When the expression evaluates to a non-empty result the alert enters a pending state. If it stays pending long enough, it fires. Fired alerts are pushed to Alertmanager, which handles routing, deduplication, grouping and silencing before sending notifications to wherever they need to go.

The for duration prevents transient spikes from generating noise. An alert that fires immediately on a single bad data point will page someone for a blip that resolved itself before they even looked at it. Requiring the condition to persist for several minutes filters out the noise while still catching real problems fast enough to matter.

Service discovery

In a static environment you can list scrape targets by IP address in the Prometheus configuration file. In practice, infrastructure moves around. Containers restart on different nodes. Services scale up and down. Maintaining a static list by hand stops being viable quickly.

Prometheus supports service discovery integrations for most of the common environments: Kubernetes, Consul, EC2, Azure, GCE and more. In a Kubernetes environment, Prometheus can discover pods and services directly from the Kubernetes API. It watches for changes and updates its scrape targets automatically. Annotations on pods tell Prometheus whether to scrape them, on which port and at which path. New deployments become visible to Prometheus without any manual configuration change.

Storage

Prometheus stores data in a custom time series database on local disk. The format is optimised for the specific access patterns of metrics: writes are append-only, reads are typically range queries over recent time windows. Compression is aggressive because the values being stored follow predictable patterns.

Local storage is fast but not designed for long-term retention at large scale. The default retention is fifteen days. For longer retention most teams integrate remote write: Prometheus streams metrics to an external system as they are collected. Thanos, Cortex and VictoriaMetrics are common choices. They add long-term storage, horizontal scalability and global query across multiple Prometheus instances. Prometheus continues handling scraping while the remote system handles retention and aggregation at scale.

Exporters

Not every piece of infrastructure can be instrumented directly. A database, an operating system kernel, a load balancer. Exporters bridge the gap. An exporter is a process that talks to a system using its native interface, translates the metrics it gets back into the Prometheus exposition format and exposes them on an HTTP endpoint ready to be scraped.