Reducing cardinality

This page is designed to guide you through the process of using the Prometheus configuration to eliminate high cardinality metrics and labels that contribute to an excessive number of active series within the platform.

Dropping metrics

remote_write configuration in Prometheus can be utilized to drop certain metrics from being sent to the remote storage endpoint. Here's how you can do it:

Identify the Metrics: First, you need to identify the metrics you wish to drop. You can use the dashboards mentioned in this section to help identify high cardinality metrics that are contributing a large number of active series.
Modify the Configuration: Once you've identified the metrics, proceed by changing the configuration in your prometheus.yml file. Under the remote_write section, you'll need to add a write_relabel_configs section. This is where you specify the metrics you want to drop. Dropping individual metrics looks like this:
```
remote_write:
  - url: 'http://remote-write-endpoint'
    write_relabel_configs:
    - source_labels: [__name__]
      regex: '(metric_to_be_dropped|another_metric)'
      action: drop
```
Dropping entire namespaces looks like this:
```
remote_write:
  - url: 'http://remote-write-endpoint'
    write_relabel_configs:
    - source_labels: [__name__]
      regex: '((?:namespace1_|namespace2_).+)'
      action: drop
```
Using regex

Regular Expressions (regex) can be utilized in the regex part of the Prometheus configuration to pattern-match the names of the metrics you wish to drop. The regex acts as a filter. Any metric name that matches the expression will be included in the action specified, in this case, the 'drop' action.

For instance, if you want to drop all metrics that start with 'http', your regex could be 'http.*'. If you want to drop specific metrics, you can list them out within parentheses, separated by '|'. For instance, 'metric1|metric2|metric3'.

Remember, regex expressions can be as simple or as complex as needed, providing you with flexible options for managing your metrics. For the complete documentation, take a look at the Google RE2 docs.
Apply the New Configuration: Save your changes and apply the new configuration. If you're running Prometheus in a Kubernetes environment, this might involve updating a ConfigMap and restarting the Prometheus pods.
Verify the Results: Lastly, go back to your active series dashboard and ensure that the dropped metrics are no longer being sent to Observe by Cyso or contributing to your total active series.

Finding metrics to be dropped

Remember, remote_write can be tuned in order to drop even more metrics or labels. Find metrics and labels to drop using our Cardinality dashboards.

The aforementioned steps should guide you in managing your active series count more effectively, optimizing both the cost and performance of your monitoring systems.

Dropping labels

Dropping labels interacts with a similar process as dropping metrics, using the write_relabel_configs feature. The difference is found in the source_labels: section where you specify the label you wish to drop instead of a metric.

Here's an example of how to drop a specific label:

remote_write:
  - url: 'http://remote-write-endpoint'
    write_relabel_configs:
    - source_labels: [label_to_be_dropped]
      action: labeldrop

In this above code snippet, simply replace label_to_be_dropped with the name of the label you want to drop.

Just like with metrics, this configuration will stop Prometheus from sending any series with this label to the remote_write endpoint, hence reducing the number of active series.

Dropping multiple labels

Dropping multiple labels follows the same procedure as dropping a single label, but with multiple label names specified within the source_labels section in the write_relabel_configs block.

Here's an example of how to drop multiple labels:

remote_write:
  - url: 'http://remote-write-endpoint'
    write_relabel_configs:
    - source_labels: [label1_to_be_dropped, label2_to_be_dropped, label3_to_be_dropped]
      action: labeldrop

In the above code snippet, replace label1_to_be_dropped, label2_to_be_dropped and label3_to_be_dropped with the names of the actual labels you want to drop.

By specifying multiple labels in the source_labels section, Prometheus will not send over to the remote_write endpoint any series with these labels, consequently reducing the number of active series. This offers a powerful and flexible way to manage the labels collected and stored by your monitoring system.

Metrics known for high cardinality

Depending on your use case, not every metric is useful. Below is a list of metrics that are known for their high volume:

Metric	Source	Expected cardinality
`etcd_*`	metrics about etcd, usually present on Kubernetes clusters	> 10k metrics on a small Kubernetes cluster
`apiserver_*`	metrics about the Kubernetes API server	> 10k metrics on a small Kubernetes cluster
`kubelet_*`	metrics about the Kubernetes Kubelet process	> 2k metrics on a small Kubernetes cluster
`node_systemd_unit_state`	metrics about the state of SystemD units, produced by node-exporter	> 8k metrics on a cluster of about 20 VMs