Tuning remote_write

In addition to simply enabling remote_write in Prometheus, you may also want to consider tuning the remote_write configuration for optimal performance. This involves adjusting parameters like queue_config to better match your environment's capacity and needs.

remote_write Tuning Parameters

queue_config

The queue_config parameter allows you to adjust various aspects of the remote write queue to improve the performance of your Prometheus server. Here is an example of queue_config with its parameters:

AMS2

remote_write:
  - url: "https://api.observe.cyso.com/{customer_id}/metrics/api/v1/push"
    basic_auth:
      username: "{customer_id}"
      password: "{token}"
    queue_config:
      capacity: 15000
      max_samples_per_send: 2500

Recommended queue_config

The above queue_config is recommended by Cyso and is different from the default config. Increasing the capacity and max_samples_per_send will improve performance and reduce resource usage by decreasing the total amount of remote_write requests made.

capacity

This is the number of samples that the queue can store in memory before blocking incoming samples. Increasing this can help reduce the blocking of incoming samples, but will increase memory usage.

max_samples_per_send

The maximum number of samples to send in one request to the remote endpoint. Increasing this can improve throughput, and reducing it can lower request latency.

Monitoring remote_write

To help with the tuning process, you can use Prometheus's own metrics. Prometheus provides detailed metrics about the remote_write component, including sent samples, retried samples, dropped samples, and queue length. Here are a few examples:

• prometheus_remote_storage_samples_in_total
• prometheus_remote_storage_dropped_samples_total
• prometheus_remote_storage_pending_samples

You can monitor these metrics to better understand how your remote_write is performing and make necessary adjustments.

More information

More information on the queue_config tuning can be found in the official Prometheus configuration.