Skip to content

Remote Write

Configuring remote write

This document will guide you on how to configure your Prometheus instance to write metrics to our Metrics API at Cyso.

For security reasons, this API is protected with Basic Authentication. This documentation will guide you through the process of setting this up.

Prerequisites

  1. Prometheus is installed and running.
  2. You have administrative access to your Prometheus server.
  3. You have your Cyso customer ID and API key on hand.

Configuration Steps

Follow the below steps to configure remote_write in your Prometheus instance:

  1. Open your Prometheus configuration file. This is typically located at /etc/prometheus/prometheus.yml, but the location can differ based on your installation.

  2. In the YAML configuration file, find the section labeled remote_write. If it does not exist, you can create it.

  3. Add the following configuration under remote_write:

Username, password and customer id

Replace the username and password in the following example with the credentials received from Cyso:

  • Username: will be your customer ID
  • Password: The API key received
  • URL: Replace {customer_id} with your customer ID
remote_write:
  - url: "https://api.ams2.observe.cyso.com/{customer_id}/metrics/api/v1/push"
    basic_auth:
      username: "{customer_id}"
      password: "{token}"
    queue_config:
      capacity: 15000
      max_samples_per_send: 2500

  1. This configuration includes the url where Prometheus will send the metrics. It also contains the basic_auth section, with your username and password.

  2. Save and close the file.

  3. Finally, to make the changes take effect, restart the Prometheus service with the following command:

sudo systemctl restart prometheus

Tuning remote_write

The above example included the Cyso recommended remote_write tuning. More information on this tuning may be found on the remote_write tuning page.

Verify Configuration

To confirm that the configuration is working correctly:

  • Check the Prometheus logs to verify that the metrics are being pushed successfully. The logs are typically found at /var/log/prometheus/prometheus.log, but the location can vary depending on your installation. If there are any errors, they will be listed here.

About remote write

Prometheus remote_write is a feature within Prometheus which allows the writing and sending of metrics to a remote storage system from the Prometheus server. This feature is part of Prometheus's storage configuration and is used for long-term storage of metrics data.

When Prometheus scrapes metrics data from your services, by default, it stores them locally. However, for large-scale environments or long-term data retention, this may not be feasible. This is where remote_write comes into play. It allows you to configure Prometheus to send the metrics to a remote storage system that can handle larger data volumes and longer retention periods.

remote_write sends each scraped sample to the configured remote endpoint immediately, keeping only the minimum necessary data in memory. It uses the remote storage as a more durable and extensive data store.

This way, you can maintain the power of Prometheus's alerting and querying capabilities while ensuring the safe and scalable long-term storage of your monitoring data.

Troubleshooting

If you encounter issues during setup, please check the following:

  • Verify that the Prometheus service is running.
  • Check the Prometheus logs for any errors.
  • Ensure that the provided credentials are correct. If they aren't, you won't be able to write metrics.

If you need any further help, don't hesitate to get in touch with us.