Skip to content

Grafana Agent

Grafana agent

Grafana Agent is an open-source, telemetry collection software released by Grafana Labs. It's designed with performance and observability in mind, and it works hand-in-hand with various Grafana-based offerings, including Observe by Cyso. While the agent stands as a foundation for gathering monitoring data, it should not be confused with full standalone monitoring software. Its primary task is to collect metrics, logs, and traces, then transmit these to an analytics or visualization tool.

The agent is a lightweight, highly-efficient, and simpler fragment of Prometheus that incorporates the best features of both Prometheus and Grafana Labs systems. This includes Prometheus's service discovery, metadata, and relabeling. By architecting the agent this way, it significantly reduces the memory footprint and resource requirements.

Official Grafana Agent Documentation

Grafana Agent has a detailed and extensive official documentation which includes comprehensive guides on installation, configuration, and usage. You can refer to it for better understanding and effective usage of the tool.

Official Grafana Agent Documentation GitHub - grafana/agent

This documentation focusses on how to use Grafana Agent with our observability platform.

Configuration

To configure the Grafana Agent to forward metrics to the Observe by Cyso endpoint, you can leverage a feature inherent to Prometheus-based systems called remote_write. This configuration allows for immediate transmission of each scraped metric to a specified remote endpoint. The following guide details the configuration steps you'll need to follow within the Grafana Agent’s environment.

Prerequisites

To proceed with the configuration, ensure that:

  1. Grafana Agent is installed and operational.
  2. You have administrative access to your Grafana Agent server.
  3. Your Cyso customer ID and API key are available.

Configuration Steps

Configuration of remote_write in Grafana Agent is slightly different from Prometheus. Below is the representation of how it is done:

  1. Open the Grafana Agent's configuration file. The file location can vary based on your installation.

  2. Look for the metrics section in the file. If it doesn't exist, create it.

  3. Under the metrics section, add or find configs.

  4. In the configs array, each item is a 'scrape set'. Find or add remote_write to it. The configuration should match the following:

    metrics:
  global:
    scrape_interval: 1m
  configs:
    - name: observe_by_cyso_metrics
      remote_write:
        - url: https://api.ams2.observe.cyso.com/{customer_id}/metrics/api/v1/push
          basic_auth:
            username: '{customer_id}'
            password: '{token}'
          queue_config:
            capacity: 15000
            max_samples_per_send: 2500

    Note

    You should replace {customer_id} with your actual Cyso Customer ID and {token} with your received API key.

  5. Save the file and close it.

  6. To enact the changes you've made, restart the Grafana Agent service using the appropriate command for your operating system and installation method.

Further, you can validate that the configuration is operating appropriately by reviewing the Grafana Agent logs to ensure the metrics are being pushed successfully. Any errors arising will be detailed in these logs.

If there are issues during setup, ensure your Grafana Agent service is active, double-check your inputted credentials, and examine the Grafana Agent logs for any discernible errors.

Troubleshooting

If you encounter issues during setup:

  • Confirm the Grafana Agent service is running.
  • Review the Grafana Agent logs for any errors.
  • Check the supplied credentials for accuracy. Incorrect credentials will inhibit metric writing.

In case you need additional assistance, contact Cyso's support team.

Reducing cardinality

The process of dropping labels and metrics in the Grafana Agent configuration is not much different than that of Prometheus, albeit with a slightly different YAML configuration.

Here's an example for dropping a specific metric:

metrics:
  global:
    scrape_interval: 1m
  configs:
    - name: 'insert_name_here'
      remote_write:
        - url: 'https://api.ams2.observe.cyso.com/{customer_id}/metrics/api/v1/push'
          basic_auth:
            username: '{customer_id}'
            password: '{token}'
          queue_config:
            capacity: 15000
            max_samples_per_send: 2500
          write_relabel_configs:
          - source_labels: [__name__]
            regex: '(metric_to_be_dropped|another_metric)'
            action: drop

For more information on cardinality please refer to this page.