Vector

Vector is an open-source, high-performance log and event router designed for modern observability and data processing pipelines. It serves as a versatile data transport and transformation tool, capable of ingesting, transforming, and routing data from various sources to multiple destinations. Vector is known for its flexibility, extensibility, and support for a wide range of data formats and outputs. It is commonly used in cloud-native and microservices architectures to collect and route logs, metrics, and other events, making it a valuable component in modern observability and data processing workflows.

Specifically: Vector supports shipping Prometheus metrics and Loki logs, and can be a better fit in some use-cases than Grafana Agent or Prometheus.

Official Vector Documentation

Vector has a detailed and extensive official documentation which includes comprehensive guides on installation, configuration, and usage. You can refer to it for better understanding and effective usage of the tool.

Official Vector Documentation

This documentation focusses on how to use Vector with our observability platform.

Configuration

Vector has native support for our Loki endpoint, so we can use it to forward logs to Observe by Cyso. Vector also supports scraping Prometheus metrics, and generates metrics about its own operation. In this example we configure both: watching logs on the local filesystem, and forwarding metrics about Vector.

Prerequisites

Before you start, make sure you have the following prerequisites in place:

1. Vector Installed: Ensure that you have Vector installed on the system where you want to collect and forward logs. You can download and install Vector from the official website.

2. Your Cyso customer ID and API key are available.

Configuration steps

1. Create a configuration file for Vector. The default location is /etc/vector and depending on the installation method the file may be named vector.yml, vector.yaml or vector.toml. The examples below assume the file format is YAML.

   sources:
     internal_metrics:
       type: internal_metrics
     system_logs:
       include:
       - /var/lib/**/*.log
       type: file
   
   transforms: {}
   
   sinks:
     observe_logs:
       endpoint: https://api.ams2.observe.cyso.com/{customer_id}}/logs
       auth:
         user: '{customer_id}'
         password: {token}
         strategy: basic
       encoding:
         codec: json
       inputs:
       - '*_logs'
       labels:
         source: my.hostname.fqdn
       type: loki
     observe_metrics:
       endpoint: https://api.ams2.observe.cyso.com/{customer_id}/metrics/api/v1/push
       auth:
         user: '{customer_id}'
         password: {token}
         strategy: basic
       inputs:
       - '*_metrics'
       type: prometheus_remote_write
   

   Note

   You should replace {customer_id} with your actual Cyso Customer ID and {token} with your received API key.

   Warning

   This example configuration will watch all files ending with .log in the /var/log folder on your system, and send them to Loki. Make sure you verify that is exactly what you want. There may be sensitive log files in this location on your system.

2. Save the file and close it.

3. Verify your configuration with the following command: vector validate --config-yaml /etc/vector/vector.yaml. Verify that no errors were found.

4. Restart Vector to load the new configuration.

Troubleshooting

If you encounter issues during setup:

• Confirm the Vector service is running.
• Review the Vector logs for any errors.
• Check the supplied credentials for accuracy. Incorrect credentials will inhibit log writing.

In case you need additional assistance, contact Cyso's support team.