Skip to main content

Ingesting Metrics To CtrlB via Prometheus Remote Write

This document provides comprehensive instructions for configuring Prometheus to forward metrics data to CtrlB instances. Prometheus Remote Write enables direct streaming of time-series metrics to external storage systems such as CtrlB.

Protocol Specifications​

The CtrlB platform supports Prometheus Remote Write ingestion with the following specifications:

  • Ingestion Protocol: Prometheus Remote Write
  • Encoding Format: Snappy-compressed Protobuf
  • Endpoint URL: http(s)://<INSTANCE_HOST>/engine/api/default/_prom/v1/write
  • Authentication Method: HTTP Basic Authentication via Authorization header
  • Authentication Format: Authorization: Basic <API_TOKEN>
  • Compression: Snappy (required)

Basic Configuration​

Update your Prometheus configuration file (prometheus.yml) with the following:

remote_write:
  - url: http://<INSTANCE_HOST>/engine/api/default/_prom/v1/write
    headers:
      Authorization: Basic <API_TOKEN>

Implementation Notes​

Prometheus automatically:

  • Compresses data using Snappy
  • Encodes payload using Protobuf WriteRequest

CtrlB ingestion pipeline:

  • Decompresses payload
  • Decodes Protobuf message
  • Processes and stores metrics internally

Deployment Considerations​

  • Ensure Prometheus version supports remote_write
  • Verify network access to CtrlB endpoint
  • Configure appropriate scrape intervals to manage ingestion load

Validation Procedures​

Service Initialization​

Restart Prometheus after configuration changes:

systemctl restart prometheus

Verification Process​

  • Access your CtrlB dashboard
  • Filter metrics within the last 5 minutes
  • Validate:
    • Metric names
    • Label sets
    • Timestamp correctness

Troubleshooting Guide​

Authentication Errors​

HTTP 401 Unauthorized

  • Verify Authorization header format
  • Ensure API token is valid and properly encoded

Endpoint Errors​

HTTP 404 Not Found

Confirm endpoint path:

/engine/api/default/_prom/v1/write

Compression Errors​

Snappy Decompression Failed

  • Ensure Prometheus remote_write is enabled correctly
  • Verify no proxy is modifying payload

Protobuf Decode Errors​

  • Indicates malformed payload
  • Check Prometheus version compatibility

Performance Issues​

  • Adjust scrape intervals
  • Reduce high-cardinality metrics
  • Scale ingestion infrastructure if needed