Skip to main content

Ingesting Logs To CtrlB via Vector

This document provides comprehensive instructions for configuring Vector to forward log data to CtrlB instances. Vector will be configured to transmit simple JSON log format to your designated instance endpoint.

Required Configuration Parameters​

  • Instance Host: <INSTANCE_HOST>
  • API Authentication Token: <API_TOKEN>
  • Stream Identifier: <STREAM_NAME>

Connection Parameters​

The following parameters are required for establishing connectivity between Vector and your CtrlB instance:

  • Endpoint URL: https://<INSTANCE_HOST>/engine/api/default/<STREAM_NAME>/v2/_json_evolving
  • Authentication Method: HTTP Basic Authentication via Authorization header
  • Authentication Format: Authorization: Basic <API_TOKEN>
  • Timestamp Specification: Unix epoch time in milliseconds within the _timestamp field

Implementation Configurations​

File-Based Log Ingestion​

This configuration applies to applications that write log files in either JSON or plain text format to the local filesystem.

sources:
  file_logs:
    type: file
    include:
      - /var/log/app/*.log
    read_from: beginning

transforms:
  add_timestamp:
    type: remap
    inputs:
      - file_logs
    source: |
      ._timestamp = to_unix_timestamp(now(), unit: "milliseconds")

sinks:
  ctrlb_logs:
    type: http
    inputs:
      - add_timestamp
    encoding:
      codec: json
    uri: "https://<INSTANCE_HOST>/engine/api/default/<STREAM_NAME>/v2/_json_evolving"
    method: post
    headers:
      authorization: "Basic <API_TOKEN>"
      content-type: "application/json"

Container Log Processing​

This configuration addresses containerized applications using Docker or CRI runtimes that output JSON-formatted log entries to standard output streams.

sources:
  docker_logs:
    type: docker_logs
    include_containers:
      - "*"

transforms:
  parse_json:
    type: remap
    inputs:
      - docker_logs
    source: |
      . = parse_json!(string!(.message))
      ._timestamp = to_unix_timestamp(now(), unit: "microseconds")

sinks:
  ctrlb_logs:
    type: http
    inputs:
      - parse_json
    encoding:
      codec: json
    uri: "https://<INSTANCE_HOST>/engine/api/default/<STREAM_NAME>/v2/_json_evolving"
    method: post
    headers:
      authorization: "Basic <API_TOKEN>"
      content-type: "application/json"

Validation Procedures​

Service Initialization​

Execute the following command to start the Vector service:

vector --config vector.yaml

Test Data Generation​

Generate a test log entry to verify configuration:

echo '{"level":"info","job":"my-service","log":"hello from vector","_timestamp": 1756684800000}' >> /var/log/app/test.log

Verification Process​

Access your CtrlB instance dashboard to confirm successful data ingestion. Verify that log entries appear in the designated stream with appropriate timestamps and field mappings.

Troubleshooting Guide​

Authentication Errors​

HTTP 401 Unauthorized

  • Root Cause: Invalid or malformed authentication credentials
  • Resolution: Verify the Authorization header follows the exact format: Authorization: Basic <API_TOKEN>
  • Validation: Confirm the API token is correctly encoded and has not been truncated

Endpoint Configuration Errors​

HTTP 404 Not Found

  • Root Cause: Incorrect endpoint URI specification
  • Resolution: Ensure the URI parameter matches the required format: /<STREAM_NAME>/v2/_json_evolving
  • Validation: Verify stream name matches your CtrlB instance configuration

Transport Layer Security Issues​

TLS Connection Failures

  • Root Cause: Certificate validation or endpoint configuration errors
  • Resolution Steps:
    1. Verify TLS configuration in the sink settings
    2. For development environments, consider adding tls.verify_certificate: false
    3. Ensure certificate chains are properly configured for production environments
    4. Validate that the instance hostname matches certificate subject names

Timestamp Processing Issues​

Missing or Invalid Timestamp Fields

  • Root Cause: Timestamp field configuration mismatch
  • Resolution Options:
    1. Use Vector's built-in timestamp functions: to_unix_timestamp(now(), unit: "milliseconds")
    2. Application-level timestamps: Ensure application generates numeric epoch millisecond values
    3. Transform existing timestamps using VRL (Vector Remap Language)

Performance Considerations​

For high-volume log processing environments:

  • Configure appropriate buffer sizes using buffer settings in sinks
  • Implement disk-based buffers for persistence during outages
  • Monitor Vector metrics using the built-in monitoring features
  • Configure log rotation policies to prevent disk space exhaustion