Skip to main content

Ingesting Logs To CtrlB via Vector

This document provides comprehensive instructions for configuring Vector to forward log data to CtrlB instances. Vector will be configured to transmit simple JSON log format to your designated instance endpoint.

Connection Parameters

The following parameters are required for establishing connectivity between Vector and your CtrlB instance:

  • Endpoint URL: http(s)://<INSTANCE_HOST>/<STREAM_NAME>/_json
  • Authentication Method: HTTP Basic Authentication via Authorization header
  • Authentication Format: Authorization: Basic <API_TOKEN>
  • Timestamp Specification: Unix epoch time in milliseconds within the _timestamp field

Implementation Configurations

File-Based Log Ingestion

This configuration applies to applications that write log files in either JSON or plain text format to the local filesystem.

sources:
file_logs:
type: file
include:
- /var/log/app/*.log
read_from: beginning

transforms:
add_timestamp:
type: remap
inputs:
- file_logs
source: |
._timestamp = to_unix_timestamp(now(), unit: "milliseconds")

sinks:
ctrlb_logs:
type: http
inputs:
- add_timestamp
encoding:
codec: json
uri: "https://<INSTANCE_HOST>/api/default/<STREAM_NAME>/_json"
method: post
headers:
authorization: "Basic <API_TOKEN>"
content-type: "application/json"

Container Log Processing

This configuration addresses containerized applications using Docker or CRI runtimes that output JSON-formatted log entries to standard output streams.

sources:
docker_logs:
type: docker_logs
include_containers:
- "*"

transforms:
parse_json:
type: remap
inputs:
- docker_logs
source: |
. = parse_json!(string!(.message))
._timestamp = to_unix_timestamp(now(), unit: "milliseconds")

sinks:
ctrlb_logs:
type: http
inputs:
- parse_json
encoding:
codec: json
uri: "https://<INSTANCE_HOST>/api/default/<STREAM_NAME>/_json"
method: post
headers:
authorization: "Basic <API_TOKEN>"
content-type: "application/json"

Validation Procedures

Service Initialization

Execute the following command to start the Vector service:

vector --config vector.yaml

Test Data Generation

Generate a test log entry to verify configuration:

echo '{"level":"info","job":"my-service","log":"hello from vector","_timestamp": 1756684800000}' >> /var/log/app/test.log

Verification Process

Access your CtrlB instance dashboard to confirm successful data ingestion. Verify that log entries appear in the designated stream with appropriate timestamps and field mappings.

Troubleshooting Guide

Authentication Errors

HTTP 401 Unauthorized

  • Root Cause: Invalid or malformed authentication credentials
  • Resolution: Verify the Authorization header follows the exact format: Authorization: Basic <API_TOKEN>
  • Validation: Confirm the API token is correctly encoded and has not been truncated

Endpoint Configuration Errors

HTTP 404 Not Found

  • Root Cause: Incorrect endpoint URI specification
  • Resolution: Ensure the URI parameter matches the required format: /<STREAM_NAME>/_json
  • Validation: Verify stream name matches your CtrlB instance configuration

Transport Layer Security Issues

TLS Connection Failures

  • Root Cause: Certificate validation or endpoint configuration errors
  • Resolution Steps:
    1. Verify TLS configuration in the sink settings
    2. For development environments, consider adding tls.verify_certificate: false
    3. Ensure certificate chains are properly configured for production environments
    4. Validate that the instance hostname matches certificate subject names

Timestamp Processing Issues

Missing or Invalid Timestamp Fields

  • Root Cause: Timestamp field configuration mismatch
  • Resolution Options:
    1. Use Vector's built-in timestamp functions: to_unix_timestamp(now(), unit: "milliseconds")
    2. Application-level timestamps: Ensure application generates numeric epoch millisecond values
    3. Transform existing timestamps using VRL (Vector Remap Language)

Performance Considerations

For high-volume log processing environments:

  • Configure appropriate buffer sizes using buffer settings in sinks
  • Implement disk-based buffers for persistence during outages
  • Monitor Vector metrics using the built-in monitoring features
  • Configure log rotation policies to prevent disk space exhaustion