Ingesting Logs To CtrlB via Vector
This document provides comprehensive instructions for configuring Vector to forward log data to CtrlB instances. Vector will be configured to transmit simple JSON log format to your designated instance endpoint.
Connection Parameters
The following parameters are required for establishing connectivity between Vector and your CtrlB instance:
- Endpoint URL:
http(s)://<INSTANCE_HOST>/<STREAM_NAME>/_json
- Authentication Method: HTTP Basic Authentication via Authorization header
- Authentication Format:
Authorization: Basic <API_TOKEN>
- Timestamp Specification: Unix epoch time in milliseconds within the
_timestamp
field
Implementation Configurations
File-Based Log Ingestion
This configuration applies to applications that write log files in either JSON or plain text format to the local filesystem.
sources:
file_logs:
type: file
include:
- /var/log/app/*.log
read_from: beginning
transforms:
add_timestamp:
type: remap
inputs:
- file_logs
source: |
._timestamp = to_unix_timestamp(now(), unit: "milliseconds")
sinks:
ctrlb_logs:
type: http
inputs:
- add_timestamp
encoding:
codec: json
uri: "https://<INSTANCE_HOST>/api/default/<STREAM_NAME>/_json"
method: post
headers:
authorization: "Basic <API_TOKEN>"
content-type: "application/json"
Container Log Processing
This configuration addresses containerized applications using Docker or CRI runtimes that output JSON-formatted log entries to standard output streams.
sources:
docker_logs:
type: docker_logs
include_containers:
- "*"
transforms:
parse_json:
type: remap
inputs:
- docker_logs
source: |
. = parse_json!(string!(.message))
._timestamp = to_unix_timestamp(now(), unit: "milliseconds")
sinks:
ctrlb_logs:
type: http
inputs:
- parse_json
encoding:
codec: json
uri: "https://<INSTANCE_HOST>/api/default/<STREAM_NAME>/_json"
method: post
headers:
authorization: "Basic <API_TOKEN>"
content-type: "application/json"
Validation Procedures
Service Initialization
Execute the following command to start the Vector service:
vector --config vector.yaml
Test Data Generation
Generate a test log entry to verify configuration:
echo '{"level":"info","job":"my-service","log":"hello from vector","_timestamp": 1756684800000}' >> /var/log/app/test.log
Verification Process
Access your CtrlB instance dashboard to confirm successful data ingestion. Verify that log entries appear in the designated stream with appropriate timestamps and field mappings.
Troubleshooting Guide
Authentication Errors
HTTP 401 Unauthorized
- Root Cause: Invalid or malformed authentication credentials
- Resolution: Verify the Authorization header follows the exact format:
Authorization: Basic <API_TOKEN>
- Validation: Confirm the API token is correctly encoded and has not been truncated
Endpoint Configuration Errors
HTTP 404 Not Found
- Root Cause: Incorrect endpoint URI specification
- Resolution: Ensure the URI parameter matches the required format:
/<STREAM_NAME>/_json
- Validation: Verify stream name matches your CtrlB instance configuration
Transport Layer Security Issues
TLS Connection Failures
- Root Cause: Certificate validation or endpoint configuration errors
- Resolution Steps:
- Verify TLS configuration in the sink settings
- For development environments, consider adding
tls.verify_certificate: false
- Ensure certificate chains are properly configured for production environments
- Validate that the instance hostname matches certificate subject names
Timestamp Processing Issues
Missing or Invalid Timestamp Fields
- Root Cause: Timestamp field configuration mismatch
- Resolution Options:
- Use Vector's built-in timestamp functions:
to_unix_timestamp(now(), unit: "milliseconds")
- Application-level timestamps: Ensure application generates numeric epoch millisecond values
- Transform existing timestamps using VRL (Vector Remap Language)
- Use Vector's built-in timestamp functions:
Performance Considerations
For high-volume log processing environments:
- Configure appropriate buffer sizes using
buffer
settings in sinks - Implement disk-based buffers for persistence during outages
- Monitor Vector metrics using the built-in monitoring features
- Configure log rotation policies to prevent disk space exhaustion