Ingesting Logs and Traces to CtrlB on Kubernetes

This document provides comprehensive instructions for configuring logging and tracing on Kubernetes clusters using pre-built configurations from the ctrlb-k8s repository.

Overview

Choose one of the following deployment options based on your requirements:

Option 1: Pure Logs (FluentBit)

Lightweight DaemonSet collecting container logs only
Best for log-only observability needs

Option 2: Logs + Traces (OpenTelemetry Collector)

Comprehensive observability with both logs and distributed traces
Best for full-stack application monitoring

Prerequisites

Kubernetes cluster with kubectl access
Sufficient permissions to create namespaces and deploy resources
git installed to clone the repository
For Option 2: Applications instrumented with OpenTelemetry SDKs

Quick Start

Clone the Repository

git clone https://github.com/ctrlb-hq/ctrlb-k8s.git
cd ctrlb-k8s

Prepare Configuration

Before deploying, you need to replace the following placeholders in the ConfigMap files:

<INSTANCE_HOST>: Your CtrlB instance hostname
<STREAM_NAME>: Target stream name in CtrlB
<API_TOKEN>: Your CtrlB API authentication token

Use a text editor or sed to replace these values:

# For FluentBit
sed -i 's/<INSTANCE_HOST>/your-instance-host/g' fluent-bit/02-configmap.yaml
sed -i 's/<STREAM_NAME>/your-stream-name/g' fluent-bit/02-configmap.yaml
sed -i 's/<API_TOKEN>/your-api-token/g' fluent-bit/02-configmap.yaml

# For OpenTelemetry Collector
sed -i 's/<INSTANCE_HOST>/your-instance-host/g' otel/02-configmap.yaml
sed -i 's/<STREAM_NAME>/your-stream-name/g' otel/02-configmap.yaml
sed -i 's/<API_TOKEN>/your-api-token/g' otel/02-configmap.yaml

Deployment

Option 1: Deploy FluentBit (Logs Only)

Deploy FluentBit as a DaemonSet to collect container logs from all nodes in your cluster.

Step 1: Review Configuration

cat fluent-bit/02-configmap.yaml

This ConfigMap defines how FluentBit collects and forwards logs to CtrlB. The configuration includes:

Log collection from /var/log/containers/*.log
JSON formatting with timestamp

Step 2: Apply Configurations

Apply all FluentBit configurations in sequence:

kubectl apply -f fluent-bit/01-namespace.yaml
kubectl apply -f fluent-bit/02-configmap.yaml
kubectl apply -f fluent-bit/03-serviceaccount.yaml
kubectl apply -f fluent-bit/04-clusterrole.yaml
kubectl apply -f fluent-bit/05-clusterrolebinding.yaml
kubectl apply -f fluent-bit/06-daemonset.yaml

Or apply all at once:

kubectl apply -f fluent-bit/

Step 3: Verify Deployment

Check if FluentBit pods are running on all nodes:

kubectl get pods -n ctrlb-logging -l app=fluent-bit

View FluentBit logs:

kubectl logs -n ctrlb-logging -l app=fluent-bit --tail=50

Check pod resource usage:

kubectl top pods -n ctrlb-logging

Option 2: Deploy OpenTelemetry Collector (Logs + Traces)

Deploy OpenTelemetry Collector as a DaemonSet to collect both logs and distributed traces.

Step 1: Review Configuration

cat otel/02-configmap.yaml

This ConfigMap defines OpenTelemetry Collector pipelines for:

Traces pipeline: Receives OTLP gRPC/HTTP traces and exports them to CtrlB
Logs pipeline: Receives OTLP logs and filelog sources, exports to CtrlB
Processors: Batch processing, memory limiting, and resource detection

Step 2: Apply Configurations

Apply all OpenTelemetry configurations in sequence:

kubectl apply -f otel/01-namespace.yaml
kubectl apply -f otel/02-configmap.yaml
kubectl apply -f otel/03-serviceaccount.yaml
kubectl apply -f otel/04-clusterrole.yaml
kubectl apply -f otel/05-clusterrolebinding.yaml
kubectl apply -f otel/06-daemonset.yaml
kubectl apply -f otel/07-service.yaml

Or apply all at once:

kubectl apply -f otel/

Step 3: Verify Deployment

Check if collector pods are running:

kubectl get pods -n otel -l app=otel-collector

View collector logs:

kubectl logs -n otel -l app=otel-collector --tail=50

Check pod resource usage:

kubectl top pods -n otel

Step 4: Configure Applications to Send Traces

Update your application deployments to send traces to the OpenTelemetry Collector.

Set the following environment variables:

For gRPC protocol:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.otel.svc.cluster.local:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

For HTTP protocol:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.otel.svc.cluster.local:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Example in a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: app
          image: my-app:latest
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: http://otel-collector.otel.svc.cluster.local:4317
            - name: OTEL_EXPORTER_OTLP_PROTOCOL
              value: grpc

Repository Structure

The configurations are organized as follows:

ctrlb-k8s/
├── fluent-bit/              # FluentBit DaemonSet configuration
│   ├── 01-namespace.yaml
│   ├── 02-configmap.yaml
│   ├── 03-serviceaccount.yaml
│   ├── 04-clusterrole.yaml
│   ├── 05-clusterrolebinding.yaml
│   └── 06-daemonset.yaml
├── otel/                    # OpenTelemetry Collector configuration
│   ├── 01-namespace.yaml
│   ├── 02-configmap.yaml
│   ├── 03-serviceaccount.yaml
│   ├── 04-clusterrole.yaml
│   ├── 05-clusterrolebinding.yaml
│   ├── 06-daemonset.yaml
│   └── 07-service.yaml
└── README.md

Files are numbered to indicate the recommended application order. However, you can apply all files in a directory at once using kubectl apply -f directory/.

Configuration Parameters Reference

FluentBit Configuration

Located in fluent-bit/02-configmap.yaml:

Parameter	Description
`<INSTANCE_HOST>`	CtrlB instance hostname
`<STREAM_NAME>`	Target stream name in CtrlB
`<API_TOKEN>`	Authentication token (Base64 encoded)
`Flush`	Flush interval in seconds
`Mem_Buf_Limit`	Memory buffer limit per input

OpenTelemetry Collector Configuration

Located in otel/02-configmap.yaml:

Parameter	Description
`<INSTANCE_HOST>`	CtrlB instance hostname
`<STREAM_NAME>`	Target stream name in CtrlB
`<API_TOKEN>`	Authentication token (Base64 encoded)
`4317`	OTLP gRPC receiver port
`4318`	OTLP HTTP receiver port
`send_batch_size`	Number of spans/logs to batch before sending
`timeout`	Maximum time to wait before sending batch

Verification and Troubleshooting

Check Collector/DaemonSet Status

# FluentBit
kubectl get daemonset -n ctrlb-logging
kubectl describe daemonset fluent-bit -n ctrlb-logging

# OpenTelemetry Collector
kubectl get daemonset -n otel
kubectl describe daemonset otel-collector -n otel

View Logs

# FluentBit logs
kubectl logs -n ctrlb-logging -l app=fluent-bit --all-containers=true -f

# OpenTelemetry Collector logs (last 100 lines, follow)
kubectl logs -n otel -l app=otel-collector --all-containers=true -f --tail=100

Check Events

# FluentBit namespace events
kubectl get events -n ctrlb-logging --sort-by='.lastTimestamp'

# OpenTelemetry namespace events
kubectl get events -n otel --sort-by='.lastTimestamp'

Test Connectivity

Create a debug pod and test connectivity to CtrlB:

kubectl run debug --image=curlimages/curl -it --restart=Never -- sh

Inside the debug pod:

curl -H "Authorization: Basic <API_TOKEN>" \
  https://<INSTANCE_HOST>/api/default/<STREAM_NAME>/_json

Monitor Resource Usage

# FluentBit resource usage
kubectl top pods -n ctrlb-logging

# OpenTelemetry Collector resource usage
kubectl top pods -n otel

Common Issues

Pods stuck in Pending state:

kubectl describe pod <pod-name> -n <namespace>

High memory usage: Check Mem_Buf_Limit in FluentBit config or limit_mib in OTEL config.

Authentication failures: Verify <API_TOKEN> is correctly Base64 encoded and has proper permissions.

Connectivity errors: Confirm <INSTANCE_HOST> is reachable from the cluster and TLS certificates are valid.

Advanced Configuration

Customize Log Collection

Edit fluent-bit/02-configmap.yaml to modify:

Log paths to monitor
Parsing rules
Filter conditions
Output destinations

Customize Trace Collection

Edit otel/02-configmap.yaml to modify:

Receiver protocols and endpoints
Processor batch sizes and timeouts
Resource detection
Exporter endpoints

Scale Resource Limits

Adjust resource requests and limits in the DaemonSet YAML files:

resources:
  requests:
    memory: 100Mi
    cpu: 100m
  limits:
    memory: 500Mi
    cpu: 500m

Support and Contributing

For issues, feature requests, or contributions, visit the ctrlb-k8s repository.

For general CtrlB support, contact: support@ctrlb.io

Connection Parameters

Endpoint URL: https://<INSTANCE_HOST>/api/default/<STREAM_NAME>/_json
Authentication Method: HTTP Basic Authentication
Authentication Format: Authorization: Basic <API_TOKEN>
Timestamp Specification: Unix epoch time in milliseconds
OTLP gRPC Endpoint: <INSTANCE_HOST>:4317
OTLP HTTP Endpoint: https://<INSTANCE_HOST>/api/default/<STREAM_NAME>/_json

Overview​

Prerequisites​

Quick Start​

Clone the Repository​

Prepare Configuration​

Deployment​

Option 1: Deploy FluentBit (Logs Only)​

Step 1: Review Configuration​

Step 2: Apply Configurations​

Step 3: Verify Deployment​

Option 2: Deploy OpenTelemetry Collector (Logs + Traces)​

Step 1: Review Configuration​

Step 2: Apply Configurations​

Step 3: Verify Deployment​

Step 4: Configure Applications to Send Traces​

Repository Structure​

Configuration Parameters Reference​

FluentBit Configuration​

OpenTelemetry Collector Configuration​

Verification and Troubleshooting​

Check Collector/DaemonSet Status​

View Logs​

Check Events​

Test Connectivity​

Monitor Resource Usage​

Common Issues​

Advanced Configuration​

Customize Log Collection​

Customize Trace Collection​

Scale Resource Limits​

Support and Contributing​

Connection Parameters​

Overview

Prerequisites

Quick Start

Clone the Repository

Prepare Configuration

Deployment

Option 1: Deploy FluentBit (Logs Only)

Step 1: Review Configuration

Step 2: Apply Configurations

Step 3: Verify Deployment

Option 2: Deploy OpenTelemetry Collector (Logs + Traces)

Step 1: Review Configuration

Step 2: Apply Configurations

Step 3: Verify Deployment

Step 4: Configure Applications to Send Traces

Repository Structure

Configuration Parameters Reference

FluentBit Configuration

OpenTelemetry Collector Configuration

Verification and Troubleshooting

Check Collector/DaemonSet Status

View Logs

Check Events

Test Connectivity

Monitor Resource Usage

Common Issues

Advanced Configuration

Customize Log Collection

Customize Trace Collection

Scale Resource Limits

Support and Contributing

Connection Parameters