Linux - using Fluentbit

This document provides comprehensive instructions for installing and configuring FluentBit on Linux systems to forward log data to CtrlB instances. FluentBit will be configured to transmit simple JSON log format to your designated instance endpoint.

Installation Steps for Linux

Prerequisites

Ensure your system meets the following requirements:

Root or sudo access
Linux distribution (Ubuntu, Debian, CentOS, RHEL, or equivalent)
Internet connectivity for package downloads
Minimum 256MB RAM available

Step 1: Update System Packages

Begin by updating your system package manager:

sudo apt update && sudo apt upgrade -y    # For Debian/Ubuntu
# OR
sudo yum update -y                         # For CentOS/RHEL

Step 2: Add FluentBit Repository

For Ubuntu/Debian systems, add the official FluentBit repository:

curl https://raw.githubusercontent.com/fluent/fluent-package-repository/master/fluent-package.key | gpg --dearmor > /usr/share/keyrings/fluentbit-keyring.gpg
echo 'deb [signed-by=/usr/share/keyrings/fluentbit-keyring.gpg] https://packages.treasuredata.com/debian/$(lsb_release -sc)/ $(lsb_release -sc) contrib' | sudo tee /etc/apt/sources.list.d/fluent-bit.list
sudo apt update

For CentOS/RHEL systems:

cat > /etc/yum.repos.d/fluent-bit.repo << EOF
[fluent-bit]
name = Fluent Bit
baseurl = https://packages.treasuredata.com/redhat/\$releasever/\$basearch/
gpgcheck = 1
gpgkey = https://packages.treasuredata.com/redhat/fluent-bit.key
enabled = 1
EOF

Step 3: Install FluentBit

For Ubuntu/Debian:

sudo apt install fluent-bit -y

For CentOS/RHEL:

sudo yum install fluent-bit -y

Step 4: Verify Installation

Verify that FluentBit has been installed successfully:

fluent-bit --version

Step 5: Enable FluentBit Service

Enable FluentBit to start automatically on system boot:

sudo systemctl enable fluent-bit

Connection Parameters

The following parameters are required for establishing connectivity between FluentBit and your CtrlB instance:

Endpoint URL: http(s)://<INSTANCE_HOST>/api/default/<STREAM_NAME>/_json_evolving
Authentication Method: HTTP Basic Authentication via Authorization header
Authentication Format: Authorization: Basic <API_TOKEN>
Timestamp Specification: Unix epoch time in milliseconds within the _timestamp field

Implementation Configuration

/var/log Files Ingestion

This configuration applies to applications that write log files in either JSON or plain text format to the /var/log directory on Linux systems.

Step 1: Create Configuration Directory

sudo mkdir -p /etc/fluent-bit/

Step 2: Create Configuration File

Create the main FluentBit configuration file:

sudo nano /etc/fluent-bit/fluent-bit.conf

Step 3: Add Configuration Content

Paste the following configuration into the file:

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info

[INPUT]
    Name         tail
    Path         /var/log/*.log
    Tag          syslog.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[OUTPUT]
    Name         http
    Match        syslog.*
    Host         <INSTANCE_HOST>
    Port         443
    URI          /api/default/<STREAM_NAME>/_json_evolving
    Format       json
    Json_Date_Key _timestamp
    Json_Date_Format epoch_ms
    Header       Authorization Basic <API_TOKEN>
    Header       Content-Type application/json
    tls          Off
    tls.verify   Off

Configuration Notes:

The configuration monitors all .log files in the /var/log directory
Replace <INSTANCE_HOST> with your CtrlB instance hostname
Replace <STREAM_NAME> with your target stream name
Replace <API_TOKEN> with your authentication token

Multiple /var/log Directory Paths

If you need to monitor multiple specific directories within /var/log, you can add multiple INPUT sections:

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info

[INPUT]
    Name         tail
    Path         /var/log/app/*.log
    Tag          app.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[INPUT]
    Name         tail
    Path         /var/log/system/*.log
    Tag          system.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[INPUT]
    Name         tail
    Path         /var/log/nginx/*.log
    Tag          nginx.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[OUTPUT]
    Name         http
    Match        *.*
    Host         <INSTANCE_HOST>
    Port         443
    URI          /api/default/<STREAM_NAME>/_json_evolving
    Format       json
    Json_Date_Key _timestamp
    Json_Date_Format epoch_ms
    Header       Authorization Basic <API_TOKEN>
    Header       Content-Type application/json
    tls          Off
    tls.verify   Off

This approach allows you to monitor different application and system log directories with a single FluentBit instance.

Starting FluentBit Service

Using systemctl

Start the FluentBit service:

sudo systemctl start fluent-bit

Check the service status:

sudo systemctl status fluent-bit

Manual Execution (For Testing)

For testing purposes, run FluentBit manually with your configuration:

sudo fluent-bit -c /etc/fluent-bit/fluent-bit.conf

Validation Procedures

Step 1: Verify Service Status

Confirm that FluentBit is running successfully:

sudo systemctl status fluent-bit

Step 2: Check Service Logs

Review FluentBit logs for any errors:

sudo journalctl -u fluent-bit -f

Or check directly:

tail -f /var/log/fluent-bit.log

Step 3: Generate Test Data

First, create the log directory if it doesn't exist:

sudo mkdir -p /var/log/app
sudo chmod 755 /var/log/app

Generate a test log entry to verify configuration:

echo '{"level":"info","job":"my-service","log":"hello from fluent-bit","_timestamp": 1756684800000}' | sudo tee -a /var/log/app/test.log

Step 4: Verification

Access your CtrlB instance dashboard to confirm successful data ingestion. Verify that log entries appear in the designated stream with appropriate timestamps and field mappings.

Troubleshooting Guide

Authentication Errors

HTTP 401 Unauthorized

Root Cause: Invalid or malformed authentication credentials
Resolution: Verify the Authorization header follows the exact format: Authorization: Basic <API_TOKEN>
Validation: Confirm the API token is correctly encoded and has not been truncated

Endpoint Configuration Errors

HTTP 404 Not Found

Root Cause: Incorrect endpoint URI specification
Resolution: Ensure the URI parameter matches the required format: /<STREAM_NAME>/_json_evolving
Validation: Verify stream name matches your CtrlB instance configuration

Transport Layer Security Issues

TLS Connection Failures

Root Cause: Certificate validation or port configuration errors
Resolution Steps:
1. Verify port configuration (443 for HTTPS, 80 for HTTP)
2. For development environments with self-signed certificates, set tls.verify Off
3. Ensure certificate chains are properly configured for production environments
4. Validate that the instance hostname matches certificate subject names

Timestamp Processing Issues

Missing or Invalid Timestamp Fields

Root Cause: Timestamp field configuration mismatch
Resolution Options:
1. Configure FluentBit to generate timestamps: Add Json_Date_Key _timestamp and Json_Date_Format epoch_ms to output configuration
2. Application-level timestamps: Ensure application generates numeric epoch millisecond values in the designated timestamp field
3. Field mapping: Verify Time_Key parameter matches the actual timestamp field name in your log format

Permission Denied Errors

Root Cause: Insufficient permissions for log file access
Resolution: Ensure the fluent-bit user has read permissions on log directories
```
sudo usermod -a -G adm fluent-bit
sudo systemctl restart fluent-bit
```

Performance Considerations

For high-volume log processing environments:

Adjust Mem_Buf_Limit based on available system memory
Configure appropriate Flush intervals to balance latency and throughput
Monitor FluentBit resource utilization and adjust buffer sizes accordingly
Implement log rotation policies to prevent disk space exhaustion
Use the following command to monitor resource usage:
```
ps aux | grep fluent-bit
```

Installation Steps for Linux​

Prerequisites​

Step 1: Update System Packages​

Step 2: Add FluentBit Repository​

Step 3: Install FluentBit​

Step 4: Verify Installation​

Step 5: Enable FluentBit Service​

Connection Parameters​

Implementation Configuration​

/var/log Files Ingestion​

Multiple /var/log Directory Paths​

Starting FluentBit Service​

Using systemctl​

Manual Execution (For Testing)​

Validation Procedures​

Step 1: Verify Service Status​

Step 2: Check Service Logs​

Step 3: Generate Test Data​

Step 4: Verification​

Troubleshooting Guide​

Authentication Errors​

Endpoint Configuration Errors​

Transport Layer Security Issues​

Timestamp Processing Issues​

Permission Denied Errors​

Performance Considerations​