Skip to main content

Ingesting Logs To CtrlB using FireLens in ECS

This guide explains how to configure AWS FireLens to forward container logs from Amazon ECS to CtrlB. FireLens works with Fluent Bit or Fluentd to enable advanced log routing capabilities.

Prerequisites​

  • An Amazon ECS cluster
  • A task definition that you want to configure with FireLens
  • CtrlB instance endpoint and API token

Configuration Steps​

1. Create FireLens Configuration​

First, create a custom FireLens configuration file that will be used in your ECS task definition.

{
  "log_config": {
    "http": [
      {
        "name": "ctrlb",
        "host": "<INSTANCE_HOST>",
        "port": "443",
        "uri": "/api/default/<STREAM_NAME>/_json",
        "header": {
          "Authorization": "Basic <API_TOKEN>",
          "Content-Type": "application/json"
        },
        "tls": "on",
        "tls.verify": "on",
        "format": "json",
        "json_date_key": "_timestamp",
        "json_date_format": "epoch_ms"
      }
    ]
  }
}

2. Task Definition Configuration​

Update your ECS task definition to include FireLens configuration. Here's an example task definition:

{
  "family": "firelens-example-task",
  "taskRoleArn": "arn:aws:iam::123456789012:role/ecs-task-role",
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecs-task-execution-role",
  "containerDefinitions": [
    {
      "essential": true,
      "image": "906394416424.dkr.ecr.us-east-1.amazonaws.com/aws-for-fluent-bit:latest",
      "name": "log_router",
      "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
          "config-file-type": "file",
          "config-file-value": "/fluent-bit/etc/extra.conf"
        }
      },
      "memoryReservation": 50
    },
    {
      "essential": true,
      "image": "your-application-image:latest",
      "name": "app",
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "http",
          "Host": "<INSTANCE_HOST>",
          "Port": "443",
          "URI": "/api/default/<STREAM_NAME>/_json",
          "Header_Authorization": "Basic <API_TOKEN>",
          "Header_Content-Type": "application/json",
          "tls": "on",
          "tls.verify": "on",
          "Format": "json",
          "Json_Date_Key": "_timestamp",
          "Json_Date_Format": "epoch_ms"
        }
      }
    }
  ]
}

Log Format Requirements​

  • Logs must be in JSON format
  • Timestamp should be in Unix epoch milliseconds in the _timestamp field
  • Each log entry will be sent as a separate JSON object

Validation and Troubleshooting​

Checking Log Router Status​

View the FireLens container logs in CloudWatch:

aws logs get-log-events \
    --log-group-name /aws/ecs/containerinsights/{cluster_name}/logs \
    --log-stream-name {task_id}/log_router/{container_id}

Common Issues and Resolutions​

Authentication Failures​

  • Issue: HTTP 401 errors in FireLens logs
  • Solution: Verify the API token is correctly formatted and not truncated
  • Check: Ensure the Authorization header is properly set in the task definition

Connection Issues​

  • Issue: Unable to reach CtrlB endpoint
  • Solution:
    • Verify security group rules allow outbound traffic
    • Check if VPC endpoints or NAT gateway is properly configured
    • Validate the endpoint URL format

TLS Certificate Errors​

  • Issue: SSL/TLS verification failures
  • Solution:
    • Set tls.verify": "off" for self-signed certificates (development only)
    • Ensure proper CA certificates are available in the FireLens container
    • Verify the hostname matches the certificate

Best Practices​

  1. Resource Allocation

    • Allocate sufficient memory for the FireLens container (minimum 50MB recommended)
    • Monitor FireLens container metrics for potential resource constraints

  2. Security

    • Use AWS Secrets Manager for storing sensitive configuration
    • Implement proper IAM roles and policies
    • Enable TLS for production environments

  3. Performance

    • Configure appropriate buffer sizes for high-volume logging
    • Implement log rotation policies
    • Monitor FireLens performance metrics in CloudWatch

  4. High Availability

    • Deploy tasks across multiple availability zones
    • Implement retry logic for log delivery
    • Configure dead letter queues for failed log delivery