Integrate CtrlB Data Lake with Grafana

This document explains how to set up Grafana using a pre-built plugin folder, run it with Docker, and configure the CtrlB Lake Data Source.

---

Overview

The CtrlB Lake Data Source plugin enables seamless integration between Grafana and CtrlB's enterprise data platform, providing real-time analytics and visualization capabilities for your organization's data infrastructure.

Key Features

• Real-time data streaming from CtrlB Lake platform
• SQL-based querying with advanced analytics support
• Enterprise-grade security with custom authentication headers
• Scalable visualization supporting multiple concurrent users
• Docker-based deployment for consistent environments

---

System Requirements

Prerequisites

Before beginning the installation, ensure you have:

• CtrlB Lake plugin distribution package (ctrlb-grafana-plugin folder)
• Valid CtrlB Lake credentials (endpoint URL and authorization token)
• Network connectivity to CtrlB services

---

Installation and Configuration

Step 1: Environment Setup

1. Create a dedicated project directory:

   mkdir ctrlb-grafana-integration
   cd ctrlb-grafana-integration

2. Extract the plugin distribution:

   • Place the provided ctrlb-grafana-plugin folder in your project directory
   • Verify the folder contains the plugin files:

     ls -la ctrlb-grafana-plugin/

Step 2: Deploy Grafana Container

Execute the following command to deploy Grafana with the CtrlB Lake plugin:

docker run -d \
  --name ctrlb-lake-grafana \
  --restart unless-stopped \
  -u root \
  -p 3000:3000 \
  -v "$(pwd)/ctrlb-grafana-plugin:/var/lib/grafana/plugins/ctrlb-lake-datasource" \
  -e GF_LOG_LEVEL=info \
  -e GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=ctrlb-lake-datasource \
  -e GF_INSTALL_PLUGINS="" \
  grafana/grafana-oss:11.1.3

Step 3: Verify Installation

1. Check container status:

   docker ps | grep ctrlb-lake-grafana

2. Verify plugin loading:

   docker logs ctrlb-lake-grafana | grep "ctrlb-lake-datasource"

3. Access Grafana interface:

   • Navigate to: http://localhost:3000
   • Default credentials: admin / admin
   • Change the default password when prompted

---

Data Source Configuration

Step 1: Access Data Source Management

1. In the Grafana interface, navigate to Connections > Add new data source
2. Search for and select CtrlB Lake Data Source

Step 2: Configure Connection Parameters

Configure the following settings:

Parameter	Description	Example
Name	Descriptive name for the data source	CtrlB Lake Production
URL	CtrlB Lake endpoint URL	https://api.ctrlb.com/lake/v1
Authentication	Custom header configuration	See below

Step 3: Authentication Setup

Configure the custom authentication header:

1. Header Name: Authorization
2. Header Value: Basic [YOUR_API_TOKEN]

Security Note: Store API tokens securely and rotate them according to your organization's security policies.

Step 4: Connection Validation

1. Click Save & test to validate the configuration
2. Verify the success message: "Data source is working"
3. If connection fails, review the troubleshooting section below

---

Dashboard Development

Method 1: Using Explore (Recommended for Initial Setup)

1. From the data source configuration page, click Explore

2. Select your target stream from the dropdown

3. Enable SQL query mode

4. Enter your query in the editor:

   SELECT *
   FROM <STREAM_NAME>
   WHERE timestamp >= NOW() - INTERVAL '1 hour'
   ORDER BY timestamp DESC
   LIMIT 1000

5. Click Run query to execute and preview results

Method 2: Dashboard Creation

1. Navigate to Dashboards → New → New Dashboard

2. Click Add visualization

3. Configure the panel:

• Data source: Select your CtrlB Lake data source
• Stream: Choose the appropriate data stream
• Query: Write your SQL query
• Visualization: Select chart type (Time series, Table, etc.)

4. Configure panel options:
   • Title: Descriptive panel name
   • Description: Context and purpose
   • Refresh rate: Set appropriate interval

5. Click Apply to save the panel

Performance Optimization

Query Optimization Best Practices

1. Use appropriate time ranges to limit data volume
2. Limit result sets using LIMIT clauses
3. Leverage aggregation functions for summary views
4. Use dashboard variables to reduce query complexity

---

Troubleshooting

Common Issues and Resolutions

Plugin Loading Failures

Symptoms: Plugin not appearing in data sources list

Resolution:

1. Verify the GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS environment variable
2. Check plugin folder permissions: docker exec ctrlb-lake-grafana ls -la /var/lib/grafana/plugins/
3. Review container logs for plugin-related errors

Connection Authentication Errors

Symptoms: "Unauthorized" or "Forbidden" responses

Resolution:

1. Verify API token validity and permissions
2. Check authorization header format
3. Confirm network connectivity to CtrlB endpoints
4. Review firewall and proxy configurations

Diagnostic Commands

# Container resource usage
docker stats ctrlb-lake-grafana

# Detailed logs with timestamps
docker logs --timestamps ctrlb-lake-grafana

# Plugin directory contents
docker exec ctrlb-lake-grafana find /var/lib/grafana/plugins/ -type f

# Network connectivity test
docker exec ctrlb-lake-grafana wget --spider https://your-ctrlb-endpoint.com

# Database integrity check
docker exec ctrlb-lake-grafana sqlite3 /var/lib/grafana/grafana.db "PRAGMA integrity_check;"