Skip to main content

Object Storage to CtrlB

This guide explains how to configure AWS, GCP, and Azure object storage in CtrlB for ingesting historical data and streaming live events.

1. Backfill Blob Uploader​

Backfill (Historical) mode allows users to ingest existing files from cloud storage into CtrlB for a specified time range. It is supported for AWS, GCP, and Azure sources.

How to Use It:​

  1. Go to the Integrations page.
  2. Under Clouds, select AWS, GCP, or Azure and click Add.
  3. For AWS and GCP, enter the Bucket name. For Azure, enter the Container name.
  4. Enter an optional Prefix to limit the files to a specific path.
  5. Enter the optional Start Date and Time and End Date and Time for the historical data to ingest.
  6. Provide the required cloud credentials by either pasting the JSON or uploading a credentials file.
  7. Click Test Connection to verify access. If the connection test fails, correct the credentials before proceeding to the next step.
  8. Click Next.
  9. Enter a Source Display Name and select the destination Dataset.
  10. Click Next.
  11. Review the source and CtrlB-related configuration.
  12. Click Create Source to start the backfill.

Availability:​

Available for users with permission to configure integrations.

Objectstorage

2. Live Blob Uploader​

Live Blob Uploader allows users to ingest newly uploaded files in real time using SQS Events. Currently, Live Blob Uploader is available only for AWS node. Users can configure the S3 bucket, SQS queue, AWS credentials, and destination dataset while creating the source.

How to Use It:​

  1. Go to the Integrations page.

  2. Under Clouds, select AWS and click Add.

  3. Enter the Bucket name and an optional Prefix.

  4. Select Live (SQS Events) under Ingestion Mode.

  5. Enter the SQS Queue URL.

  6. Provide AWS credentials by either pasting the JSON or uploading a credentials file.

  7. Click Test Connection to verify the connection.

  8. Click Next.

  9. Enter a Source Display Name and select the destination Dataset.

  10. Click Next.

  11. Review the source and CtrlB-related configuration.

  12. Click Create Source to complete the setup.

    Note:​

    Ensure the AWS credentials have the required permissions to access the configured S3 bucket and SQS queue.

Availability:​

Available for users with permission to configure integrations.

Objectstorage