CloudQuery Platform
  • Introduction
    • Welcome to CloudQuery Platform
    • Getting Help
  • Quickstart
    • Creating a New Account
    • Platform Activation
  • Core Concepts
    • Integrations
    • Syncs
    • Filters & Queries
    • SQL Console
    • Reports
      • Built-in Report Templates
      • Reports Yaml Documentation with Examples
        • Full Report Example
  • Integration Guides
    • Setting up an AWS Integration
    • Setting up an AWS Cost and Usage Integration
    • Setting up a GCP Integration
    • Setting up an Azure Integration
    • Setting up a GitHub Integration
    • Setting up a K8s Integration
      • Using AWS EKS
      • Using Azure AKS
      • Using GCP GKE
    • General Integration Setup Guide
    • General Destination Setup Guide
  • Syncs
    • Setting up a Sync
    • Monitoring Sync Status
  • Cloud insights
    • From cloud asset inventory to insights
      • Security-focused queries
      • Compliance-focused queries
      • FinOps-focused queries
  • Production Deployment
    • Enabling Single Sign-on (SSO)
      • Single Sign-On with Google
      • Single Sign-On with Microsoft
      • Single Sign-On with Okta
  • User Management
    • Platform Roles Overview
    • Workspace Roles Overview
  • Advanced Topics
    • Custom Columns
    • Understanding Platform Views
    • Performance Tuning
  • Reference
    • Search & Filter Query Syntax
  • API Reference
  • CLI Docs
  • CloudQuery Hub
Powered by GitBook
On this page
  • Source Integrations
  • Destinations
  • Transformers
  • Example

Was this helpful?

  1. Core Concepts

Integrations

One of CloudQuery's key strengths is its ability to bring together data from hundreds of different sources. It does this through a plugin-based architecture, with source, transformer and destination plugins acting as independent components.

Source Integrations

The core responsibilities of a source integration are:

  • Define the schema (tables).

  • Authenticate with the supported API, SaaS service and/or cloud provider.

  • Extracting data from the supported APIs and transform them into the defined schema.

  • Send the data for further processing by the rest of the pipeline.

Destinations

The core responsibilities of a destination integration are:

  • Authenticate with the destination (such as database, message queue, storage).

  • Auto-migrate the schemas defined by the source integrations.

  • Save each incoming object in the appropriate table.

Transformers

Transformers can only be used with CloudQuery CLI at the moment

Transformers are optional components that sit in between the source and destination. They allow you to make changes to the data before they are written to the database, such as modifying the table name, obfuscating secrets, or adding additional columns.

Transformers are only supported with CloudQuery CLI syncs at the moment.

Example

For example, consider setting up an AWS integration (Source) to fetch resources from AWS, such as EC2 instances, S3 buckets, network topology, and more. This source integration delivers the data to the next step in the pipeline, which could be a transformer or multiple destinations.

By default, the CloudQuery platform stores data synced by source integrations in ClickHouse as the default destination. You can also sync the data to additional destinations simultaneously, without triggering extra API requests from the source integration.

PreviousPlatform ActivationNextSyncs

Last updated 3 months ago

Was this helpful?