Destination Integrations

Destination integrations are responsible for writing data from source integrations to various storage systems. They handle schema migration, data persistence, and optimization for different storage backends.

How Destination Integrations Work

Schema Migration: Automatically create and update database schemas
Authentication: Connect to the destination system
Data Writing: Efficiently insert data using appropriate methods
Optimization: Use destination-specific optimizations for performance

Browse All Destination Integrations

For a complete list of available destination integrations with detailed documentation, visit the CloudQuery Hub.

Popular Destination Categories

Relational Databases

PostgreSQL: Full-featured relational database with JSON support
MySQL: Popular open-source database with wide compatibility
SQLite: Lightweight, file-based database for development and testing
SQL Server: Microsoft’s enterprise database platform

Data Warehouses

BigQuery: Google’s serverless data warehouse
Snowflake: Cloud-based data platform for analytics
ClickHouse: High-performance analytical database
Redshift: Amazon’s cloud data warehouse

NoSQL Databases

MongoDB: Document-based NoSQL database
Elasticsearch: Search and analytics engine
Neo4j: Graph database for relationship data

Message Queues & Streaming

Kafka: Distributed streaming platform
RabbitMQ: Message broker for applications
Apache Pulsar: Cloud-native messaging and streaming

File Storage

S3: Amazon’s object storage service
GCS: Google Cloud Storage
Azure Blob: Microsoft’s object storage
Local Files: CSV, JSON, Parquet formats

Destination Configuration Reference

Basic Configuration

Destination integrations are configured in your CloudQuery configuration file. Each destination requires:

Name: Unique identifier for the destination
Path: Plugin path (e.g., cloudquery/postgresql)
Version: Plugin version to use
Write Mode: How to handle data updates
Connection Details: Database connection information

Example configuration:

kind: destination
spec:
  name: postgresql
  path: cloudquery/postgresql
  registry: cloudquery
  version: "v8.13.0"
  spec:
    connection_string: "postgresql://user:pass@localhost:5432/dbname"
    write_mode: "overwrite-delete-stale"

Complete Destination Spec Reference

This goes through all the available options for the destination plugin spec object.

name

(string, required)

Name of the plugin. If you have multiple destination plugins, this must be unique.

The name field may be used to uniquely identify a particular destination configuration. For example, if you have two configs for the PostgreSQL plugin for syncing different databases, one may be named db-1 and the other db-2. In this case, the path option below must be used to specify the download path for the plugin.

registry

(string, optional, default: cloudquery, available: github, cloudquery, local, grpc, docker)

cloudquery: CloudQuery will look for and download the plugin from the official CloudQuery registry, and then execute it.
github: Deprecated. CloudQuery will look for and download the plugin from GitHub, and then execute it.
local: CloudQuery will execute the plugin from a local path.
grpc: mostly useful in debug mode when plugin is already running in a different terminal, CloudQuery will connect to the gRPC plugin server directly without spawning the process.
docker: CloudQuery will run the plugin in a Docker container. This is most useful for plugins written in Python, as they do not support the local, github and cloudquery registries.

path

(string, required)

Configures how to retrieve the plugin. The contents depend on the value of registry (github by default).

For plugins hosted on GitHub, path should be of the form "<org>/<repository>". For official plugins, should be cloudquery/<plugin-name>.
For plugins that are located in the local filesystem, path should a filesystem path to the plugin binary.
To connect to a running plugin via grpc (mostly useful for debugging), path should be the host-port of the plugin (e.g. localhost:7777).

version

(string, required)

version must be a valid SemVer), e.g. vMajor.Minor.Patch. You can find all official plugin versions under our GitHub releases page, and for community plugins you can find it in the relevant community repository.

write_mode

(string, optional, default: overwrite-delete-stale. Available: overwrite-delete-stale, overwrite, append)

Specifies the update method to use when inserting rows. The exact semantics depend on the destination plugin, and all destinations don’t support all options, so check the destination plugin documentation for details.

overwrite-delete-stale: syncs overwrite existing rows with the same primary key, and delete rows that are no longer present in the cloud.
overwrite: Same as overwrite-delete-stale, but doesn’t delete stale rows from previous syncs.
append: Rows are never overwritten or deleted, only appended.

Switching from overwrite-delete-stale or overwrite to append, or from append to overwrite-delete-stale or overwrite is not supported without dropping all tables specified in the configuration. To drop tables automatically, use the migrate_mode: forced option.

migrate_mode

(string, optional, default: safe. Available: safe, forced)

Specifies the migration mode to use when source tables are changed. In safe mode (the default), CloudQuery will not run migrations that would result in data loss, and will print an error instead. In forced mode, CloudQuery will run migrations that may result in data loss and the migration should succeed without errors, unless a table has user created dependent objects (e.g. views). Not all destination plugins support migrate_mode: forced, refer to the specific destination plugin page to see if it is supported.

Read more about how CloudQuery handles migrations here.

pk_mode

(string, optional, default: default, Available: default, cq-id-only introduced in CLI v2.5.2)

Specifies the Primary Keys that the destination will configure when using the overwrite or overwrite-delete-stale mode.

default: The default primary keys are used.
cq-id-only: The _cq_id field is used as the only primary key for each table. This is useful when you don’t want breaking changes to primary keys to impact your schema. It is highly recommended that if you are using this feature you should also use the deterministic_cq_id feature in the source. If you are using overwrite mode and a source updates a primary key, this will result in a new row being inserted. If you are using overwrite-delete-stale mode, a new row will be inserted and the old row will be deleted as a stale resource. Note: using this parameter might result in changes to query performance as CloudQuery will not be creating indexes for the default primary key columns.

Supported by destination plugins released on 2023-03-21 and later

sync_group_id

⚠️

Supported only for write_mode: append and write_mode: overwrite modes at the moment.

A value for an additional column named _cq_sync_group_id that will be added to each table. In overwrite mode the column will be added as an additional primary key. This is useful when splitting a sync into multiple parallel jobs. Using the same sync_group_id allows identifying separate syncs jobs as belonging to the same group. The value supports the following placeholders: {{SYNC_ID}}, {{YEAR}}, {{MONTH}}, {{DAY}}, {{HOUR}}, {{MINUTE}} which are set at sync time. Common use cases include:

Setting sync_group_id: "{{YEAR}}-{{MONTH}}-{{DAY}}" to group syncs by day, in order to provide a historical view of the data, partitioned by day.
Setting sync_group_id: "{{SYNC_ID}}" to enable joining data from different tables that were all part of the same sync job. The value of SYNC_ID can be controlled using the --invocation-id flag passed to the cloudquery sync command.

send_sync_summary

(bool, optional)

When set to true, CloudQuery will send a summary of the sync to the destination plugin. The summary includes the number of resources synced, number of errors and details about the plugins (both source and destination). This information will be available in the destination as a separate table named cloudquery_sync_summaries.

spec

(object, optional)

Plugin specific configurations. Visit destination plugins documentation for more information.

The following options are available for most destination plugins under the nested plugin spec:

batch_size

(int, optional)

The number of resources to insert in a single batch. Only applies to plugins that utilize batching. This setting works in conjunction with batch_size_bytes, and batches are written whenever either batch_size or batch_size_bytes is reached. Every plugin has its own default value for batch_size.

batch_size_bytes

(int, optional)

The max number of bytes to use for a single batch. Only applies to plugins that utilize batching. This setting works in conjunction with batch_size, and batches are written whenever either batch_size or batch_size_bytes is reached. Every plugin has its own default value for batch_size_bytes. Note that the size in bytes is calculated based on the size of data in memory, not the serialized data, and it is best to choose a batch_size_bytes significantly lower than any hard limits.

Write Modes

Different write modes control how data is updated:

overwrite-delete-stale: Replace existing data and remove stale records
overwrite: Replace existing data but keep stale records
append: Only add new data, never update or delete

Performance Optimization

Batch Settings

batch_size: Number of records per batch
batch_size_bytes: Maximum bytes per batch
batch_timeout: Maximum time to wait before flushing

Schema Optimization

Primary Keys: Configure primary key strategy
Indexes: Create indexes for better query performance
Partitioning: Use table partitioning for large datasets

Migration Handling

CloudQuery automatically handles schema migrations when source schemas change:

Safe Mode: Prevents data loss, requires manual intervention
Forced Mode: Automatically applies migrations that may cause data loss

Creating Custom Destinations

Need a destination that doesn’t exist? Learn how to create your own destination integration.

Sources Transformers