Setting up a Sync

Learn how to set up a sync from source to destination

A CloudQuery sync fetches data from a source integration and delivers it to one or more destination integrations. This might mean fetching data from AWS and delivering it to ClickHouse, or it could mean fetching data from GCP and delivering it to BigQuery, Kafka and Neo4j, all at once. It all depends on the pipeline. Learn more about Syncs in the Core Concepts section.

Syncs are part of the Data Pipelines feature of CloudQuery platform. They connect Integrations and Destinations and specify how often the data should be moved.

Prerequisites

To create a sync, you need to create a new integration first. See the General Integration Setup Guide or one of the specific Integration Guides in the Integration Guides section.

Additionally, you will need a destination to sync to. CloudQuery Platform comes with the default ClickHouse database destination. We recommend using this destination if you want to benefit from other CloudQuery Platform features, such as SQL Console.

If you want to set up a new destination, see General Destination Setup Guide.

Creating a new Sync with the default destination

Navigate to Data Pipelines → Syncs and click the Create Sync button. If you have an integration set up, choose Use existing integration. Otherwise leave the Create a new integration selected and click the Next button.

Using an existing integration

Click the integration you want to set up the sync for. Then proceed to Select a destination.

Create a new integration

Follow the steps in the General Integration Setup Guide. Then proceed to Select a destination.

Select a destination

If you want to sync to CloudQuery default destination or to another existing destination, leave the Use existing integration selected. Use the checkboxes on the left to select the desired destinations and then click Sync selected destinations. Proceed to Configuring Sync options.

If you do not want to sync to CloudQuery default destination and you do not have another destination created yet, switch to Create a new destination and follow the General Destination Setup Guide. Then proceed to Configuring Sync options.

Configuring Sync options

Specify the name for the sync that will help you recognize it. Proceed with specifying the schedule for the sync and allocation resources for it.

Schedule

You can choose to run syncs Daily, Weekly, Monthly, or manually (No Schedule). Syncs scheduled with one of these values will run at the time specified below the dropdown with options.

If you want to run syncs at a different frequency or at a different time, use the Advanced option and enter a cron expression. The actual frequency and time of the sync will be explained below the input.

Allocating resources

To limit or upgrade the CPU and Memory used by the sync, use the advanced options. We recommend keeping the defaults unless you experience issues with the sync. For syncs with slow sources or low amount of rows, you can lower the amount of CPU and Memory.

The CPU and Memory used by the sync may be additionally limited by the underlying infrastructure used for the syncs.

Next Steps

To run the sync immediately, click the Schedule and Run button. This will start the sync immediately and then run the sync on the specified schedule.

To only schedule the sync withour running it immediately, click the Schedule button.

Once the sync is started, you will be redirected to the Sync Runs page.