Using the S3 Destination

To use the S3 destination for running syncs outside of CloudQuery platform, follow the steps below:

• Get the connection DSN for the CloudQuery ClickHouse destination. You may need to contact CloudQuery support to get it.

• Create an S3 bucket to be used for syncing data

• Add the following destination configuration your existing sync configuration

kind: destination
spec:
  name: "s3"
  path: "cloudquery/s3"
  version: "v7.9.1" # latest version of destination s3 plugin
  write_mode: "append"
  sync_group_id: "{{SYNC_ID}}"
  send_sync_summary: true
  spec:
    bucket: "<bucket>"
    region: "us-east-1" # Example: us-east-1
    path: "data/{{SYNC_ID}}/{{TABLE}}.{{FORMAT}}"
    format: "parquet"
    test_write: false
    no_rotate: true

• Setup S3 to SQS notifications