Running syncs outside the Platform

Running syncs outside from outside Platform is not recommended in general, but it may be useful during a migration period where you are migrating from CloudQuery CLI-based syncs to Platform.

While CloudQuery platform supports running syncs out of the box, if you already have syncs configured on your own infrastructure, you can integrate them with the CloudQuery platform asset inventory.

To ensure compatibility with CloudQuery platform please add the following configurations to your sources configuration spec:

• At the top level spec level add

  • deterministic_cq_id: true

  • destinations: [s3] (see below on configuring the S3 destination)

• At the plugin spec level add

  • add_cq_client_id: true

For example, the following example spec:

kind: source
spec:
  name: aws
  path: cloudquery/aws
  version: "v32.16.0" # latest version of source aws plugin
  tables: ["aws_ec2_instances"]
  destinations: ["postgresql"]

Should change to:

kind: source
spec:
  name: aws
  path: cloudquery/aws
  version: "v32.16.0" # latest version of source aws plugin
  tables: ["aws_ec2_instances"]
  destinations: ["postgresql", "s3"]
  deterministic_cq_id: true
  spec:
    add_cq_client_id: true

To continue setting up self hosted syncs follow either S3 destination guide.