Destination Spec Reference
This goes through all the available options for the destination plugin spec
object.
Example
This example configures the postgresql
plugin to connect to a PostgreSQL database located at localhost:5432
.
kind: destination
spec:
name: "postgresql"
path: "cloudquery/postgresql"
registry: "cloudquery"
version: "v8.7.4"
spec:
connection_string: "postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable"
Spec
name
(string
, required)
Name of the plugin. If you have multiple destination plugins, this must be unique.
The name field may be used to uniquely identify a particular destination configuration. For example, if you have two configs for the PostgreSQL plugin for syncing different databases, one may be named db-1
and the other db-2
. In this case, the path
option below must be used to specify the download path for the plugin.
registry
(string
, optional, default: cloudquery
, available: github
, cloudquery
, local
, grpc
, docker
)
cloudquery
: CloudQuery will look for and download the plugin from the official CloudQuery registry, and then execute it.github
: Deprecated. CloudQuery will look for and download the plugin from GitHub, and then execute it.local
: CloudQuery will execute the plugin from a local path.grpc
: mostly useful in debug mode when plugin is already running in a different terminal, CloudQuery will connect to the gRPC plugin server directly without spawning the process.docker
: CloudQuery will run the plugin in a Docker container. This is most useful for plugins written in Python, as they do not support thelocal
,github
andcloudquery
registries.
path
(string
, required)
Configures how to retrieve the plugin. The contents depend on the value of registry
(github
by default).
- For plugins hosted on GitHub,
path
should be of the form"<org>/<repository>"
. For official plugins, should becloudquery/<plugin-name>
. - For plugins that are located in the local filesystem,
path
should a filesystem path to the plugin binary. - To connect to a running plugin via
grpc
(mostly useful for debugging),path
should be the host-port of the plugin (e.g.localhost:7777
).
version
(string
, required)
version
must be a valid SemVer (opens in a new tab)), e.g. vMajor.Minor.Patch
. You can find all official plugin versions under our GitHub releases page (opens in a new tab), and for community plugins you can find it in the relevant community repository.
write_mode
(string
, optional, default: overwrite-delete-stale
. Available: overwrite-delete-stale
, overwrite
, append
)
Specifies the update method to use when inserting rows. The exact semantics depend on the destination plugin, and all destinations don't support all options, so check the destination plugin documentation for details.
overwrite-delete-stale
:sync
s overwrite existing rows with the same primary key, and delete rows that are no longer present in the cloud.overwrite
: Same asoverwrite-delete-stale
, but doesn't delete stale rows from previoussync
s.append
: Rows are never overwritten or deleted, only appended.
Switching from
overwrite-delete-stale
oroverwrite
toappend
, or fromappend
tooverwrite-delete-stale
oroverwrite
is not supported without dropping all tables specified in the configuration. To drop tables automatically, use themigrate_mode: forced
option.
migrate_mode
(string
, optional, default: safe
. Available: safe
, forced
)
Specifies the migration mode to use when source tables are changed. In safe
mode (the default), CloudQuery will not run migrations that would result in data loss, and will print an error instead. In forced
mode, CloudQuery will run migrations that may result in data loss and the migration should succeed without errors, unless a table has user created dependent objects (e.g. views).
Not all destination plugins support migrate_mode: forced
, refer to the specific destination plugin page to see if it is supported.
Read more about how CloudQuery handles migrations here.
pk_mode
(string
, optional, default: default
, Available: default
, cq-id-only
introduced in CLI v2.5.2
)
Specifies the Primary Keys that the destination will configure when using the overwrite
or overwrite-delete-stale
mode.
default
: The default primary keys are used.cq-id-only
: The_cq_id
field is used as the only primary key for each table. This is useful when you don't want breaking changes to primary keys to impact your schema. It is highly recommended that if you are using this feature you should also use thedeterministic_cq_id
feature in the source. If you are usingoverwrite
mode and a source updates a primary key, this will result in a new row being inserted. If you are usingoverwrite-delete-stale
mode, a new row will be inserted and the old row will be deleted as a stale resource. Note: using this parameter might result in changes to query performance as CloudQuery will not be creating indexes for the default primary key columns.
Supported by destination plugins released on 2023-03-21 and later
sync_group_id (preview)
Supported only for write_mode: append
and write_mode: overwrite
modes at the moment.
A value for an additional column named _cq_sync_group_id
that will be added to each table. In overwrite
mode the column will be added as an additional primary key.
This is useful when splitting a sync into multiple parallel jobs (opens in a new tab). Using the same sync_group_id
allows identifying separate syncs jobs as belonging to the same group.
The value supports the following placeholders: {{SYNC_ID}}, {{YEAR}}, {{MONTH}}, {{DAY}}, {{HOUR}}, {{MINUTE}}
which are set at sync time.
Common use cases include:
- Setting
sync_group_id: "{{YEAR}}-{{MONTH}}-{{DAY}}"
to group syncs by day, in order to provide a historical view of the data, partitioned by day. - Setting
sync_group_id: "{{SYNC_ID}}"
to enable joining data from different tables that were all part of the same sync job. The value ofSYNC_ID
can be controlled using the--invocation-id
flag passed to thecloudquery sync
command.
send_sync_summary (preview)
(bool
, optional)
When set to true
, CloudQuery will send a summary of the sync to the destination plugin. The summary includes the number of resources synced, number of errors and details about the plugins (both source and destination). This information will be available in the destination as a separate table named cloudquery_sync_summaries
.
spec
(object
, optional)
Plugin specific configurations. Visit destination plugins (opens in a new tab) documentation for more information.
The following options are available for most destination plugins under the nested plugin spec:
batch_size
(int
, optional)
The number of resources to insert in a single batch. Only applies to plugins that utilize batching. This setting works in conjunction with batch_size_bytes
, and batches are written whenever either batch_size
or batch_size_bytes
is reached. Every plugin has its own default value for batch_size
.
batch_size_bytes
(int
, optional)
The max number of bytes to use for a single batch. Only applies to plugins that utilize batching. This setting works in conjunction with batch_size
, and batches are written whenever either batch_size
or batch_size_bytes
is reached. Every plugin has its own default value for batch_size_bytes
. Note that the size in bytes is calculated based on the size of data in memory, not the serialized data, and it is best to choose a batch_size_bytes
significantly lower than any hard limits.