Environment and file variable substitution

CloudQuery configuration .yml files support substitution of values from environment variables. This allows you to keep sensitive data (like passwords & tokens) or variable data (that you want to change without touching CloudQuery configuration) outside the configuration file and load them from environment variables at run-time.

Environment variable substitution example

Inside postgresql.yml:

kind: "destination"
spec:
  name: "postgresql"
  spec:
    connection_string: ${PG_CONNECTION_STRING}

PG_CONNECTION_STRING will be sourced from the environment and replaced with the value of ${PG_CONNECTION_STRING} before processing.

File variable substitution example

Inside postgresql.yml:

kind: "destination"
spec:
  name: "postgresql"
  spec:
    connection_string: ${file:./path/to/secret/file}

Local path ./path/to/secret/file will be read and replaced with the contents of the file before processing.

Environment variables with multi-line JSON

Multi-line JSON, such as those required by the service account key for the GCP integration, can be imported by using pipe '|' operator. The substitution should be in the next line and it should be indented by a single tab before. You don't need to escape any characters while passing the variable.

Inside gcp.yml:

kind: "source"
spec:
  name: "gcp"
  spec:
    service_account_key_json: |
      ${GCP_SERVICE_ACCOUNT_KEY_JSON}

JSON files in older versions

If the file or environment variable being substituted in contains JSON, it should be imported as-is. If you're using CloudQuery version 3.5.0 or prior, it should be imported inside single quotes and content should be escaped with newlines removed.

kind: "destination"
spec:
  name: "bigquery"
  spec:
    service_account_key_json: '${file:./path/to/secret/file.json}' # single quotes only for CLI versions 3.5.0 or prior

Time variable substitution example

CloudQuery configuration files support value substitution using both relative and fixed timestamps. Relative timestamps can be specified as now, x seconds [ago|from now], x minutes [ago|from now], x hours [ago|from now], x days [ago|from now], or until. Fixed timestamps follow the RFC3339 (e.g. 2025-03-21T08:53:19+00:00) or dateOnly formats (e.g. 2025-01-01)

Inside aws.yml:

kind: source
spec:
  # Source spec section
  name: aws
  spec:
    table_options:
      aws_cloudtrail_events:
        lookup_events:
          - start_time: ${time:5 days ago}
            end_time: ${time:now}
      aws_cloudwatch_metrics:
        - list_metrics:
            namespace: AWS/RDS
            metric_name: DatabaseConnections
          get_metric_statistics:
            - start_time: ${time:2022-01-01} # dateOnly format
              end_time: ${time:2022-04-02T15:04:05Z} # RFC3339 format

For the lookup_events option, end_time will be replaced with the current time, and start_time will be replaced with the time 5 days ago. For the get_metric_statistics option end_time and start_time will be replaced by the full RFC3339 formatted value of the fixed time.

Kestra Migrations