Orchestrating CloudQuery Syncs with Kestra

This tutorial shows how to run CloudQuery as a Kestra (opens in a new tab) flow, using the HackerNews API as a source and CSV file as a destination. Kestra is an open-source event-driven orchestrator, which has a dedicated CloudQuery plugin (opens in a new tab).

You can use Kestra to:

• schedule CloudQuery syncs (opens in a new tab) in a simple, declarative way using a built-in code editor
• run your syncs based on events from other systems e.g. a new file in your S3 bucket, new message in a queue, or a new webhook request
• monitor your syncs from a friendly UI and get notified when they fail.

Prerequisites

• Docker installed (opens in a new tab)
• A CloudQuery API key

Step 1: Install Kestra

Follow the Getting started guide (opens in a new tab) to launch Kestra locally in Docker. It's as easy as running a single Docker command:

docker run --pull=always --rm -it -p 8080:8080 --user=root -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp kestra/kestra:latest-full server local

Once the container is running, open http://localhost:8080 (opens in a new tab) in your browser.

Step 2: Create a Kestra flow

Inside the Kestra UI, go to the Flows page and click on Create. You can now paste the following content:

id: cloudquery_sync
namespace: dev

tasks:
  - id: hn_to_csv
    type: io.kestra.plugin.cloudquery.Sync
    env:
      # Visit https://kestra.io/docs/concepts/secret to learn how to manage secrets in Kestra
      CLOUDQUERY_API_KEY: "{{ secret('CLOUDQUERY_API_KEY') }}"
    incremental: false
    outputFiles:
      - "cloudquery/**"
    configs:
      - kind: source
        spec:
          name: hackernews
          path: cloudquery/hackernews
          registry: cloudquery
          version: v3.0.21
          tables: ["*"]
          destinations:
            - file
          spec:
            item_concurrency: 100
            start_time: "2024-01-22T00:00:00Z"
      - kind: destination
        spec:
          name: file
          path: cloudquery/file
          version: v3.4.17
          spec:
            path: "cloudquery/{{TABLE}}/{{UUID}}.{{FORMAT}}"
            format: csv

The flow will extract all tables from the HackerNews API and load them to a CSV file. You can download that CSV file from the Outputs tab in the Kestra UI after each flow execution. You may notice that the configs property is virtually identical to your CloudQuery configuration file. All we had to add was to move the CloudQuery sources and destinations into a list of configs in the cloudquery.Sync task.

Once you configured the flow in the editor, click on Save.

Step 3: Run the flow

To manually start a Kestra flow, click on the Execute button in the top right corner. Then, confirm by clicking on Execute.

You should now see the sync running in the Executions tab. You can navigate to the Logs tab to check the logs and validate everything runs as expected. Also, you can download the extracted CSV files from the Outputs tab:

Step 4: Schedule the flow

To run the flow periodically, we can add a trigger to run it on a schedule. Back in the Flow editor, add the following section:

triggers:
  - id: schedule
    type: io.kestra.core.models.triggers.types.Schedule
    cron: "0 9 * * *" # every day at 9am
    timezone: US/Eastern

This cron expression will run the flow every day at 09:00. You can use crontab.guru (opens in a new tab) to generate cron expressions and replace the one in the example above. Kestra also supports these special values for cron:

@yearly
@annually
@monthly
@weekly
@daily
@midnight
@hourly

With this in place, remember to click Save again. Your CloudQuery sync will now run on a regular schedule.

Next steps

This tutorial was just a quick introduction to help you get started with a CloudQuery deployment on Kestra (opens in a new tab). You can now create additional Kestra tasks to perform transformations, send notifications and more. For more information, check out the CloudQuery docs and the Kestra docs (opens in a new tab).

To productionize your Kestra deployment, you will likely need to deploy it to a cloud environment, such as Kubernetes. For more information, see the Kestra Deployment with Kubernetes guide (opens in a new tab) and if you have any questions, you can reach the Kestra team via Community Slack (opens in a new tab).