Add observability to your CloudQuery sync runs with Datadog
In this guide we will discuss how to leverage Datadog (opens in a new tab) to add observability to CloudQuery sync jobs. A common use case for CloudQuery is to run it as a scheduled job to sync cloud resources to data warehouse, or even multiple jobs each with a different configuration. In order to get insights into runs of sync jobs, we can use Datadog (opens in a new tab) to monitor the sync jobs and get alerts when something goes wrong.
- A Datadog account. See here (opens in a new tab) for getting started
- The Datadog agent installed in the machine(s) running the CloudQuery sync jobs. See here (opens in a new tab) for more information
- Log collection enabled in the Datadog agent configuration. See here (opens in a new tab) for more information
First we recommended passing the
--log-format json flag to the CloudQuery sync command. This ensures that the logs are in a structured format that Datadog can parse.
Datadog can collect logs in non JSON format as well, but using JSON format ensure that we can query the logs.
Now, let's create the custom log source in Datadog:
- Locate the Datadog
conf.ddirectory. You can find the location of the directory based on our OS in the Datadog documentation here (opens in a new tab).
- Under the
conf.ddirectory create a directory named
- Under the
cloudquery.ddirectory create a file named
conf.yamlwith the following content:
logs: - type: "file" path: "<full path to the CloudQuery log file>" # Example path: /var/log/cloudquery/cloudquery.log service: "CloudQuery" source: "CloudQuery"
- Restart (opens in a new tab) the Datadog agent for changes to take effect.
You can find more information on setting up a custom log source in Datadog here (opens in a new tab)
Once you run a CloudQuery sync job, you should see the logs in the Datadog logs explorer:
If you don't see any logs you might need to chose a different time range in the Datadog logs explorer:
By using JSON format we can immediately query the logs in Datadog.
For example the
@errors:>0 @table:* query filters all tables that had errors during the sync job:
@resources:>10000 @table:* query filters all tables that had more than 10,000 resources during the sync job:
In this guide we created a custom log source in Datadog to collect CloudQuery logs and query them.
Datadog is a powerful tool for monitoring and observability and we only scratched the surface of what you can do with it. We recommend reading the Datadog documentation (opens in a new tab) to learn more about its capabilities, for example creating monitors (opens in a new tab) and dashboards (opens in a new tab).