CLIDeploymentDocker - Offline Installation

Docker - Offline Installation

You can run CloudQuery in a container with integrations pre-installed. This is useful for isolated deployments where you don’t want to download integrations from the internet.

Downloading integrations requires users to be authenticated, normally this means running cloudquery login but that is not doable in a CI environment or inside of a docker build process. The recommended way to handle this is to use an API key. More information on generating an API Key can be found here

To download the integrations based on your configuration file, use the cloudquery plugin install command. Below is an example Dockerfile based on the CloudQuery container. It uses a build.spec.yaml with the minimum configuration required to download the integrations.

# build.spec.yaml
kind: source
spec:
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v32.40.1"
  tables: ["aws_ec2_instances"]
  destinations: ["postgresql"]
---
kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  registry: "cloudquery"
  version: "v8.12.0"
  spec:
# Dockerfile
FROM ghcr.io/cloudquery/cloudquery:latest AS build
WORKDIR /app
COPY ./build.spec.yaml /app/build.spec.yaml
ARG CLOUDQUERY_API_KEY

RUN /app/cloudquery plugin install build.spec.yaml

FROM ghcr.io/cloudquery/cloudquery:latest

WORKDIR /app

# Copy the .cq directory which contains the integrations
COPY --from=build /app/.cq /app/.cq

Build this container as you would normally do:

docker build --build-arg CLOUDQUERY_API_KEY=<your-api-key> ./ -t my-cq-container:latest

Run the Container

Run the container as you would run the default CloudQuery container. Here is an example:

docker run \
  # you can mount a different config file that uses the same integrations as in the build.spec
  -v <ABSOLUTE_PATH_TO_CONFIG_FILE>:/config.yml \
  # set any env variable with -e <ENV_VAR_NAME>=<ENV_VAR_VALUE>
  my-cq-container:latest \
  sync /config.yml

Troubleshooting

If you encounter the following error when running the cloudquery plugin install command:

tls: failed to verify certificate: x509: certificate signed by unknown authority

You probably need to install certificates in your container image. To identify which certificates are needed, you can run the following command:

openssl s_client -showcerts -connect api.cloudquery.io:443

To extract the certificates to files, you can use the following command. This will create a file for each certificate in the current directory.

openssl s_client -showcerts -connect api.cloudquery.io:443 </dev/null |
awk '/BEGIN CERTIFICATE/{n++;fname="cert" n ".pem"} /BEGIN CERTIFICATE/,/END CERTIFICATE/{print > fname}'

Then update your Dockerfile to copy over the certificates to your container and install them. The full Dockerfile should look like this:

# Dockerfile
FROM ghcr.io/cloudquery/cloudquery:latest AS build
RUN apk add --no-cache ca-certificates

WORKDIR /app
COPY ./build.spec.yaml /app/build.spec.yaml
COPY ./cert*.pem /usr/local/share/ca-certificates/
ARG CLOUDQUERY_API_KEY

ENV SSL_CERT_DIR=/usr/local/share/ca-certificates/

RUN /app/cloudquery plugin install build.spec.yaml

FROM ghcr.io/cloudquery/cloudquery:latest

WORKDIR /app

# Copy the .cq directory which contains the integrations
COPY --from=build /app/.cq /app/.cq

Related docs

Read more about the plugin install command in the CLI Documentation.

DockerGenerate an API Key