CloudQuery Platform
  • Introduction
    • Welcome to CloudQuery Platform
    • Getting Help
  • Quickstart
    • Creating a New Account
    • Platform Activation
  • Core Concepts
    • Integrations
    • Syncs
    • Filters & Queries
    • SQL Console
    • Reports
  • Integration Guides
    • Setting up an AWS Integration
    • Setting up an AWS Cost and Usage Integration
    • Setting up a GCP Integration
    • Setting up an Azure Integration
    • Setting up a GitHub Integration
    • Setting up a K8s Integration
      • Using AWS EKS
      • Using Azure AKS
      • Using GCP GKE
    • General Integration Setup Guide
    • General Destination Setup Guide
  • Syncs
    • Setting up a Sync
    • Monitoring Sync Status
  • Cloud insights
    • From cloud asset inventory to insights
      • Security-focused queries
      • Compliance-focused queries
      • FinOps-focused queries
  • Production Deployment
    • Enabling Single Sign-on (SSO)
      • Single Sign-On with Google
      • Single Sign-On with Microsoft
      • Single Sign-On with Okta
  • User Management
    • Platform Roles Overview
    • Workspace Roles Overview
  • Advanced Topics
    • Custom Columns
    • Understanding Platform Views
    • Performance Tuning
  • Reference
    • Search & Filter Query Syntax
  • API Reference
  • CLI Docs
  • CloudQuery Hub
Powered by GitBook
On this page
  • Overview
  • Architecture
  • Infrastructure Components
  • Platform Components
  • Installation
  • Prerequisites
  • AWS Permissions
  • Option 1: Complete Infrastructure Deployment
  • Option 2: Minimal Deployment
  • Installing Dependencies
  • Install CloudQuery Platform
  • Set up a domain
  • Verification
  • Log in
  • Next steps

Was this helpful?

  1. Production Deployment

AWS Deployment Guide

Last updated 2 months ago

Was this helpful?

Overview

This Quick Start guide provides instructions for deploying a self-hosted CloudQuery Platform in Amazon Web Services (AWS). CloudQuery is an open source high-performance data integration platform that helps you transform your cloud infrastructure into queryable datasets.

This deployment is for customers who want to self-host a complete CloudQuery Cloud Governance solution. During deployment, customers can configure the AWS CloudFormation templates to define the desired cluster configuration and settings.

Architecture

Deploying this Quick Start for a new virtual private cloud (VPC) with default parameters builds the following CloudQuery environment in AWS Cloud using a combination of CloudFormation (for infrastructure deployment) and Helm (for application deployment in the k8s clusters).

Infrastructure Components

As shown in the diagram, the Quick Start sets up the following:

  • A highly available architecture that spans two Availability Zones.

  • A VPC configured with public and private subnets, according to AWS best practices, to provide you with your own virtual network on AWS.

  • An internet gateway to allow internet access for load balancers.

In the public subnets:

  • Managed NAT gateways to allow outbound internet access for resources in the private subnets.

  • Public load balancers for ingress traffic to the platform.

In the private subnets:

  • An Amazon EKS cluster for running the CloudQuery platform components

  • Worker nodes in an Auto Scaling group

  • Private subnets for the database tier

In the database subnets:

  • Amazon RDS PostgreSQL instance for platform state storage

  • Isolated network tier with no direct internet access

Additional Components:

  • AWS Secrets Manager for secure credentials storage

  • IAM roles and security groups for least-privilege access

Platform Components

The NGINX ingress controller provides L7 load balancing:

  • Terminates TLS at the edge

  • Provides URL-based routing

  • Integrates with cert-manager for automatic TLS

External Secrets Operator (ESO) provides a vendor-agnostic approach to secrets management in Kubernetes. It allows CloudQuery to:

  • Securely fetch secrets from AWS Secrets Manager

  • Automatically sync secrets to Kubernetes

  • Rotate credentials without pod restarts

  • Support multiple secret backends if needed

Cert-Manager automates certificate management within Kubernetes:

  • Automatically provisions TLS certificates from Let's Encrypt

  • Handles certificate renewal

  • Integrates with ingress resources

  • Provides vendor-agnostic certificate management

Installation

The installation guide provides two installation options using a CloudFormation template:

  • a complete infrastructure deployment (including VPC, EKS, and PostgreSQL), and

  • a minimal deployment that works with your existing EKS installation.

Prerequisites

  • AWS CLI v2.x

  • Kubectl v1.30 or later

  • Helm v3.x

The expected format for the ClickHouse DSN is: clickhouse://<user>:<password>@<host>:9440/<db>?secure

The minimal deployment requires additional prerequisites:

  • An existing EKS cluster

  • A Postgres Database and a connection string

The expected format for the Postgres DSN is: postgres://postgres:[email protected]:5432/postgres?sslmode=disable

  • A VPC

AWS Permissions

The AWS user/role executing the installation needs permissions for:

  • CloudFormation full access

  • EKS admin access

  • IAM create/modify permission

  • VPC full access

  • RDS full access

  • Secrets Manager write access

Option 1: Complete Infrastructure Deployment

This option will install everything you need as part of one CloudFormation stack, including a VPC, EKS, and PostgreSQL.

  1. Generate a private key:

openssl genrsa -out platform_private.pem
export ACTIVATION_KEY=<your-cloudquery-activation-key>
export CLICKHOUSE_DSN=<your-clickhouse-dsn>
export PRIVATE_KEY=$(cat platform_private.pem)
  1. Deploy the CloudFormation stack

     aws cloudformation create-stack \
     --stack-name cq-platform \
     --template-body file://template.yml \
     --capabilities CAPABILITY_NAMED_IAM \
     --parameters \
         ParameterKey=CreateInfrastructure,ParameterValue=true \
         ParameterKey=ActivationKey,ParameterValue=$ACTIVATION_KEY \
         ParameterKey=ClickHouseDSN,ParameterValue=$CLICKHOUSE_DSN \
         ParameterKey=PrivateKey,ParameterValue=$PRIVATE_KEY
       
  2. Wait for stack deployment to complete (this can take around 20 minutes).

     aws cloudformation wait stack-create-complete --stack-name cq-platform

Option 2: Minimal Deployment

This option assumes you already have a VPC, Postgres database and EKS cluster configured.

  1. Generate a private key:

openssl genrsa -out platform_private.pem
  1. Configure the following environment variables

export CLUSTER_NAME=<your-cluster-name>
export ACTIVATION_KEY=<your-cloudquery-activation-key>
export POSTGRES_DSN=<your-postgres-dsn>
export CLICKHOUSE_DSN=<your-clickhouse-dsn>
export EKSOIDCProviderURL=<your-eks-oidc-provider-url>
export PRIVATE_KEY=$(cat platform_private.pem)

The EKS OIDC provider URL can be found on the EKS cluster overview page:

  1. Deploy the CloudFormation stack

     aws cloudformation create-stack \
     --stack-name cq-platform \
     --template-body file://template.yml \
     --capabilities CAPABILITY_NAMED_IAM \
     --parameters \
         ParameterKey=CreateInfrastructure,ParameterValue=false \
         ParameterKey=ClusterName,ParameterValue=$CLUSTER_NAME \
         ParameterKey=ActivationKey,ParameterValue=$ACTIVATION_KEY \
         ParameterKey=ClickHouseDSN,ParameterValue=$CLICKHOUSE_DSN \
         ParameterKey=PostgresDSN,ParameterValue=$POSTGRES_DSN \
         ParameterKey=EKSOIDCProviderURL,ParameterValue=$EKSOIDCProviderURL
  2. Wait for stack deployment to complete (this can take around 20 minutes).

    aws cloudformation wait stack-create-complete --stack-name cq-platform

Installing Dependencies

  1. Once the stack creation is complete, configure access to the cluster

aws eks update-kubeconfig --name <stack-name>-cloudquery-platform
  1. Add and update the required Helm dependencies

helm repo add jetstack https://charts.jetstack.io
helm repo add external-secrets https://charts.external-secrets.io
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo add cloudquery https://cloudquery.github.io/helm-charts/
helm repo update
  1. Install the dependant operators

kubectl create namespace platform-system

helm upgrade --install external-secrets external-secrets/external-secrets \
 -n platform-system --wait

helm upgrade --install cert-manager jetstack/cert-manager \
 -n platform-system \
 --set crds.enabled=true --wait

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
 -n platform-system \
 --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"="internet-facing" \
 --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-attributes"="load_balancing.cross_zone.enabled=true" \
 --wait

Install CloudQuery Platform

  1. Create a helm values yaml file (e.g. platform-values.yml) filling in the relevant variables. See table below for explanation.

platform-values.yml
persistence:
  enabled: true
otelCollector:
  database: "<clickhouse_db_name_from_clickhouse_dsn>" # Update this
externalSecrets:
 enabled: true
 region: "<secrets-region>" # Update this
 cloudquerySecretsKey: "<secrets-key>" # Update this
 externalSecretsRoleARN: "<secret-role-ARN>" # Update this
letsEncrypt:
 enabled: true
 email: "<email-address>"  # Update this
ingress:
 enabled: true
 className: "nginx"
 annotations:
   cert-manager.io/cluster-issuer: letsencrypt-production
 hosts:
   - host: "<domain>"  # Update this
     paths:
       - path: /
         pathType: ImplementationSpecific
 tls:
     - secretName: cloudquery-platform-tls
       hosts:
         - "<domain>"  # Update this
Variable
Description

secrets-region

The AWS region where the cluster secrets are stored, which should be the same as the cluster

secrets-key

The full ARN of the secrets key in AWS. This can be found in the CloudFormation stack output variable CloudQueryPlatformSecrets

secret-role-ARN

The full ARN of the AWS role used by the secrets operator to fetch the secrets. This can be found in the CloudFormation stack output variable ExternalSecretsRole

email-address

The email address to use with Lets Encrypt for certificate registration

domain

The domain for which a certificate will be created. This needs to be a domain you own and are able to create DNS records for.

  1. Install the CloudQuery platform

helm upgrade --install cloudquery cloudquery/platform \
  -n cloudquery \
  --create-namespace \
  --values platform-values.yml \
  --wait
  1. Install the CloudQuery operator

helm upgrade scheduler cloudquery/operator \
  -n cloudquery --install --atomic

Set up a domain

  1. Retrieve the hostname of the NLB

kubectl get svc -n platform-system ingress-nginx-controller \
 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
  1. Add a DNS CNAME record to your DNS provider pointing the domain you selected in the previous section, which will direct traffic to your ingress URL and allow the certificate verification to complete.

Verification

  1. Verify all components are running

kubectl get pods -n cloudquery
kubectl get pods -n platform-system
  1. Check ingress status

kubectl get ingress -n cloudquery
  1. Check certificates status

kubectl get certificates -n cloudquery

Note: It may take from a few minutes to a few hours for the certificates to be installed. This mostly depends on DNS and its caching mechanism.

Log in

Next steps

See Platform Activation for how to activate your new CloudQuery installation.

See Adding S3-to-SQS notifications for information on how to set up an SQS queue which will receive `s3:ObjectCreated:*` messages when items are added to an existing bucket.

A CloudQuery activation key. This is usually an API key from . For more information, see .

ClickHouse Database and a ClickHouse DSN string (See for help).

Configure the following environment variables (see above for how to obtain these):

Download the CloudFormation template from: (Example: wget https://cq-cloud-cloudformation-templates.s3.us-east-1.amazonaws.com/cq-platform/template.yml)

Continue to .

Download the CloudFormation template from: (Example: wget https://cq-cloud-cloudformation-templates.s3.us-east-1.amazonaws.com/cq-platform/template.yml )

Continue to .

Once installed, log into the platform at https://<your-domain> using a username of and password pass - this will be changed on first login. Follow the setup instructions in the platform UI.

https://cloud.cloudquery.io/team-api-keys
Platform Activation
Deploying ClickHouse
https://cq-cloud-cloudformation-templates.s3.us-east-1.amazonaws.com/cq-platform/template.yml
https://cq-cloud-cloudformation-templates.s3.us-east-1.amazonaws.com/cq-platform/template.yml
admin@localhost
prerequisites
Installing Dependencies
Installing Dependencies
Reference Architecture for CloudQuery Platform on AWS
Where to find EKS OIDC provider URL on the EKS cluster overview page
CloudFormation outputs containing the secret and secret role ARNs
Reference Architecture for CloudQuery Platform on AWS
Where to find EKS OIDC provider URL on the EKS cluster overview page
CloudFormation outputs containing the secret and secret role ARNs