CloudQuery Platform
  • Introduction
    • Welcome to CloudQuery Platform
    • Getting Help
  • Quickstart
    • Creating a New Account
    • Platform Activation
  • Core Concepts
    • Integrations
    • Syncs
    • Filters & Queries
    • SQL Console
    • Reports
  • Integration Guides
    • Setting up an AWS Integration
    • Setting up an AWS Cost and Usage Integration
    • Setting up a GCP Integration
    • Setting up an Azure Integration
    • Setting up a GitHub Integration
    • Setting up a K8s Integration
      • Using AWS EKS
      • Using Azure AKS
      • Using GCP GKE
    • General Integration Setup Guide
    • General Destination Setup Guide
  • Syncs
    • Setting up a Sync
    • Monitoring Sync Status
  • Cloud insights
    • From cloud asset inventory to insights
      • Security-focused queries
      • Compliance-focused queries
      • FinOps-focused queries
  • Production Deployment
    • Enabling Single Sign-on (SSO)
      • Single Sign-On with Google
      • Single Sign-On with Microsoft
      • Single Sign-On with Okta
  • User Management
    • Platform Roles Overview
    • Workspace Roles Overview
  • Advanced Topics
    • Custom Columns
    • Understanding Platform Views
    • Performance Tuning
  • Reference
    • Search & Filter Query Syntax
  • API Reference
  • CLI Docs
  • CloudQuery Hub
Powered by GitBook
On this page
  • Prerequisites
  • IAM Role And Permissions
  • Creating AWS Integration
  • Next Steps

Was this helpful?

  1. Integration Guides

Setting up an AWS Integration

PreviousReportsNextSetting up an AWS Cost and Usage Integration

Last updated 9 days ago

Was this helpful?

CloudQuery Platform supports authentication with AWS through (IRSA).

The AWS accounts involved are:

  • CloudQuery Account: This is the AWS account where CloudQuery Platform is deployed. This account hosts the IAM role that CloudQuery uses to assume roles in other accounts.

  • Your Account: This is the AWS account that you want to sync resources from. This account will have a role that allows the CloudQuery account's role to assume and read resources.

Prerequisites

Before starting, configure the following environment variables - the account ID (this will be provided by the CloudQuery team) and the sub-domain of your installation.

export CLOUDQUERY_ACCOUNT_ID="<CloudQuery AWS account>"
export SUB_DOMAIN="<your installation subdomain>"

An external ID should be added as recommended by to provide an additional verification layer when assuming roles in a third-party account. This can be any alphanumeric string between 2 and 1224 characters, but in this example we use a UUID.

export EXTERNAL_ID=$(uuidgen)

IAM Role And Permissions

  1. Create the trust relationship for the cross-account role:

cat >third-party-trust.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::${CLOUDQUERY_ACCOUNT_ID}:role/${SUB_DOMAIN}-cloudquery-sync"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "${EXTERNAL_ID}"
                }
            }
        }
    ]
}
EOF
  1. Create the cross-account role and attach the ReadOnly policy:

aws iam create-role --role-name cross-account-readonly-role \
    --assume-role-policy-document file://third-party-trust.json

aws iam attach-role-policy \
    --role-name cross-account-readonly-role \
    --policy-arn="arn:aws:iam::aws:policy/ReadOnlyAccess"
  1. Run the following command replacing <ROOT_ORG_ID> with the OU of the root:

aws cloudformation create-stack --stack-name CloudQueryOrg-Deploy \
 --template-body file://./template.yml \
 --capabilities CAPABILITY_NAMED_IAM \
 --parameters \
    ParameterKey=OrganizationUnitList,ParameterValue=<ROOT_ORG_ID> \
    ParameterKey=ExternalID,ParameterValue=${EXTERNAL_ID} \
    ParameterKey=AdditionalTrustedArns,ParameterValue=arn:aws:iam::${CLOUDQUERY_ACCOUNT_ID}:role/${SUB_DOMAIN}-cloudquery-sync
  1. To retrieve the ARN of role in the Admin and the role deployed in each member account:

aws cloudformation describe-stacks --stack-name CloudQueryOrg-Deploy \
  --query "Stacks[].Outputs"

Continue to Creating AWS Integration

Creating AWS Integration

  1. Navigate to Data Pipelines → Integrations in CloudQuery Platform

  2. Click Create Integration and select AWS.

  1. Update the YAML configuration to sync to either a single account or multiple accounts e.g.

kind: source
spec:
  name: aws
  path: cloudquery/aws
  version: "v32.16.0" # latest version of source aws plugin
  tables:
    - aws_ec2_instances
  spec:
    accounts:
      - account_name: your_account
        role_arn: <your_account_role_arn>
        external_id: <external_id>

Note: the role_arn should be in the following form and correspond to the cross-account-readonly-role created in your AWS account: arn:aws:iam::<your_account>:role/cross-account-readonly-role

kind: source
spec:
  name: aws
  path: cloudquery/aws
  version: "v32.16.0" # latest version of source aws plugin
  tables:
    - aws_ec2_instances
  spec:
    org:
      admin_account:
        role_arn: arn:aws:iam::<your_account>:role/cloudquery-mgmt-ro
        external_id: <external_id>
      member_role_name: cloudquery-ro
  1. Click Test Connection to verify the setup

Next Steps

To automatically deploy the required roles across accounts within an organization clone the repository

With your AWS integration created, you can now proceed to use it in a . This will give you the opportunity to specify when your AWS sync should be run, and to which destination databases.

IAM Roles for Service Account
AWS best practices
iam-for-aws-orgs
new sync