PlatformIntegration GuidesAWS

Setting up an AWS Integration

CloudQuery Platform supports authentication with AWS through IAM Roles for Service Account (IRSA).

The AWS accounts involved are:

  • CloudQuery Account: This is the AWS account where CloudQuery Platform is deployed. This account hosts the IAM role that CloudQuery uses to assume roles in other accounts.
  • Your Account: This is the AWS account that you want to sync resources from. This account will have a role that allows the CloudQuery account’s role to assume and read resources.

Prerequisites

Before starting, configure the following environment variables - the account ID (this will be provided by the CloudQuery team) and the sub-domain of your installation.

export CLOUDQUERY_ACCOUNT_ID="<CloudQuery AWS account>"
export SUB_DOMAIN="<your installation subdomain>"

An external ID should be added as recommended by AWS best practices to provide an additional verification layer when assuming roles in a third-party account. This can be any alphanumeric string between 2 and 1224 characters, but in this example we use a UUID.

export EXTERNAL_ID=$(uuidgen)

IAM Role And Permissions

Use this approach if you want to sync only from one AWS Account.

  1. Create the trust relationship for the cross-account role:
cat >third-party-trust.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::${CLOUDQUERY_ACCOUNT_ID}:role/${SUB_DOMAIN}-cloudquery-sync"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "${EXTERNAL_ID}"
                }
            }
        }
    ]
}
EOF
  1. Create the cross-account role and attach the ReadOnly policy:
aws iam create-role --role-name cross-account-readonly-role \
    --assume-role-policy-document file://third-party-trust.json

aws iam attach-role-policy \
    --role-name cross-account-readonly-role \
    --policy-arn="arn:aws:iam::aws:policy/ReadOnlyAccess"

Creating AWS Integration

  1. Navigate to Data Pipelines → Integrations in CloudQuery Platform
  2. Click Create Integration and select AWS.

AWS Yaml Configuration

  1. Update the YAML configuration to sync to either a single account or multiple accounts e.g.
kind: source
spec:
  name: aws
  path: cloudquery/aws
  version: "v32.26.0" # latest version of source aws plugin
  tables:
    - aws_ec2_instances
  spec:
    accounts:
      - account_name: your_account
        role_arn: <your_account_role_arn>
        external_id: <external_id>

Note: the role_arn should be in the following form and correspond to the cross-account-readonly-role created in your AWS account:

arn:aws:iam::<your_account>:role/cross-account-readonly-role

  1. Click Test Connection to verify the setup

Next Steps

With your AWS integration created, you can now proceed to use it in a new sync. This will give you the opportunity to specify when your AWS sync should be run, and to which destination databases.

AlertsAWS Cost & Usage