Setting up an AWS Integration

CloudQuery Platform supports authentication with AWS through IAM Roles for Service Account (IRSA).

The AWS accounts involved are:

• CloudQuery Account: This is the AWS account where CloudQuery Platform is deployed. This account hosts the IAM role that CloudQuery uses to assume roles in other accounts.

• Your Account: This is the AWS account that you want to sync resources from. This account will have a role that allows the CloudQuery account's role to assume and read resources.

Prerequisites

Before starting, configure the following environment variables - the account ID (this will be provided by the CloudQuery team) and the sub-domain of your installation.

export CLOUDQUERY_ACCOUNT_ID="<CloudQuery AWS account>"
export SUB_DOMAIN="<your installation subdomain>"

An external ID should be added as recommended by AWS best practices to provide an additional verification layer when assuming roles in a third-party account. This can be any alphanumeric string between 2 and 1224 characters, but in this example we use a UUID.

export EXTERNAL_ID=$(uuidgen)

IAM Role And Permissions

Single Account

1. Create the trust relationship for the cross-account role:

cat >third-party-trust.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::${CLOUDQUERY_ACCOUNT_ID}:role/${SUB_DOMAIN}-cloudquery-sync"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "${EXTERNAL_ID}"
                }
            }
        }
    ]
}
EOF

1. Create the cross-account role and attach the ReadOnly policy:

aws iam create-role --role-name cross-account-readonly-role \
    --assume-role-policy-document file://third-party-trust.json

aws iam attach-role-policy \
    --role-name cross-account-readonly-role \
    --policy-arn="arn:aws:iam::aws:policy/ReadOnlyAccess"

Multi Account

1. To automatically deploy the required roles across accounts within an organization clone the iam-for-aws-orgs repository

2. Run the following command replacing <ROOT_ORG_ID> with the OU of the root:

aws cloudformation create-stack --stack-name CloudQueryOrg-Deploy \
 --template-body file://./template.yml \
 --capabilities CAPABILITY_NAMED_IAM \
 --parameters \
    ParameterKey=OrganizationUnitList,ParameterValue=<ROOT_ORG_ID> \
    ParameterKey=ExternalID,ParameterValue=${EXTERNAL_ID} \
    ParameterKey=AdditionalTrustedArns,ParameterValue=arn:aws:iam::${CLOUDQUERY_ACCOUNT_ID}:role/${SUB_DOMAIN}-cloudquery-sync

1. To retrieve the ARN of role in the Admin and the role deployed in each member account:

aws cloudformation describe-stacks --stack-name CloudQueryOrg-Deploy \
  --query "Stacks[].Outputs"

Continue to Creating AWS Integration

Creating AWS Integration

1. Navigate to Data Pipelines → Integrations in CloudQuery Platform

2. Click Create Integration and select AWS.

1. Update the YAML configuration to sync to either a single account or multiple accounts e.g.

Single Account

kind: source
spec:
  name: aws
  path: cloudquery/aws
  version: "v32.15.0" # latest version of source aws plugin
  tables:
    - aws_ec2_instances
  spec:
    accounts:
      - account_name: your_account
        role_arn: <your_account_role_arn>
        external_id: <external_id>

Note: the role_arn should be in the following form and correspond to the cross-account-readonly-role created in your AWS account: arn:aws:iam::<your_account>:role/cross-account-readonly-role

Multi Account

kind: source
spec:
  name: aws
  path: cloudquery/aws
  version: "v32.15.0" # latest version of source aws plugin
  tables:
    - aws_ec2_instances
  spec:
    org:
      admin_account:
        role_arn: arn:aws:iam::<your_account>:role/cloudquery-mgmt-ro
        external_id: <external_id>
      member_role_name: cloudquery-ro

1. Click Test Connection to verify the setup

Next Steps

With your AWS integration created, you can now proceed to use it in a new sync. This will give you the opportunity to specify when your AWS sync should be run, and to which destination databases.