Setting up an AWS Integration

CloudQuery Platform can connect to your AWS Organization or Account using a set of deployed IAM roles. Follow the steps in this document to set up a new integration with AWS.

Creating AWS Integration

Setting up an onboarding IAM role

In this step, CloudQuery will set up an onboarding IAM role that will make the following steps of configuring the integration easier. This onboarding role will be cleaned up after the integration is configured.

If you interrupt or pause the configuration, CloudQuery will clean up the onboarding IAM role after 24 hours.

1. Navigate to Data Pipelines → Integrations in CloudQuery Platform.

2. Click Create Integration and select AWS.

3. If you are planning to sync from only one AWS Account, select the Single account option. If you want to sync from an Organizational Unit or multiple Accounts, keep the Multiple accounts selected.

   

Multiple accounts

1. Click the Open AWS Console button. You will be taken to AWS Console and will be prompted to create a new CloudFormation Stack with IAM roles that will grant CloudQuery access to list the Organizational Units and Accounts and set up read-only IAM roles for the syncs.

2. Wait for the CloudFormation stack to be created in the AWS Console. Then return back to CloudQuery Platform. You will see a set of your Organizational Units and Accounts ready to be selected for syncs.

Deploying the IAM roles for syncs

1. Select the Organizational Units and Accounts to sync. All Accounts in the selected OUs will be synced, including those that will be added later. If you need to sync a specific account, move it to a separate Organizational Unit.

2. Click the Continue button, CloudQuery will start setting up read-only IAM roles for the individual Organizational units to enable the sync.

Single Account

1. Click the Open AWS Console button. You will be taken to AWS Console and will be prompted to create a new CloudFormation Stack with an IAM role that will grant CloudQuery access to the Account.

2. Wait for the CloudFormation stack to be created in the AWS Console. Then return back to CloudQuery Platform and click Continue.

Apply the CloudFormation stack and then click Continue

Selecting the services to sync

Select services

Select the services that you want to sync the data from. This will affect the API endpoints that will be queried and the tables created in your destination database.

By default, all enabled AWS services will be synced. Selecting services to the left will filter down the tables that will be synced.

The most popular services are listed on top. Note that some services may take a while to sync depending on the amount of resources used.

For the full list of tables and their schema, see AWS Integration Documentation.

Select regions

By default, all enabled AWS regions will be synced. Selecting regions will filter down the regions that will be synced.

Wait for the IAM Roles to be set up

CloudQuery may still be creating the IAM roles on the Organizational Units you selected in the previous step. Wait for this process to complete.

Test the connection and save

Click the Test and save button to check if CloudQuery can connect and to submit the configuration.

Next Steps

With your AWS integration created, you can now proceed to use it in a new sync. This will give you the opportunity to specify when your AWS sync should be run, and to which destination databases.