Setting up an AWS Cost and Usage Integration
CloudQuery Platform supports authentication with an S3 bucket that has Cost and Usage reports data through IAM Roles for Service Account (IRSA).
The AWS accounts involved are:
CloudQuery Account: This is the AWS account where CloudQuery Platform is deployed. This account hosts the IAM role that CloudQuery uses to assume roles in other accounts.
Your Account: This is the AWS account that you want to sync resources from. This account will have a role that allows the CloudQuery account's role to assume and read resources.
Prerequisites
Before starting, configure the following environment variables - the account ID (this will be provided by the CloudQuery team) and the sub-domain of your installation.
export CLOUDQUERY_ACCOUNT_ID="<CloudQuery AWS account>"
export SUB_DOMAIN="<your installation subdomain>"
export S3_COST_REPORTS_BUCKET="<your bucket with the cost reports>"
An external ID should be added as recommended by AWS best practices to provide an additional verification layer when assuming roles in a third-party account. This can be any alphanumeric string between 2 and 1224 characters, but in this example we use a UUID.
export EXTERNAL_ID=$(uuidgen)
IAM Role And Permissions
Create the trust relationship for the cross-account role:
cat >third-party-trust.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${CLOUDQUERY_ACCOUNT_ID}:role/${SUB_DOMAIN}-cloudquery-sync"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "${EXTERNAL_ID}"
}
}
}
]
}
EOF
Create a policy to allow reading the Cost and Usage reports from your bucket
cat >bucket-policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::${S3_COST_REPORTS_BUCKET}"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::${S3_COST_REPORTS_BUCKET}/*"
}
]
}
EOF
Create the cross-account role and attach the ReadOnly policy:
aws iam create-role --role-name cross-account-readonly-role-cost-usage \
--assume-role-policy-document file://third-party-trust.json
aws iam put-role-policy --role-name cross-account-readonly-role-cost-usage \
--policy-name ReadS3Policy --policy-document file://bucket-policy.json
Continue to Creating AWS Cost and Usage Integration
Creating AWS Cost and Usage Integration
Navigate to Data Pipelines → Integrations in CloudQuery Platform
Click Create Integration and select AWS Cost and Usage.
Update the YAML configuration to sync to either a single account or multiple accounts e.g.
kind: source
spec:
name: awscur
path: cloudquery/awscur
version: v1.0.0 # latest version of source awscur plugin
tables:
- *
spec:
bucket: '<bucket>'
region: '<region>'
reports:
- path: '<path-to-parquet-files>/'
role_to_assume:
arn: arn:aws:iam::<your_account>:role/cross-account-readonly-role-cost-usage
external_id: <external_id>
Click Test Connection to verify the setup
Next Steps
With your AWS Cost and Usage integration created, you can now proceed to use it in a new sync. This will give you the opportunity to specify when your AWS Cost and Usage sync should be run, and to which destination databases.
Last updated
Was this helpful?