Setting up an AWS Cost and Usage Integration

CloudQuery Platform supports authentication with an S3 bucket that has Cost and Usage reports data through IAM Roles for Service Account (IRSA).

The AWS accounts involved are:

• CloudQuery Account: This is the AWS account where CloudQuery Platform is deployed. This account hosts the IAM role that CloudQuery uses to assume roles in other accounts.
• Your Account: This is the AWS account that you want to sync resources from. This account will have a role that allows the CloudQuery account’s role to assume and read resources.

Prerequisites

An external ID should be added as recommended by AWS best practices to provide an additional verification layer when assuming roles in a third-party account. This can be any alphanumeric string between 2 and 1224 characters, but in this example we use a UUID.

export EXTERNAL_ID=$(uuidgen)

IAM Role And Permissions

1. Create the trust relationship for the cross-account role:

Replace <TENANT_ID> with your tenant ID that you can find in the “Setup guide” section when configuring the integration. Replace <EXTERNAL_ID> with your generated EXTERNAL_ID.

cat >third-party-trust.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::686255977389:role/syncs-<TENANT_ID>-role"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "<EXTERNAL_ID>"
                }
            }
        }
    ]
}
EOF

2. Create a policy to allow reading the Cost and Usage reports from your bucket.

Replace <S3_COST_REPORTS_BUCKET> with your S3 bucket name.

cat >bucket-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::<S3_COST_REPORTS_BUCKET>"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::<S3_COST_REPORTS_BUCKET>/*"
        }
    ]
}
EOF

3. Create the cross-account role and attach the ReadOnly policy:

aws iam create-role --role-name cross-account-readonly-role-cost-usage \
    --assume-role-policy-document file://third-party-trust.json
 
aws iam put-role-policy --role-name cross-account-readonly-role-cost-usage \
   --policy-name ReadS3Policy --policy-document file://bucket-policy.json

Continue to #creating-aws-cost-and-usage-integration

Creating AWS Cost and Usage Integration

1. Navigate to Data Pipelines → Integrations in CloudQuery Platform
2. Click Create Integration and select AWS Cost and Usage.
3. Update the YAML configuration to sync to either a single account or multiple accounts e.g.

kind: source
spec:
  name: awscur
  path: cloudquery/awscur
  version: v1.0.0 # latest version of source awscur plugin
  tables:
    - *
  spec:
    bucket: '<bucket>'
    region: '<region>'
    reports:
      - path: '<path-to-parquet-files>/'
    role_to_assume:
      arn: arn:aws:iam::<your_account>:role/cross-account-readonly-role-cost-usage
      external_id: <external_id>
 

4. Click Test Connection to verify the setup

Next Steps

With your AWS Cost and Usage integration created, you can now proceed to use it in a new sync. This will give you the opportunity to specify when your AWS Cost and Usage sync should be run, and to which destination databases.