Setting up an AWS Cost and Usage Integration

CloudQuery Platform supports authentication with an S3 bucket that has Cost and Usage reports data through IAM Roles for Service Account (IRSA).

The AWS accounts involved are:

CloudQuery Account: This is the AWS account where CloudQuery Platform is deployed. This account hosts the IAM role that CloudQuery uses to assume roles in other accounts.
Your Account: This is the AWS account that you want to sync resources from. This account will have a role that allows the CloudQuery account’s role to assume and read resources.

Prerequisites

Before starting, configure the following environment variables - the account ID (this will be provided by the CloudQuery team) and the sub-domain of your installation.

export CLOUDQUERY_ACCOUNT_ID="<CloudQuery AWS account>"
export SUB_DOMAIN="<your installation subdomain>"
export S3_COST_REPORTS_BUCKET="<your bucket with the cost reports>"

An external ID should be added as recommended by AWS best practices to provide an additional verification layer when assuming roles in a third-party account. This can be any alphanumeric string between 2 and 1224 characters, but in this example we use a UUID.

export EXTERNAL_ID=$(uuidgen)

IAM Role And Permissions

Create the trust relationship for the cross-account role:

cat >third-party-trust.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::${CLOUDQUERY_ACCOUNT_ID}:role/${SUB_DOMAIN}-cloudquery-sync"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "${EXTERNAL_ID}"
                }
            }
        }
    ]
}
EOF

Create a policy to allow reading the Cost and Usage reports from your bucket

cat >bucket-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::${S3_COST_REPORTS_BUCKET}"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::${S3_COST_REPORTS_BUCKET}/*"
        }
    ]
}
EOF

Create the cross-account role and attach the ReadOnly policy:

aws iam create-role --role-name cross-account-readonly-role-cost-usage \
    --assume-role-policy-document file://third-party-trust.json

aws iam put-role-policy --role-name cross-account-readonly-role-cost-usage \
   --policy-name ReadS3Policy --policy-document file://bucket-policy.json

Continue to #creating-aws-cost-and-usage-integration

Creating AWS Cost and Usage Integration

Navigate to Data Pipelines → Integrations in CloudQuery Platform
Click Create Integration and select AWS Cost and Usage.
Update the YAML configuration to sync to either a single account or multiple accounts e.g.

kind: source
spec:
  name: awscur
  path: cloudquery/awscur
  version: v1.0.0 # latest version of source awscur plugin
  tables:
    - *
  spec:
    bucket: '<bucket>'
    region: '<region>'
    reports:
      - path: '<path-to-parquet-files>/'
    role_to_assume:
      arn: arn:aws:iam::<your_account>:role/cross-account-readonly-role-cost-usage
      external_id: <external_id>

Note: the arn should be in the following form and correspond to the cross-account-readonly-role-cost-usage created in your AWS account:

arn:aws:iam::<your_account>:role/cross-account-readonly-role-cost-usage

Click Test Connection to verify the setup

Next Steps

With your AWS Cost and Usage integration created, you can now proceed to use it in a new sync. This will give you the opportunity to specify when your AWS Cost and Usage sync should be run, and to which destination databases.

AWS GCP