CloudQuery Platform
  • Introduction
    • Welcome to CloudQuery Platform
    • Getting Help
  • Quickstart
    • Creating a New Account
    • Platform Activation
  • Core Concepts
    • Integrations
    • Syncs
    • Filters & Queries
    • SQL Console
    • Reports
      • Built-in Report Templates
      • Reports Yaml Documentation with Examples
        • Full Report Example
  • Integration Guides
    • Setting up an AWS Integration
    • Setting up an AWS Cost and Usage Integration
    • Setting up a GCP Integration
    • Setting up an Azure Integration
    • Setting up a GitHub Integration
    • Setting up a K8s Integration
      • Using AWS EKS
      • Using Azure AKS
      • Using GCP GKE
    • General Integration Setup Guide
    • General Destination Setup Guide
  • Syncs
    • Setting up a Sync
    • Monitoring Sync Status
  • Cloud insights
    • From cloud asset inventory to insights
      • Security-focused queries
      • Compliance-focused queries
      • FinOps-focused queries
  • Production Deployment
    • Enabling Single Sign-on (SSO)
      • Single Sign-On with Google
      • Single Sign-On with Microsoft
      • Single Sign-On with Okta
  • User Management
    • Platform Roles Overview
    • Workspace Roles Overview
  • Advanced Topics
    • Custom Columns
    • Understanding Platform Views
    • Performance Tuning
  • Reference
    • Search & Filter Query Syntax
  • API Reference
  • CLI Docs
  • CloudQuery Hub
Powered by GitBook
On this page
  • Prerequisites
  • IAM Role And Permissions
  • Creating AWS Cost and Usage Integration
  • Next Steps

Was this helpful?

  1. Integration Guides

Setting up an AWS Cost and Usage Integration

PreviousSetting up an AWS IntegrationNextSetting up a GCP Integration

Last updated 1 month ago

Was this helpful?

CloudQuery Platform supports authentication with an S3 bucket that has Cost and Usage reports data through (IRSA).

The AWS accounts involved are:

  • CloudQuery Account: This is the AWS account where CloudQuery Platform is deployed. This account hosts the IAM role that CloudQuery uses to assume roles in other accounts.

  • Your Account: This is the AWS account that you want to sync resources from. This account will have a role that allows the CloudQuery account's role to assume and read resources.

Prerequisites

Before starting, configure the following environment variables - the account ID (this will be provided by the CloudQuery team) and the sub-domain of your installation.

export CLOUDQUERY_ACCOUNT_ID="<CloudQuery AWS account>"
export SUB_DOMAIN="<your installation subdomain>"
export S3_COST_REPORTS_BUCKET="<your bucket with the cost reports>"

An external ID should be added as recommended by to provide an additional verification layer when assuming roles in a third-party account. This can be any alphanumeric string between 2 and 1224 characters, but in this example we use a UUID.

export EXTERNAL_ID=$(uuidgen)

IAM Role And Permissions

  1. Create the trust relationship for the cross-account role:

cat >third-party-trust.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::${CLOUDQUERY_ACCOUNT_ID}:role/${SUB_DOMAIN}-cloudquery-sync"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "${EXTERNAL_ID}"
                }
            }
        }
    ]
}
EOF
  1. Create a policy to allow reading the Cost and Usage reports from your bucket

cat >bucket-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::${S3_COST_REPORTS_BUCKET}"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::${S3_COST_REPORTS_BUCKET}/*"
        }
    ]
}
EOF
  1. Create the cross-account role and attach the ReadOnly policy:

aws iam create-role --role-name cross-account-readonly-role-cost-usage \
    --assume-role-policy-document file://third-party-trust.json

aws iam put-role-policy --role-name cross-account-readonly-role-cost-usage \
   --policy-name ReadS3Policy --policy-document file://bucket-policy.json

Continue to Creating AWS Cost and Usage Integration

Creating AWS Cost and Usage Integration

  1. Navigate to Data Pipelines → Integrations in CloudQuery Platform

  2. Click Create Integration and select AWS Cost and Usage.

  3. Update the YAML configuration to sync to either a single account or multiple accounts e.g.

kind: source
spec:
  name: awscur
  path: cloudquery/awscur
  version: v1.0.0 # latest version of source awscur plugin
  tables:
    - *
  spec: 
    bucket: '<bucket>'
    region: '<region>'
    reports:
      - path: '<path-to-parquet-files>/'
    role_to_assume:
      arn: arn:aws:iam::<your_account>:role/cross-account-readonly-role-cost-usage
      external_id: <external_id>
        

Note: the arn should be in the following form and correspond to the cross-account-readonly-role-cost-usage created in your AWS account: arn:aws:iam::<your_account>:role/cross-account-readonly-role-cost-usage

  1. Click Test Connection to verify the setup

Next Steps

With your AWS Cost and Usage integration created, you can now proceed to use it in a . This will give you the opportunity to specify when your AWS Cost and Usage sync should be run, and to which destination databases.

IAM Roles for Service Account
AWS best practices
new sync