Setting up a GCP Integration

At the moment, CloudQuery Platform only supports authentication with GCP through Service Accounts. This document explains the steps in this process.

We are working on adding support for more authentication methods, such as OIDC and Workload Identity. Reach out to us if you have any questions.

Step 1: Set up a Service Account

CloudQuery will use a Service Account to read resources from your GCP environment. Follow these steps to set up a new Service Account with read-only access:

1. Open https://console.cloud.google.com/iam-admin/serviceaccounts

2. Select the project to create the service account in (we can assign access to other projects later)

3. Click Create Service Account

4. Enter the details:

   1. Service account display name, e.g. CloudQuery Readonly

   2. Service account ID, e.g. cloudquery-readonly

   3. A description to help you and others identify the purpose of this service account later, e.g. Service account for CloudQuery to fetch resources in GCP

   4. Click Create and Continue

1. Under Basic, Select Viewerrole for the service account.

1. Click Continue and Done.

2. You should now see the new service account in the list. Click on it, and go to the Keys tab. Click Add Key → Create New Key

3. Select JSON and click Create. This will download a file to your computer. You will need this when setting up the integration later.

Optional: Assign Organization or folder-wide access to the Service Account

To sync resources across all our GCP projects, we can grant the required access to the service account we just created. Depending on your case, you may want do this on the organization-level, or on the folder level.

1. In the Console Project selection screen, select your top-level Organization (or folder)

2. Go to IAM and Admin / IAM, and click Grant Access

3. Paste the email address of the service account we created above in the New Principals textbox. Again, assign a Viewer role.

4. Click Save

Optional: Assign more projects to the Service Account

Similar to the process for Organizations and folders described above, you can also follow the same steps to add individual projects for CloudQuery to sync, if desired. This is not required if you already followed the steps for organizations or folders above.

• In the Console Project selection screen, select the relevant project

• Go to IAM and Admin / IAM, and click Grant Access

• Paste the email address of the service account we created above in the New Principals textbox. Again, assign a Viewer role.

• Click Save

Step 2: Create Integration

1. In CloudQuery Platform, go to Data Pipelines → Integrations. Click Create Integration and type GCP to find the GCP integration.

1. Choose a name for your integration (e.g. GCP) and update the YAML to add an entry for service_account_key_json:

       service_account_key_json: |
         ${service_account_key_json}

The presence of the pipe symbol followed by a newline in the the YAML is required: without it, YAML parsing will not work as expected.

1. Add a new secret with Key service_account_key_json

2. In a text editor, open the JSON file you downloaded from GCP in Step 1, and copy-paste the contents into the Value field:

1. You may want to make further adjustments to your YAML file, according to your requirements. For more information, see the GCP Integration Documentation

2. Click Test Connection

After a successful test connection, you can now safely delete the JSON file from your local disk.

Next Steps

With your GCP integration created, you can now proceed to use it in a new sync. This will give you the opportunity to specify when your GCP sync should be run, and to which destination databases.