Setting up a GCP Integration

At the moment, CloudQuery Platform only supports authentication with GCP through Service Accounts. This document explains the steps in this process.

We are working on adding support for more authentication methods, such as OIDC and Workload Identity. Reach out to us if you have any questions.

Step 1: Enable Cloud Resource Manager API

Head to the Google Cloud Console and enable the Cloud Resource Manager API:

On the top, make sure you select the project you want to grant CloudQuery access to. Then, click Next.

Click Enable.

Step 2: Authorize CloudQuery

CloudQuery will use a Service Account to read resources from your GCP environment. Follow these steps to set up a new Service Account with read-only access:

1. Open https://console.cloud.google.com/iam-admin/serviceaccounts

2. Select the project to create the service account in (we can assign access to other projects later)

3. Click Create Service Account

4. Enter the details:

   1. Service account display name, e.g. CloudQuery Readonly

   2. Service account ID, e.g. cloudquery-readonly

   3. A description to help you and others identify the purpose of this service account later, e.g. Service account for CloudQuery to fetch resources in GCP

   4. Click Create and Continue

1. Under Basic, Select Viewerrole for the service account.

1. Click Continue and Done.

2. You should now see the new service account in the list. Click on it, and go to the Keys tab. Click Add Key → Create New Key

3. Select JSON and click Create. This will download a file to your computer. You will need this when setting up the integration later.

Optional: Assign Organization or folder-wide access to the Service Account

To sync resources across all our GCP projects, we can grant the required access to the service account we just created. Depending on your case, you may want do this on the organization-level, or on the folder level.

1. In the Console Project selection screen, select your top-level Organization (or folder)

2. Go to IAM and Admin / IAM, and click Grant Access

3. Paste the email address of the service account we created above in the New Principals textbox. Again, assign a Viewer role.

4. Click Save

Optional: Assign more projects to the Service Account

Similar to the process for Organizations and folders described above, you can also follow the same steps to add individual projects for CloudQuery to sync, if desired. This is not required if you already followed the steps for organizations or folders above.

• In the Console Project selection screen, select the relevant project

• Go to IAM and Admin / IAM, and click Grant Access

• Paste the email address of the service account we created above in the New Principals textbox. Again, assign a Viewer role.

• Click Save

Step 3: Create the Integration

1. In CloudQuery Platform, go to Data Pipelines → Integrations. Click Create Integration and type GCP to find the GCP integration.

1. Choose a name for your integration (e.g. GCP).

2. Click the Upload JSON file or drag the credentials JSON file created in Step 2 in the designated space.

1. Click Continue

2. On the next page, select the services to sync. The services represent a group of tables the GCP integration can sync. For the full list of tables, see the GCP Plugin Documentation.

If you are looking for a particular table, use the search above the list of services. You can also expand individual services to select/unselect indivudual tables using this toggle button (visible when you move your mouse pointer over the service name):

1. Click the Test and Save to test the connection and save the integration.

After a successful test connection, you can now safely delete the JSON file from your local disk.

Next Steps

With your GCP integration created, you can now proceed to use it in a new sync. This will give you the opportunity to specify when your GCP sync should be run, and to which destination databases.