Setting up a GitHub Integration

The GitHub integration supports two authentication methods: Personal Access Token (PAT) and GitHub App authentication. Choose the method that best aligns with your organization's security requirements.

Personal Access Token Authentication

Keep in mind rate limits for GitHub Apps are higher than for personal access tokens. Review GitHub rate limits documentation for details

Generate token

Personal Access Tokens provide a simple way to authenticate with GitHub APIs. For detailed instructions, see the guide.

To authenticate using a Personal Access Token:

1. Go to your GitHub account settings

2. Navigate to Developer settings > Personal access tokens > Tokens (classic)

3. Click Generate new token and select Generate new token (classic)

4. Name your token and set the read-only permissions to the resources you need

5. Save generated token

Create integration

1. In CloudQuery Platform, go to Data Pipelines → Integrations. Click Create Integration and type Github to find the Github integration.

1. Update the YAML to add an entry for github_personal_access_token:

spec:
  access_token: "${github_personal_access_token}"

1. Add a new secret with Key github_personal_access_token

GitHub App Authentication

Create GitHub App

For App authentication, you need to create a GitHub App and install it on your organization. GitHub App authentication provides higher rate limits and better security through fine-grained permissions. For detailed instructions, see the guide.

To set up:

1. Create a GitHub App in your organization:

   • Go to Organization Settings > GitHub Apps > New GitHub App

   • Configure the app with read-only permissions for the resources you need

   • Generate and download the private key

2. Install the app in your organization

3. Note down the following details:

   • App ID: Found in the app's settings

   • Installation ID: Found in the URL when viewing the app installation

   • Private key: The downloaded RSA key

Create Integration

1. In CloudQuery Platform, go to Data Pipelines → Integrations. Click Create Integration and type GitHub to find the GitHub integration.

2. Update the YAML to add entries:

    app_auth:
      - org: your-organization
        app_id: "your-app-id"
        installation_id: "your-installation-id"
        private_key: |
          -----BEGIN RSA PRIVATE KEY-----
          Your private key here
          -----END RSA PRIVATE KEY-----

1. Configure your GitHub App integration:

Configuration

To start using the GitHub source plugin, update a plugin configuration section. Here's a minimal example that extracts issues from the cloudquery/cloudquery repository. For complete configuration options, see the plugin docs.

kind: source
spec:
  # Source spec section
  name: github
  path: cloudquery/github
  registry: cloudquery
  version: "v13.3.4" # latest version of source github plugin
  tables: ["github_issues"]
  spec:
    access_token: "${GITHUB_PERSONAL_ACCESS_TOKEN}"
    repos: ["cloudquery/cloudquery"]

Next Steps

With your GitHub integration created, you can now proceed to use it in a new sync. This will give you the opportunity to specify when your GitHub sync should be run, and to which destination databases.