CloudQuery Platform
  • Introduction
    • Welcome to CloudQuery Platform
    • Getting Help
  • Quickstart
    • Creating a New Account
    • Platform Activation
  • Core Concepts
    • Integrations
    • Syncs
    • Filters & Queries
    • SQL Console
    • Reports
  • Integration Guides
    • Setting up an AWS Integration
    • Setting up an AWS Cost and Usage Integration
    • Setting up a GCP Integration
    • Setting up an Azure Integration
    • Setting up a GitHub Integration
    • Setting up a K8s Integration
      • Using AWS EKS
      • Using Azure AKS
      • Using GCP GKE
    • General Integration Setup Guide
    • General Destination Setup Guide
  • Syncs
    • Setting up a Sync
    • Monitoring Sync Status
  • Cloud insights
    • From cloud asset inventory to insights
      • Security-focused queries
      • Compliance-focused queries
      • FinOps-focused queries
  • Production Deployment
    • Enabling Single Sign-on (SSO)
      • Single Sign-On with Google
      • Single Sign-On with Microsoft
      • Single Sign-On with Okta
  • User Management
    • Platform Roles Overview
    • Workspace Roles Overview
  • Advanced Topics
    • Custom Columns
    • Understanding Platform Views
    • Performance Tuning
  • Reference
    • Search & Filter Query Syntax
  • API Reference
  • CLI Docs
  • CloudQuery Hub
Powered by GitBook
On this page
  • Cloud Assets View
  • Table Views

Was this helpful?

  1. Advanced Topics

Understanding Platform Views

Learn how the normalization layer of CloudQuery Platform works underneath the hood.

PreviousCustom ColumnsNextPerformance Tuning

Last updated 3 months ago

Was this helpful?

Cloud Assets View

When a sync runs on CloudQuery Platform, the sync is configured (through use of a transformer) to prefix all output tables with `raw_`. As the sync progresses, a post-load transformer detects when tables have completed syncing, and starts copying the latest data to two tables: cloud_assets_historicaland cloud_assets_incremental.

cloud_assets_historicalstores data for all non-incremental tables. It uses a ClickHouse ORDER BY clause that allows stale records to be removed after a period of time. It does this through incorporation of _cq_sync_group_idinto the clause.

cloud_assets_incremental stores dat for all incremental tables. As incremental tables can only add new data, the ORDER BY clause does not include _cq_sync_group_id.

Finally a view, cloud_assets, creates a single unified view over these two underlying tables, creating a cross-cloud asset inventory. The view defines how to identify the latest snapshot, and ensures de-duplication through the addition of a FINAL clause to all queries. The view is updated only once a table from a _cq_sync_group_idis complete, guaranteeing consistency on a per-table level.

Table Views

Table views are less complicated. As alluded to above, syncs first write to tables prefixed with raw. A view, with the original table name, is created for every table. This view is defined in a way that it always points to the latest complete snapshot of the data. This ensures data stay consistent during syncs, and switches atomically, while still allowing records to be appended efficiently into ClickHouse.

Diagram illustrating how the cloud assets view is built.
Diagram illustrating how table views are defined.