OpenTelemetry and Grafana
In this example we will show how to visualize CloudQuery OpenTelemetry traces, metrics and logs with Grafana. We will use Docker Compose to run Grafana and related services, so make sure you it installed on your machine.
Step 1: Creating a Docker Compose file
We will use Tempo for ingesting traces, Loki for logs, Prometheus for metrics, and the OpenTelemetry collector for collecting and forwarding the data to each service.
Create a file named docker-compose.yml
with the following content:
version: "3.8"
services:
tempo:
image: grafana/tempo:latest
command: ["-config.file=/etc/tempo.yaml"]
volumes:
- tempo_data:/tmp
- ./tempo/tempo.yaml:/etc/tempo.yaml
ports:
- "3200"
- "4318"
loki:
image: grafana/loki:latest
ports:
- "3100"
command: -config.file=/etc/loki/local-config.yaml
collector:
image: otel/opentelemetry-collector-contrib:latest
ports:
- "4318:4318" # 4318 needs to be exposed to the host for the collector to ingest data
- "8090"
volumes:
- ./collector/collector.yaml:/etc/otelcol-contrib/config.yaml
prometheus:
image: prom/prometheus:latest
command:
- "--enable-feature=remote-write-receiver"
- "--config.file=/etc/prometheus/prometheus.yaml"
ports:
- "9090"
volumes:
- prometheus:/prometheus
- ./prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml
grafana:
image: grafana/grafana-enterprise
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
- ./grafana/dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml
- ./grafana/cloudquery-dashboard.json:/var/lib/grafana/dashboards/cloudquery-dashboard.json
environment:
GF_FEATURE_TOGGLES_ENABLE: "tempoApmTable"
ports:
- "3000:3000" # 3000 needs to be exposed to the host for the Grafana UI
volumes:
prometheus:
driver: local
grafana_data:
driver: local
tempo_data:
driver: local
This Docker Compose file configures Prometheus, Tempo, an OpenTelemetry collector and Grafana with a custom configuration, and Loki with the default configuration.
Step 2: Configure Prometheus
Create a file with the path prometheus/prometheus.yaml
with the following content:
global:
scrape_interval: 15s
scrape_configs:
- job_name: "opentelemetry"
static_configs:
- targets: ["collector:8090"]
This configuration will tell Prometheus to scrape the OpenTelemetry collector every 15 seconds.
Step 3: Configure Tempo
Create a file with the path tempo/tempo.yaml
with the following content:
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
http:
storage:
trace:
backend: local
wal:
path: /tmp/tempo/wal
local:
path: /tmp/tempo/blocks
# Needed for aggregation functions, e.g. quantile_over_time
# Visit https://grafana.com/docs/tempo/latest/traceql/metrics-queries/ for more information
query_frontend:
search:
max_duration: 0
metrics:
max_duration: 0
overrides:
metrics_generator_processors: ["local-blocks"]
metrics_generator:
processor:
local_blocks:
filter_server_spans: false
storage:
path: /var/tempo/generator/wal
traces_storage:
path: /var/tempo/generator/traces
This configuration will tell Tempo to listen on port 3200 and receive OpenTelemetry traces via HTTP on the default port of 4318.
Step 4: Configure the OpenTelemetry collector
Create a file with the path collector/collector.yaml
with the following content:
receivers:
otlp:
protocols:
http:
endpoint: "0.0.0.0:4318"
processors:
batch:
exporters:
prometheus:
endpoint: collector:8090
otlphttp:
endpoint: http://tempo:4318
loki:
endpoint: http://loki:3100/loki/api/v1/push
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]
logs:
receivers: [otlp]
exporters: [loki]
This configuration will tell the OpenTelemetry collector to receive traces, metrics, and logs and forward them to Tempo, Prometheus and Loki, respectively.
Step 5: Configure Grafana Data Sources
Create a file with the path grafana/datasources.yaml
with the following content:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
orgId: 1
url: http://prometheus:9090
basicAuth: false
isDefault: false
version: 1
editable: true
uid: prometheus
- name: Loki
type: loki
access: proxy
orgId: 1
url: http://loki:3100
basicAuth: false
isDefault: false
version: 1
editable: true
uid: loki
- name: Tempo
type: tempo
access: proxy
orgId: 1
url: http://tempo:3200
basicAuth: false
isDefault: true
version: 1
editable: true
apiVersion: 1
uid: tempo
This configuration will tell Grafana to use Prometheus, Loki, and Tempo as data sources.
Step 6: Download the CloudQuery Grafana Dashboard
Create a file with the path grafana/cloudquery-dashboard.json
with the content from here.
If you’d like to import the dashboard to an existing Grafana instance, you can download an external version of it from here.
Step 7: Configure Grafana with the CloudQuery Dashboard
Create a file with the path grafana/dashboards.yaml
with the following content:
apiVersion: 1
providers:
- name: CloudQuery
folder: CloudQuery
type: file
allowUiUpdates: true
options:
path: /var/lib/grafana/dashboards
This configuration will tell Grafana to load the CloudQuery dashboard.
Step 8: Start the services
Run docker-compose up
to start the services. Once the services are up and running, you should be able to access Grafana at http://localhost:3000 with the default credentials admin:admin
.
Step 9: Configure a Source Plugin with OpenTelemetry
You can use the example source configuration below to start a sync with OpenTelemetry enabled:
kind: source
spec:
name: "aws"
path: "cloudquery/aws"
registry: "cloudquery"
version: "v32.40.1"
tables: ["aws_s3_buckets"]
destinations: ["postgresql"]
otel_endpoint: "0.0.0.0:4318"
otel_endpoint_insecure: true
spec:
Step 10: Run the sync
Run cloudquery sync spec.yml --log-level debug
.
import { Callout } from ‘nextra/components’;
After ingestion starts, you can access the dashboard to see sync insights, traces, metrics, and logs.