OpenTelemetry and Grafana
In this example we will show how to visualize CloudQuery (opens in a new tab) OpenTelemetry (opens in a new tab) traces, metrics and logs with Grafana (opens in a new tab). We will use Docker Compose (opens in a new tab) to run Grafana (opens in a new tab) and related services, so make sure you it installed on your machine.
Step 1: Creating a Docker Compose file
We will use Tempo (opens in a new tab) for ingesting traces, Loki (opens in a new tab) for logs, Prometheus (opens in a new tab) for metrics, and the OpenTelemetry collector (opens in a new tab) for collecting and forwarding the data to each service.
Create a file named docker-compose.yml
with the following content:
version: "3.8"
services:
tempo:
image: grafana/tempo:latest
command: ["-config.file=/etc/tempo.yaml"]
volumes:
- tempo_data:/tmp
- ./tempo/tempo.yaml:/etc/tempo.yaml
ports:
- "3200"
- "4318"
loki:
image: grafana/loki:latest
ports:
- "3100"
command: -config.file=/etc/loki/local-config.yaml
collector:
image: otel/opentelemetry-collector-contrib:latest
ports:
- "4318:4318" # 4318 needs to be exposed to the host for the collector to ingest data
- "8090"
volumes:
- ./collector/collector.yaml:/etc/otelcol-contrib/config.yaml
prometheus:
image: prom/prometheus:latest
command:
- "--enable-feature=remote-write-receiver"
- "--config.file=/etc/prometheus/prometheus.yaml"
ports:
- "9090"
volumes:
- prometheus:/prometheus
- ./prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml
grafana:
image: grafana/grafana-enterprise
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
- ./grafana/dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml
- ./grafana/cloudquery-dashboard.json:/var/lib/grafana/dashboards/cloudquery-dashboard.json
environment:
GF_FEATURE_TOGGLES_ENABLE: "tempoApmTable"
ports:
- "3000:3000" # 3000 needs to be exposed to the host for the Grafana UI
volumes:
prometheus:
driver: local
grafana_data:
driver: local
tempo_data:
driver: local
This Docker Compose file configures Prometheus (opens in a new tab), Tempo (opens in a new tab), an OpenTelemetry collector (opens in a new tab) and Grafana with a custom configuration, and Loki (opens in a new tab) with the default configuration.
Step 2: Configure Prometheus
Create a file with the path prometheus/prometheus.yaml
with the following content:
global:
scrape_interval: 15s
scrape_configs:
- job_name: "opentelemetry"
static_configs:
- targets: ["collector:8090"]
This configuration will tell Prometheus (opens in a new tab) to scrape the OpenTelemetry (opens in a new tab) collector every 15 seconds.
Step 3: Configure Tempo
Create a file with the path tempo/tempo.yaml
with the following content:
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
http:
storage:
trace:
backend: local
wal:
path: /tmp/tempo/wal
local:
path: /tmp/tempo/blocks
# Needed for aggregation functions, e.g. quantile_over_time
# Visit https://grafana.com/docs/tempo/latest/traceql/metrics-queries/ for more information
query_frontend:
search:
max_duration: 0
metrics:
max_duration: 0
overrides:
metrics_generator_processors: ["local-blocks"]
metrics_generator:
processor:
local_blocks:
filter_server_spans: false
storage:
path: /var/tempo/generator/wal
traces_storage:
path: /var/tempo/generator/traces
This configuration will tell Tempo (opens in a new tab) to listen on port 3200 and receive OpenTelemetry (opens in a new tab) traces via HTTP on the default port of 4318.
Step 4: Configure the OpenTelemetry collector
Create a file with the path collector/collector.yaml
with the following content:
receivers:
otlp:
protocols:
http:
endpoint: "0.0.0.0:4318"
processors:
batch:
exporters:
prometheus:
endpoint: collector:8090
otlphttp:
endpoint: http://tempo:4318
loki:
endpoint: http://loki:3100/loki/api/v1/push
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]
logs:
receivers: [otlp]
exporters: [loki]
This configuration will tell the OpenTelemetry collector (opens in a new tab) to receive traces, metrics, and logs and forward them to Tempo (opens in a new tab), Prometheus (opens in a new tab) and Loki (opens in a new tab), respectively.
Step 5: Configure Grafana Data Sources
Create a file with the path grafana/datasources.yaml
with the following content:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
orgId: 1
url: http://prometheus:9090
basicAuth: false
isDefault: false
version: 1
editable: true
uid: prometheus
- name: Loki
type: loki
access: proxy
orgId: 1
url: http://loki:3100
basicAuth: false
isDefault: false
version: 1
editable: true
uid: loki
- name: Tempo
type: tempo
access: proxy
orgId: 1
url: http://tempo:3200
basicAuth: false
isDefault: true
version: 1
editable: true
apiVersion: 1
uid: tempo
This configuration will tell Grafana (opens in a new tab) to use Prometheus (opens in a new tab), Loki (opens in a new tab), and Tempo (opens in a new tab) as data sources.
Step 6: Download the CloudQuery Grafana Dashboard
Create a file with the path grafana/cloudquery-dashboard.json
with the content from here.
If you'd like to import the dashboard to an existing Grafana instance, you can download an external version of it from here.
Step 7: Configure Grafana with the CloudQuery Dashboard
Create a file with the path grafana/dashboards.yaml
with the following content:
apiVersion: 1
providers:
- name: CloudQuery
folder: CloudQuery
type: file
allowUiUpdates: true
options:
path: /var/lib/grafana/dashboards
This configuration will tell Grafana (opens in a new tab) to load the CloudQuery (opens in a new tab) dashboard.
Step 8: Start the services
Run docker-compose up
to start the services. Once the services are up and running, you should be able to access Grafana (opens in a new tab) at http://localhost:3000 (opens in a new tab) with the default credentials admin:admin
.
Step 9: Configure a Source Plugin with OpenTelemetry
You can use the example source configuration below to start a sync with OpenTelemetry (opens in a new tab) enabled:
kind: source
spec:
name: "aws"
path: "cloudquery/aws"
registry: "cloudquery"
version: "v30.0.0"
tables: ["aws_s3_buckets"]
destinations: ["postgresql"]
otel_endpoint: "0.0.0.0:4318"
otel_endpoint_insecure: true
spec:
Step 10: Run the sync
Run cloudquery sync spec.yml --log-level debug
.
Running with --log-level debug
is recommended to get more detailed logs about requests retries and errors.
After ingestion starts, you can access the dashboard (opens in a new tab) to see sync insights, traces, metrics, and logs.