From 092ccd750d853498547e5a0e49280c0f657b7160 Mon Sep 17 00:00:00 2001 From: Arnob kumar saha Date: Wed, 15 Jul 2026 18:42:47 +0600 Subject: [PATCH 1/2] docs(platform): add telemetry data verification guide Signed-off-by: Arnob kumar saha --- .../telemetry-data-check.md | 218 ++++++++++++++++++ 1 file changed, 218 insertions(+) create mode 100644 docs/platform/guides/cluster-management/telemetry-data-check.md diff --git a/docs/platform/guides/cluster-management/telemetry-data-check.md b/docs/platform/guides/cluster-management/telemetry-data-check.md new file mode 100644 index 0000000..e58e723 --- /dev/null +++ b/docs/platform/guides/cluster-management/telemetry-data-check.md @@ -0,0 +1,218 @@ +--- +layout: docs +menu: + docsplatform_{{.version}}: + identifier: cluster-management-telemetry-data-check + name: Verifying Telemetry Data + parent: cluster-management + weight: 100 +menu_name: docsplatform_{{.version}} +section_menu_id: guides +--- + +# Verifying Telemetry Data (Logs & Metrics) + +This guide shows how to confirm your logs and metrics are actually being stored — not just +collected — in the monitoring stack: **ClickHouse** for logs, **Thanos + S3** for metrics. +See [OpenTelemetry Monitoring](otel-monitoring.md) for how the stack is set up. + +## Basics + +- Logs and metrics are stored in the **same S3 bucket**, under different prefixes: + - Logs → `/logs/` + - Metrics → `/metrics/` +- Bucket, endpoint, and prefixes are defined in your `TelemetryStack` resource: + ```bash + kubectl get telemetrystack -n monitoring -o yaml + ``` + Look under `spec.logs.clickhouse.s3` and `spec.metrics.thanos.s3`. +- ClickHouse database name is your tenant ID, not a fixed name like `otel` or `default`. + Find it with `SHOW DATABASES` (see below) — the table inside is `otel_logs`. +- All checks below run from inside the cluster via `kubectl exec`, so no port-forwarding + or extra credentials are required except for the optional S3 listing step. + +--- + +## 1. Check logs are landing in ClickHouse + +Find the ClickHouse pod: + +```bash +kubectl get pods -n monitoring -l app.kubernetes.io/name=clickhouse +``` + +List databases and confirm the `otel_logs` table exists: + +```bash +kubectl exec -n monitoring -- clickhouse-client --query "SHOW DATABASES" +kubectl exec -n monitoring -- clickhouse-client --query "SHOW TABLES FROM " +``` + +Check row count and freshness: + +```bash +kubectl exec -n monitoring -- clickhouse-client --query \ + "SELECT count(), min(Timestamp), max(Timestamp), now() FROM .otel_logs" + +kubectl exec -n monitoring -- clickhouse-client --query \ + "SELECT count() FROM .otel_logs WHERE Timestamp > now() - INTERVAL 5 MINUTE" +``` + +**Healthy result:** `max(Timestamp)` is close to `now()`, and the 5-minute count is greater +than zero. If both queries return recent data, logs are flowing end-to-end. + +--- + +## 2. Check metrics are landing in Thanos / S3 + +Find the Thanos query pod: + +```bash +kubectl get pods -n monitoring -l app.kubernetes.io/name=thanos-query +``` + +Ask it which StoreAPIs it's talking to and what time range each one covers: + +```bash +kubectl exec -n monitoring -- \ + wget -qO- --header="THANOS-TENANT: " "http://localhost:9090/api/v1/stores" +``` + +Look at the `minTime`/`maxTime` fields (Unix epoch **milliseconds**) for each store: + +```bash +date -u -d @ # drop the last 3 digits first to convert ms -> s +``` + +**Healthy result:** `maxTime` should be within the last few minutes for the live/receive +store, and within your compaction interval for the long-term S3-backed store. + +For a second confirmation, check the ingester's own TSDB head timestamp: + +```bash +kubectl get pods -n monitoring -l app.kubernetes.io/name=thanos-receive-ingester +kubectl exec -n monitoring -- wget -qO- http://localhost:10902/metrics \ + | grep prometheus_tsdb_head_max_time +``` + +If this timestamp is stale (not advancing over successive checks), metrics are not being +ingested. Check that Prometheus (or your agent) has `remoteWrite` configured to point at +the Thanos receive router: + +```bash +kubectl get prometheus -n monitoring -o jsonpath='{.spec.remoteWrite}' +``` + +An empty result means nothing is shipping samples to Thanos, regardless of whether +Prometheus itself is scraping normally. + +--- + +## 3. Check the S3 bucket directly (optional) + +Read the bucket, endpoint, and credentials from the `TelemetryStack` spec first — the same +values back both the ClickHouse and Thanos configs: + +```bash +kubectl get telemetrystack -n monitoring -o jsonpath='{.spec.metrics.thanos.s3}' +``` + +Which client you use depends on where the bucket actually lives. The backend is +S3-compatible in every case, so pick the matching variant below. + +### a. MinIO (in-cluster or S3-compatible with a custom endpoint) + +Use the MinIO client (`mc`) with the endpoint and access credentials from the spec. +The `--insecure` flag is only needed when the endpoint serves a self-signed certificate: + +```bash +kubectl run mc-check -n monitoring --rm -it --restart=Never --image=minio/mc:latest -- sh +mc alias set myminio --insecure +mc ls myminio//logs/ --insecure --recursive | tail +mc ls myminio//metrics/ --insecure --recursive | tail +``` + +### b. Real AWS S3 + +When the bucket is a genuine AWS S3 bucket, there is no custom endpoint — just pass the +AWS access key, secret, and region. Drop `--endpoint-url` entirely so the CLI talks to +the regional AWS endpoint: + +```bash +kubectl run aws-check -n monitoring --rm -it --restart=Never --image=amazon/aws-cli:latest \ + --env AWS_ACCESS_KEY_ID= \ + --env AWS_SECRET_ACCESS_KEY= \ + --env AWS_DEFAULT_REGION= \ + -- s3 ls s3:///logs/ --recursive | tail +# repeat for the metrics prefix +kubectl run aws-check -n monitoring --rm -it --restart=Never --image=amazon/aws-cli:latest \ + --env AWS_ACCESS_KEY_ID= \ + --env AWS_SECRET_ACCESS_KEY= \ + --env AWS_DEFAULT_REGION= \ + -- s3 ls s3:///metrics/ --recursive | tail +``` + +### c. s3grid (S3-compatible service with a custom endpoint) + +s3grid is S3-compatible but reached through its own endpoint, so use the AWS CLI with +`--endpoint-url` pointing at the s3grid endpoint from the spec: + +```bash +kubectl run s3grid-check -n monitoring --rm -it --restart=Never --image=amazon/aws-cli:latest \ + --env AWS_ACCESS_KEY_ID= \ + --env AWS_SECRET_ACCESS_KEY= \ + --env AWS_DEFAULT_REGION= \ + -- s3 ls s3:///logs/ --endpoint-url --recursive | tail +# repeat for the metrics prefix +kubectl run s3grid-check -n monitoring --rm -it --restart=Never --image=amazon/aws-cli:latest \ + --env AWS_ACCESS_KEY_ID= \ + --env AWS_SECRET_ACCESS_KEY= \ + --env AWS_DEFAULT_REGION= \ + -- s3 ls s3:///metrics/ --endpoint-url --recursive | tail +``` + +> The same `mc`-based flow from (a) works against s3grid too — `mc alias set` accepts any +> S3-compatible endpoint. Use whichever client you already have handy. + +**Healthy result (any variant):** recent objects under both prefixes, with timestamps +matching your retention/compaction schedule. + +--- + +## Summary + +| Pipeline | Where to look | Healthy signal | +|---|---|---| +| Logs | ClickHouse `.otel_logs` | `max(Timestamp)` ≈ now, non-zero rows in last 5 min | +| Metrics | Thanos Query `/api/v1/stores`, receive ingester `/metrics` | `maxTime`/head timestamp advancing, close to now | +| S3 | `/logs/` and `/metrics/` prefixes | Recent objects under both prefixes | + +--- + +## Troubleshooting: metrics stale but logs healthy + +Logs and metrics are independent paths through the same `TelemetryStack` — one being +healthy says nothing about the other, so always run both checks above rather than +assuming one implies the other. + +If the metrics checks show a stale `maxTime`/head timestamp that isn't advancing, the +most common cause is that nothing is configured to remote-write samples into Thanos. +Check the `Prometheus` CR's remote-write config: + +```bash +kubectl get prometheus -n monitoring -o jsonpath='{.spec.remoteWrite}' +``` + +An empty result means Prometheus may be scraping fine locally, but nothing is shipping +samples to the Thanos receive router. Also check for a `PrometheusAgent` resource that +might be expected to do remote-write instead: + +```bash +kubectl get prometheusagents -A +``` + +**Fix:** add `spec.remoteWrite` to the `Prometheus` CR, pointing at the Thanos receive +router's remote-write endpoint (`http://thanos-receive-router-.monitoring.svc:19291/api/v1/receive`) +with the tenant header the router's hashring expects (e.g. `THANOS-TENANT: `). +After applying, re-run the `prometheus_tsdb_head_max_time` check above over a couple of +minutes to confirm the timestamp starts advancing. From 93dbf26effff4d27704475aea0e856cd33fd0a4a Mon Sep 17 00:00:00 2001 From: Rokibul Hasan Date: Wed, 15 Jul 2026 21:21:34 +0600 Subject: [PATCH 2/2] Update Troubleshooting section Signed-off-by: Rokibul Hasan --- .../telemetry-data-check.md | 139 ++++++++++++++---- 1 file changed, 114 insertions(+), 25 deletions(-) diff --git a/docs/platform/guides/cluster-management/telemetry-data-check.md b/docs/platform/guides/cluster-management/telemetry-data-check.md index e58e723..d69e6b8 100644 --- a/docs/platform/guides/cluster-management/telemetry-data-check.md +++ b/docs/platform/guides/cluster-management/telemetry-data-check.md @@ -96,15 +96,9 @@ kubectl exec -n monitoring -- wget -qO- http://loc ``` If this timestamp is stale (not advancing over successive checks), metrics are not being -ingested. Check that Prometheus (or your agent) has `remoteWrite` configured to point at -the Thanos receive router: - -```bash -kubectl get prometheus -n monitoring -o jsonpath='{.spec.remoteWrite}' -``` - -An empty result means nothing is shipping samples to Thanos, regardless of whether -Prometheus itself is scraping normally. +ingested. Metrics are shipped from each connected cluster by the OpenTelemetry stack +(`appscode-otel-stack`). See the [Troubleshooting](#troubleshooting-metrics-stale-but-logs-healthy) +section below for how to trace that pipeline. --- @@ -191,28 +185,123 @@ matching your retention/compaction schedule. ## Troubleshooting: metrics stale but logs healthy -Logs and metrics are independent paths through the same `TelemetryStack` — one being -healthy says nothing about the other, so always run both checks above rather than -assuming one implies the other. +There is no Prometheus doing remote-write in this setup. Telemetry is shipped from each +connected cluster by the **`appscode-otel-stack`** chart (built on `opentelemetry-kube-stack`), +which runs two OpenTelemetry collectors in the `monitoring` namespace of the *connected* +cluster: -If the metrics checks show a stale `maxTime`/head timestamp that isn't advancing, the -most common cause is that nothing is configured to remote-write samples into Thanos. -Check the `Prometheus` CR's remote-write config: +- **Daemonset (agent) collector**: runs on every node. Its `prometheus` receiver scrapes + targets assigned by the **Target Allocator** (which discovers `ServiceMonitor`/`PodMonitor` + CRs), and its `filelog` receiver collects container logs. Everything is forwarded over + OTLP to the gateway collector. +- **Gateway collector** (deployment): receives OTLP and exports to the monitoring cluster + through `prom-label-proxy`: + - **Metrics** → `prometheusremotewrite` exporter → `https://:10001/api/v1/receive` + → Thanos receive. The `THANOS-TENANT` header is injected per batch by the + `headers_setter` extension. + - **Logs** → `clickhouse` exporter → the same `prom-label-proxy` endpoint → ClickHouse + (`otel_logs` table). + +The `` depends on how the platform is deployed: + +- **DNS mode**: the exporters point at the monitoring cluster's domain name. +- **IP mode**: the exporters use the fixed hostname `prom-label-proxy.monitoring.svc.cluster.local`, + which is mapped to the monitoring cluster's IP via a `hostAliases` entry on the gateway + collector pod (required for TLS certificate verification against a bare IP). + +Confirm the endpoint actually in use from the gateway collector's config: ```bash -kubectl get prometheus -n monitoring -o jsonpath='{.spec.remoteWrite}' +kubectl get opentelemetrycollectors -n monitoring -o yaml | grep -B2 -A2 'endpoint:' ``` -An empty result means Prometheus may be scraping fine locally, but nothing is shipping -samples to the Thanos receive router. Also check for a `PrometheusAgent` resource that -might be expected to do remote-write instead: +Logs and metrics share the daemon → gateway OTLP hop and the mTLS client cert +(`otel-client-cert`), but split into different exporters at the gateway. So if logs are +healthy and metrics are stale, connectivity and certs are fine; the problem is specific +to the metrics path: either **scraping** (daemon collector / Target Allocator) or the +**remote-write export** (gateway → Thanos). Run these checks **on the connected cluster**: + +### a. Collector pods are running ```bash -kubectl get prometheusagents -A +kubectl get pods -n monitoring -l otel-collector-type=daemonset +kubectl get pods -n monitoring -l otel-collector-type=deployment ``` -**Fix:** add `spec.remoteWrite` to the `Prometheus` CR, pointing at the Thanos receive -router's remote-write endpoint (`http://thanos-receive-router-.monitoring.svc:19291/api/v1/receive`) -with the tenant header the router's hashring expects (e.g. `THANOS-TENANT: `). -After applying, re-run the `prometheus_tsdb_head_max_time` check above over a couple of -minutes to confirm the timestamp starts advancing. +The daemonset collector must have one Running pod per node; a node without an agent pod +silently drops that node's scrape targets (allocation is per-node). + +### b. Gateway is exporting metrics without errors + +Check the gateway collector's own telemetry for sent vs. failed metric points. The +collector image is minimal (no shell or `wget` inside), so `kubectl exec` will not work; +port-forward its internal telemetry port (8888) and query it from your machine instead: + +```bash +kubectl port-forward -n monitoring 8888:8888 & +sleep 2 # give the port-forward a moment to establish +curl -s http://localhost:8888/metrics \ + | grep -E 'otelcol_exporter_(sent|send_failed|enqueue_failed).*metric_points' +``` + +These counters are cumulative since the pod started, so a single reading proves nothing: +a large `send_failed` value may just be leftover from a past incident (for example a +monitoring-cluster restart) while everything is healthy now. Run the `curl` again after +30 seconds or so and compare the two readings; only the counters that are still +*advancing* matter. There are three possible outcomes: + +1. **`sent_metric_points{exporter="prometheusremotewrite"}` is increasing and + `send_failed`/`enqueue_failed` stay flat.** The gateway is exporting successfully, so + this cluster's pipeline is healthy. The problem is downstream (prom-label-proxy or + Thanos receive on the monitoring cluster) or a tenant mismatch; continue with step (d). +2. **`send_failed_metric_points` is climbing.** Exports are failing. Read the gateway logs + for the actual `prometheusremotewrite` error (HTTP status codes, TLS failures) against + the `prom-label-proxy` endpoint: + + ```bash + kubectl logs -n monitoring --tail=100 | grep -iE "error|prometheusremotewrite" + ``` + + TLS or certificate errors point at the `otel-client-cert` secret (check it exists and + is not expired). Connection refused or timeout points at a wrong or unreachable ingest + endpoint. HTTP 4xx/5xx responses point at the prom-label-proxy / Thanos receive side + on the monitoring cluster. +3. **All counters are flat or zero.** The gateway has nothing to export, which means the + daemon collectors are not scraping anything. Continue with step (c). + +A growing `enqueue_failed_metric_points` means the exporter's sending queue is +overflowing because sends are too slow or failing, and those points are dropped; treat +it the same as outcome 2. + +Note the exporter retries for a bounded time (`max_elapsed_time: 120s`) and then drops the +batch, so persistent export errors mean permanent data loss, not delayed delivery. + +When finished, stop the background port-forward with `kill %1`. + +### c. Scrape targets exist and are being allocated + +If the gateway shows nothing to export (`sent` flat, no errors), the daemon collectors +aren't scraping anything. Check that the Target Allocator is running and that +`ServiceMonitor`/`PodMonitor` resources exist for your workloads: + +```bash +kubectl get pods -n monitoring -l app.kubernetes.io/component=opentelemetry-targetallocator +kubectl get servicemonitors,podmonitors -A +``` + +The Target Allocator watches these CRs and distributes their scrape targets across the +daemonset collectors. No matching monitors (or an unhealthy allocator) means the +`prometheus` receiver has nothing to scrape: no data flows and no errors are logged +anywhere, which is exactly the "silently stale" symptom. + +### d. Data flowing under the wrong tenant + +Metrics may be arriving in Thanos but under a different tenant than the one you're +querying. The gateway derives the tenant from resource attributes: it defaults to +`default`, and is set to the source namespace name only for namespaces labeled +`ace.appscode.com/client-org: "true"`; batches with no tenant attribute fall back to +`anonymous`. Re-run the store/query checks from section 2 with `THANOS-TENANT: default` +(and `anonymous`) before concluding data is missing. + +After fixing whichever stage was broken, re-run the `prometheus_tsdb_head_max_time` check +from section 2 over a couple of minutes to confirm the timestamp starts advancing.