From 4fd16e18c12d68f438ad8c0dd183f4cdb1aa40cf Mon Sep 17 00:00:00 2001 From: Alex Ormenisan Date: Tue, 25 Aug 2026 15:20:52 +0200 Subject: [PATCH 1/2] [HWORKS-2912] Service logs are silently dropped from .services: unbounded Kubernetes pod and node labels exhaust the 1000-field mapping limit https://hopsworks.atlassian.net/browse/HWORKS-2912 Bounding the Kubernetes metadata that Filebeat attaches to service logs makes the kept label set part of the cluster's configuration surface, so it needs to be documented rather than read out of the chart. An operator who adds a routing label to their own workloads has no way to know from the outside which keys survive collection. Add a Service Log Labels page next to Services Logs, listing the pod, namespace and node labels kept by default and what each is for, and showing how to append with the extra lists, how to replace a base list, and how to opt into annotations and the deployment/cronjob names. Record the dedotting rule, since searching Dashboards for a dotted key returns nothing. The page warns about the four defaults that the collection itself depends on: replacing a base list without them stops Filebeat excluding its own logs, or stops project namespaces being collected, and neither failure reports an error. Signed-off-by: Alex Ormenisan Co-Authored-By: Claude Opus 5 (1M context) --- .../admin/monitoring/services-logs-labels.md | 152 ++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 153 insertions(+) create mode 100644 docs/setup_installation/admin/monitoring/services-logs-labels.md diff --git a/docs/setup_installation/admin/monitoring/services-logs-labels.md b/docs/setup_installation/admin/monitoring/services-logs-labels.md new file mode 100644 index 0000000000..b9e1602565 --- /dev/null +++ b/docs/setup_installation/admin/monitoring/services-logs-labels.md @@ -0,0 +1,152 @@ +# Service Log Labels + +## Introduction + +Filebeat attaches Kubernetes metadata to every service log document before Logstash forwards it to OpenSearch. +Only an explicit set of pod, namespace and node labels is kept. + +The set is bounded because every distinct label key becomes a field in the shared `.services-*` index mapping. +OpenSearch rejects documents once an index exceeds `index.mapping.total_fields.limit`, which defaults to 1000 fields. +A cluster that attaches every pod and node label, including the label sets that cloud providers and node feature discovery add, reaches that limit and then stops indexing service logs. + +In this guide you will learn which labels are kept by default, and how to append to or replace that set. + +## Default pod labels + +| Label | Purpose | +| --- | --- | +| `name` | Excludes Filebeat's own logs from collection. | +| `app` | Service identification, read by most pipelines. | +| `app.kubernetes.io/name` | Service identification for charts using the recommended Kubernetes labels. | +| `service` | Service name. | +| `component` | Component within a service. | +| `rondbService` | RonDB process type. | +| `user` | Owner of a job, notebook or serving instance. | +| `job-type` | Job type. | +| `job-id` | Job identifier. | +| `job-name` | Job name. | +| `execution` | Execution identifier of a job run. | +| `jupyter` | Marks a Jupyter pod. | +| `jupyter-id` | Jupyter instance identifier. | +| `jupyter-settings-id` | Jupyter settings identifier. | +| `kernel-id` | Jupyter kernel identifier. | +| `spark-role` | Driver or executor. | +| `spark-app-selector` | Spark application identifier. | +| `sparkoperator.k8s.io/launched-by-spark-operator` | Marks pods created by the Spark operator. | +| `serving.hops.works/id` | Deployment identifier. | +| `serving.hops.works/name` | Deployment name. | +| `serving.hops.works/tool` | Serving tool. | +| `serving.hops.works/model-name` | Model name. | +| `serving.hops.works/model-version` | Model version. | +| `serving.hops.works/model-server` | Model server. | +| `serving.hops.works/project-id` | Project that owns the deployment. | + +## Default namespace labels + +| Label | Purpose | +| --- | --- | +| `hopsworks.ai/project` | Marks a project namespace. | +| `hopsworks.ai/onlinefs-cluster` | Marks an online feature store namespace. | + +Filebeat collects logs from the release namespace, from namespaces carrying either of these two labels, and from any namespace listed in `olk.filebeat.extraNamespaces`. + +## Default node labels + +| Label | Purpose | +| --- | --- | +| `kubernetes.io/hostname` | Node name, read by the services, Spark, Python and serving pipelines. | + +## How labels appear in a log document + +```json +{ + "kubernetes": { + "labels": { "app": "namenode", "app_kubernetes_io/name": "hopsfs" }, + "namespace_labels": { "hopsworks_ai/project": "demo" }, + "node": { "labels": { "kubernetes_io/hostname": "worker-1" } } + } +} +``` + +Dots in a label key are replaced by underscores. +In OpenSearch Dashboards, search for `kubernetes.labels.app_kubernetes_io/name`, not `kubernetes.labels.app.kubernetes.io/name`. + +## Append a label + +Use the `extra` lists to keep the defaults and add to them: + +```yaml +olk: + filebeat: + kubernetesMetadata: + extraPodLabels: + - "my.corp/team" + extraNamespaceLabels: + - "my.corp/cost-center" + extraNodeLabels: + - "topology.kubernetes.io/zone" +``` + +The extra lists are appended to the defaults and de-duplicated, so repeating a default is harmless. +Label keys named by `olk.logstash.extendServicesPipeline` are added automatically and do not need an entry here. + +An appended label is stored on the log document and is searchable and aggregatable in OpenSearch Dashboards. +Routing and field extraction are done by the Logstash pipelines, which read a fixed set of keys, so an appended label does not change how a log line is parsed. + +## Replace the default set + +Set the base list to replace the defaults instead of adding to them: + +```yaml +olk: + filebeat: + kubernetesMetadata: + podLabels: + - "app" + - "my.corp/team" +``` + +!!! warning + + Four defaults are read by the log collection itself rather than by a pipeline. + Keep `name` in `podLabels`, keep `hopsworks.ai/project` and `hopsworks.ai/onlinefs-cluster` in `namespaceLabels`, and keep `kubernetes.io/hostname` in `nodeLabels`. + Without `name`, Filebeat collects its own logs. + Without the two `hopsworks.ai` labels, project and online feature store namespaces are no longer collected. + Neither failure reports an error. + +## Collect annotations + +Annotations are not collected by default. +Add the keys you need: + +```yaml +olk: + filebeat: + kubernetesMetadata: + podAnnotations: + - "my.corp/owner" + namespaceAnnotations: [] + nodeAnnotations: [] +``` + +!!! warning + + Annotations count towards the same 1000-field limit as labels. + List individual keys rather than collecting all annotations. + +## Deployment and cron job names + +Filebeat adds `kubernetes.deployment.name` and `kubernetes.cronjob.name` by default. +Set either to `false` to drop it: + +```yaml +olk: + filebeat: + kubernetesMetadata: + deployment: false + cronjob: false +``` + +## Going Further + +See [Services Logs](services-logs.md) for accessing the collected logs in OpenSearch Dashboards. diff --git a/mkdocs.yml b/mkdocs.yml index 2fe6db28ed..627e8785c9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -285,6 +285,7 @@ nav: - Services Dashboards: setup_installation/admin/monitoring/grafana.md - Export metrics: setup_installation/admin/monitoring/export-metrics.md - Services Logs: setup_installation/admin/monitoring/services-logs.md + - Service Log Labels: setup_installation/admin/monitoring/services-logs-labels.md - WebSocket Proxy Pool: setup_installation/admin/monitoring/websocket-pool.md - Authentication: - Configure Authentication: setup_installation/admin/auth.md From eb96f4961cf1fa865f60bc0ba7797a2ee59da277 Mon Sep 17 00:00:00 2001 From: Alex Ormenisan Date: Tue, 25 Aug 2026 15:56:40 +0200 Subject: [PATCH 2/2] [HWORKS-2912] Service logs are silently dropped from .services: unbounded Kubernetes pod and node labels exhaust the 1000-field mapping limit https://hopsworks.atlassian.net/browse/HWORKS-2912 Review of the new page found the sharpest failure mode missing and one statement contradicting a table on the same page. Document that Kubernetes metadata collection is switched on per entry of logs_locations, and that because Helm replaces lists an override of that list drops the flag with it: log lines keep arriving while every pipeline branch routing on kubernetes.* stops matching, with nothing reporting an error. Correct the replace warning, which described `kubernetes.io/hostname` as read by the collection itself when the page's own node-label table records it as read by four pipelines; three defaults are collection-internal and that one is not, and all four still have to be carried forward. Link the page from Services Logs, which is where a reader hits the dedotted-key trap. Signed-off-by: Alex Ormenisan Co-Authored-By: Claude Opus 5 (1M context) --- .../admin/monitoring/services-logs-labels.md | 33 +++++++++++++++++-- .../admin/monitoring/services-logs.md | 2 ++ 2 files changed, 33 insertions(+), 2 deletions(-) diff --git a/docs/setup_installation/admin/monitoring/services-logs-labels.md b/docs/setup_installation/admin/monitoring/services-logs-labels.md index b9e1602565..3ee645e917 100644 --- a/docs/setup_installation/admin/monitoring/services-logs-labels.md +++ b/docs/setup_installation/admin/monitoring/services-logs-labels.md @@ -71,6 +71,34 @@ Filebeat collects logs from the release namespace, from namespaces carrying eith Dots in a label key are replaced by underscores. In OpenSearch Dashboards, search for `kubernetes.labels.app_kubernetes_io/name`, not `kubernetes.labels.app.kubernetes.io/name`. +## Metadata collection per log location + +Collection is switched on per entry of `olk.filebeat.logs_locations`: + +```yaml +olk: + filebeat: + logs_locations: + - name: containerd + path: /var/log/containers + mountPaths: + - /var/log/containers + - /var/log/pods + logtype: log + glob: "/*.log" + addKubernetesMetadata: true + processors: [] +``` + +Helm replaces lists instead of merging them, so overriding `logs_locations` replaces the chart's entry in full, `addKubernetesMetadata` included. + +!!! warning + + An override that omits `addKubernetesMetadata` and does not supply its own `add_kubernetes_metadata` processor collects no Kubernetes metadata at all. + Log lines still reach OpenSearch, but every pipeline branch that routes on `kubernetes.*` stops matching, and nothing reports an error. + +Set `addKubernetesMetadata: false` only for a location that supplies its own `add_kubernetes_metadata` in `processors`, which keeps two metadata processors from running on the same location. + ## Append a label Use the `extra` lists to keep the defaults and add to them: @@ -108,11 +136,12 @@ olk: !!! warning - Four defaults are read by the log collection itself rather than by a pipeline. + Three defaults are read by the log collection itself rather than by a pipeline, and one is read by the pipelines. Keep `name` in `podLabels`, keep `hopsworks.ai/project` and `hopsworks.ai/onlinefs-cluster` in `namespaceLabels`, and keep `kubernetes.io/hostname` in `nodeLabels`. Without `name`, Filebeat collects its own logs. Without the two `hopsworks.ai` labels, project and online feature store namespaces are no longer collected. - Neither failure reports an error. + Without `kubernetes.io/hostname`, the services, Spark, Python and serving pipelines lose the node field. + None of these failures reports an error. ## Collect annotations diff --git a/docs/setup_installation/admin/monitoring/services-logs.md b/docs/setup_installation/admin/monitoring/services-logs.md index 4a2f466124..d9bbc7d019 100644 --- a/docs/setup_installation/admin/monitoring/services-logs.md +++ b/docs/setup_installation/admin/monitoring/services-logs.md @@ -48,4 +48,6 @@ These are the core component of the platform, additional logs will be added in t ## Going Further +See [Service Log Labels](services-logs-labels.md) for the Kubernetes labels attached to each log document, and how to append to or replace that set. + You can read [OpenSearch Dashboards Documentation](https://opensearch.org/docs/latest/dashboards/) to learn how to use them advancedly.