[HWORKS-2912] Service logs are silently dropped from .services: unbounded Kubernetes pod and node labels exhaust the 1000-field mapping limit - #638
Draft
o-alex wants to merge 2 commits into
Conversation
…nded Kubernetes pod and node labels exhaust the 1000-field mapping limit https://hopsworks.atlassian.net/browse/HWORKS-2912 Bounding the Kubernetes metadata that Filebeat attaches to service logs makes the kept label set part of the cluster's configuration surface, so it needs to be documented rather than read out of the chart. An operator who adds a routing label to their own workloads has no way to know from the outside which keys survive collection. Add a Service Log Labels page next to Services Logs, listing the pod, namespace and node labels kept by default and what each is for, and showing how to append with the extra lists, how to replace a base list, and how to opt into annotations and the deployment/cronjob names. Record the dedotting rule, since searching Dashboards for a dotted key returns nothing. The page warns about the four defaults that the collection itself depends on: replacing a base list without them stops Filebeat excluding its own logs, or stops project namespaces being collected, and neither failure reports an error. Signed-off-by: Alex Ormenisan <alex@logicalclocks.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nded Kubernetes pod and node labels exhaust the 1000-field mapping limit https://hopsworks.atlassian.net/browse/HWORKS-2912 Review of the new page found the sharpest failure mode missing and one statement contradicting a table on the same page. Document that Kubernetes metadata collection is switched on per entry of logs_locations, and that because Helm replaces lists an override of that list drops the flag with it: log lines keep arriving while every pipeline branch routing on kubernetes.* stops matching, with nothing reporting an error. Correct the replace warning, which described `kubernetes.io/hostname` as read by the collection itself when the page's own node-label table records it as read by four pipelines; three defaults are collection-internal and that one is not, and all four still have to be carried forward. Link the page from Services Logs, which is where a reader hits the dedotted-key trap. Signed-off-by: Alex Ormenisan <alex@logicalclocks.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Service Log Labels page documenting which Kubernetes labels are attached to service logs by default, and how to append to or replace that set.
Read
docs/setup_installation/admin/monitoring/services-logs-labels.md; themkdocs.ymlchange is its nav entry.Why
The chart change in logicalclocks/hopsworks-helm#2241 stops filebeat attaching every pod and node label to each log document, because each distinct key becomes a field in the shared
.services-*index and the index stops accepting documents once it passes 1000 fields. That makes the kept set part of a cluster's configuration surface: an operator who labels their own workloads for routing cannot tell from the outside which keys survive collection, and a label that does not survive fails silently.The page therefore documents the defaults as tables with a purpose per key (25 pod, 2 namespace, 1 node), the append and replace paths, annotations and the deployment/cronjob names, and the dedotting rule, since searching Dashboards for a dotted key such as
kubernetes.labels.app.kubernetes.io/namereturns nothing and gives no hint why.It also carries the warning that four defaults are read by the collection itself rather than by a pipeline, so replacing a base list without them stops filebeat excluding its own logs or stops project namespaces being collected, with no error either way.
Verification
npx markdownlint-cli2reports 0 errors on the new file. A strict build (uv run mkdocs build -s) completes with 0 errors and 0 warnings and emits the page. Note that the strict build requiresuv pip install "git+https://github.com/logicalclocks/hopsworks-api.git@main#subdirectory=python"first, per.claude/CLAUDE.md, otherwise it fails onhopsworks_commonin the API reference section.Risk and rollout
Documentation only. The page describes values that ship in the linked chart PR, so it should land with or after it.
Rollback: