diff --git a/docs/setup_installation/admin/monitoring/services-logs-labels.md b/docs/setup_installation/admin/monitoring/services-logs-labels.md new file mode 100644 index 0000000000..3ee645e917 --- /dev/null +++ b/docs/setup_installation/admin/monitoring/services-logs-labels.md @@ -0,0 +1,181 @@ +# Service Log Labels + +## Introduction + +Filebeat attaches Kubernetes metadata to every service log document before Logstash forwards it to OpenSearch. +Only an explicit set of pod, namespace and node labels is kept. + +The set is bounded because every distinct label key becomes a field in the shared `.services-*` index mapping. +OpenSearch rejects documents once an index exceeds `index.mapping.total_fields.limit`, which defaults to 1000 fields. +A cluster that attaches every pod and node label, including the label sets that cloud providers and node feature discovery add, reaches that limit and then stops indexing service logs. + +In this guide you will learn which labels are kept by default, and how to append to or replace that set. + +## Default pod labels + +| Label | Purpose | +| --- | --- | +| `name` | Excludes Filebeat's own logs from collection. | +| `app` | Service identification, read by most pipelines. | +| `app.kubernetes.io/name` | Service identification for charts using the recommended Kubernetes labels. | +| `service` | Service name. | +| `component` | Component within a service. | +| `rondbService` | RonDB process type. | +| `user` | Owner of a job, notebook or serving instance. | +| `job-type` | Job type. | +| `job-id` | Job identifier. | +| `job-name` | Job name. | +| `execution` | Execution identifier of a job run. | +| `jupyter` | Marks a Jupyter pod. | +| `jupyter-id` | Jupyter instance identifier. | +| `jupyter-settings-id` | Jupyter settings identifier. | +| `kernel-id` | Jupyter kernel identifier. | +| `spark-role` | Driver or executor. | +| `spark-app-selector` | Spark application identifier. | +| `sparkoperator.k8s.io/launched-by-spark-operator` | Marks pods created by the Spark operator. | +| `serving.hops.works/id` | Deployment identifier. | +| `serving.hops.works/name` | Deployment name. | +| `serving.hops.works/tool` | Serving tool. | +| `serving.hops.works/model-name` | Model name. | +| `serving.hops.works/model-version` | Model version. | +| `serving.hops.works/model-server` | Model server. | +| `serving.hops.works/project-id` | Project that owns the deployment. | + +## Default namespace labels + +| Label | Purpose | +| --- | --- | +| `hopsworks.ai/project` | Marks a project namespace. | +| `hopsworks.ai/onlinefs-cluster` | Marks an online feature store namespace. | + +Filebeat collects logs from the release namespace, from namespaces carrying either of these two labels, and from any namespace listed in `olk.filebeat.extraNamespaces`. + +## Default node labels + +| Label | Purpose | +| --- | --- | +| `kubernetes.io/hostname` | Node name, read by the services, Spark, Python and serving pipelines. | + +## How labels appear in a log document + +```json +{ + "kubernetes": { + "labels": { "app": "namenode", "app_kubernetes_io/name": "hopsfs" }, + "namespace_labels": { "hopsworks_ai/project": "demo" }, + "node": { "labels": { "kubernetes_io/hostname": "worker-1" } } + } +} +``` + +Dots in a label key are replaced by underscores. +In OpenSearch Dashboards, search for `kubernetes.labels.app_kubernetes_io/name`, not `kubernetes.labels.app.kubernetes.io/name`. + +## Metadata collection per log location + +Collection is switched on per entry of `olk.filebeat.logs_locations`: + +```yaml +olk: + filebeat: + logs_locations: + - name: containerd + path: /var/log/containers + mountPaths: + - /var/log/containers + - /var/log/pods + logtype: log + glob: "/*.log" + addKubernetesMetadata: true + processors: [] +``` + +Helm replaces lists instead of merging them, so overriding `logs_locations` replaces the chart's entry in full, `addKubernetesMetadata` included. + +!!! warning + + An override that omits `addKubernetesMetadata` and does not supply its own `add_kubernetes_metadata` processor collects no Kubernetes metadata at all. + Log lines still reach OpenSearch, but every pipeline branch that routes on `kubernetes.*` stops matching, and nothing reports an error. + +Set `addKubernetesMetadata: false` only for a location that supplies its own `add_kubernetes_metadata` in `processors`, which keeps two metadata processors from running on the same location. + +## Append a label + +Use the `extra` lists to keep the defaults and add to them: + +```yaml +olk: + filebeat: + kubernetesMetadata: + extraPodLabels: + - "my.corp/team" + extraNamespaceLabels: + - "my.corp/cost-center" + extraNodeLabels: + - "topology.kubernetes.io/zone" +``` + +The extra lists are appended to the defaults and de-duplicated, so repeating a default is harmless. +Label keys named by `olk.logstash.extendServicesPipeline` are added automatically and do not need an entry here. + +An appended label is stored on the log document and is searchable and aggregatable in OpenSearch Dashboards. +Routing and field extraction are done by the Logstash pipelines, which read a fixed set of keys, so an appended label does not change how a log line is parsed. + +## Replace the default set + +Set the base list to replace the defaults instead of adding to them: + +```yaml +olk: + filebeat: + kubernetesMetadata: + podLabels: + - "app" + - "my.corp/team" +``` + +!!! warning + + Three defaults are read by the log collection itself rather than by a pipeline, and one is read by the pipelines. + Keep `name` in `podLabels`, keep `hopsworks.ai/project` and `hopsworks.ai/onlinefs-cluster` in `namespaceLabels`, and keep `kubernetes.io/hostname` in `nodeLabels`. + Without `name`, Filebeat collects its own logs. + Without the two `hopsworks.ai` labels, project and online feature store namespaces are no longer collected. + Without `kubernetes.io/hostname`, the services, Spark, Python and serving pipelines lose the node field. + None of these failures reports an error. + +## Collect annotations + +Annotations are not collected by default. +Add the keys you need: + +```yaml +olk: + filebeat: + kubernetesMetadata: + podAnnotations: + - "my.corp/owner" + namespaceAnnotations: [] + nodeAnnotations: [] +``` + +!!! warning + + Annotations count towards the same 1000-field limit as labels. + List individual keys rather than collecting all annotations. + +## Deployment and cron job names + +Filebeat adds `kubernetes.deployment.name` and `kubernetes.cronjob.name` by default. +Set either to `false` to drop it: + +```yaml +olk: + filebeat: + kubernetesMetadata: + deployment: false + cronjob: false +``` + +## Going Further + +See [Services Logs](services-logs.md) for accessing the collected logs in OpenSearch Dashboards. diff --git a/docs/setup_installation/admin/monitoring/services-logs.md b/docs/setup_installation/admin/monitoring/services-logs.md index 4a2f466124..d9bbc7d019 100644 --- a/docs/setup_installation/admin/monitoring/services-logs.md +++ b/docs/setup_installation/admin/monitoring/services-logs.md @@ -48,4 +48,6 @@ These are the core component of the platform, additional logs will be added in t ## Going Further +See [Service Log Labels](services-logs-labels.md) for the Kubernetes labels attached to each log document, and how to append to or replace that set. + You can read [OpenSearch Dashboards Documentation](https://opensearch.org/docs/latest/dashboards/) to learn how to use them advancedly. diff --git a/mkdocs.yml b/mkdocs.yml index 2fe6db28ed..627e8785c9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -285,6 +285,7 @@ nav: - Services Dashboards: setup_installation/admin/monitoring/grafana.md - Export metrics: setup_installation/admin/monitoring/export-metrics.md - Services Logs: setup_installation/admin/monitoring/services-logs.md + - Service Log Labels: setup_installation/admin/monitoring/services-logs-labels.md - WebSocket Proxy Pool: setup_installation/admin/monitoring/websocket-pool.md - Authentication: - Configure Authentication: setup_installation/admin/auth.md