From efc320c5715649516b33f1fb740dd976c8e6245e Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 7 Aug 2026 11:31:48 +0200 Subject: [PATCH 1/5] [HWORKS-2954] Replace the OpenSearch deployment log history with HopsFS archives https://hopsworks.atlassian.net/browse/HWORKS-2954 Deployment logs no longer go through OpenSearch. Live logs are read from the running component through the Kubernetes pod-logs API, and history is archives that each instance writes to the project's Logs dataset from inside its own container. Both troubleshooting guides described the removed model, telling users to click a See logs button that opened OpenSearch Dashboards and to filter records by fields that no longer exist anywhere. Rewrites the transient and historical log sections of both guides against what the product does now, and documents the disk logging setting that governs whether history is written at all, including the per-model-server defaults and why the HopsFS sidecar is attached to every instance of a revision even in one-replica mode. Adds tail_logs and download_logs to the code section, since a snapshot is no longer the only thing the client can do. The two guides are near-duplicates and stay that way, differing only in the wording each already had for its own audience. The admin services-logs guide is deliberately untouched: platform-service logs still flow to OpenSearch, and only the workload routes were removed. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- .../mlops/serving/troubleshooting.md | 86 ++++++++++++------- .../python-deployment/troubleshooting.md | 86 +++++++++++++------ 2 files changed, 115 insertions(+), 57 deletions(-) diff --git a/docs/user_guides/mlops/serving/troubleshooting.md b/docs/user_guides/mlops/serving/troubleshooting.md index 673dddcc5a..a4ff4bd5d2 100644 --- a/docs/user_guides/mlops/serving/troubleshooting.md +++ b/docs/user_guides/mlops/serving/troubleshooting.md @@ -66,57 +66,63 @@ In those cases, you can explore the deployments logs in search of the cause of t ### Step 3: Explore transient logs Each deployment is composed of several components depending on its configuration and the model being served. -Transient logs refer to component-specific logs that are directly retrieved from the component itself. -Therefore, these logs can only be retrieved as long as the deployment components are reachable. +Transient logs refer to component-specific logs that are read directly from the running component. +Therefore, these logs can only be retrieved as long as the deployment components are running. !!! info "" Transient logs are informative and fast to retrieve, facilitating the troubleshooting of deployment components at a glance Transient logs are convenient when access to the most recent logs of a deployment is needed. +To follow them in the UI, click the `Logs` button at the top of the deployment overview page. +The pane tails the selected component every two seconds and lets you search, copy and download what it has buffered. +You can also read them with the Hopsworks Machine Learning Python library, as shown in [Step 4](#step-4-explore-transient-logs) of the code section. + !!! info When a deployment is in idle state, there are no components running (i.e., scaled to zero) and, thus, no transient logs are available. + Use historical logs to inspect an instance that is already gone. !!! note - In the current version of Hopsworks, transient logs can only be accessed using the Hopsworks Machine Learning Python library. - See [an example](#step-4-explore-transient-logs). + Standard output and standard error arrive as a single interleaved stream. + Kubernetes merges them at the container runtime, so the two cannot be separated after the fact. ### Step 4: Explore historical logs -Transient logs are continuously collected and stored in OpenSearch, where they become historical logs accessible using the integrated OpenSearch Dashboards. -Therefore, historical logs contain the same information than transient logs. -However, there might be cases where transient logs could not be collected in time for a specific component and, thus, not included in the historical logs. +Historical logs are archives that each instance writes to the project's `Logs` dataset from inside its own container. +An instance archives its output when it exits, is restarted, or is stopped, which means an instance removed by scale-to-zero or replaced by a new deployment revision still leaves its logs behind. !!! info "" - Historical logs are persisted transient logs that can be queried, filtered and sorted using OpenSearch Dashboards, facilitating a more sophisticated exploration of past records. + Historical logs are convenient when a deployment fails occasionally, or when the instance you need to inspect is no longer running -Historical logs are convenient when a deployment fails occasionally, either at inference time or without a clear reason. -In this case, narrowing the inspection of component-specific logs at a concrete point in time and searching for keywords can be helpful. +Archives are written to `Logs/Serving//` and named `__.log`, one file per instance run. +Browse them under the `Logs` section of the deployment overview page, or in the `Logs` dataset, and open one to read it. -To access the OpenSearch Dashboards, click on the `See logs` button at the top of the deployment overview page. +Historical logs are only written for components that have disk logging enabled. +See [configuring disk logging](#configuring-disk-logging) below, and note that Python predictors have it on by default while every other model server has it off. -

-

- See logs button -
Access to historical logs of a deployment
-
-

+!!! warning + The number of archives kept per deployment is capped by the `log_history_limit` cluster variable, which defaults to 30. + Once the cap is reached, the oldest archive is deleted each time a new one is written, so long-lived deployments do not fill the project with logs. -!!! note - In case you are not familiar with the interface, you may find the [official documentation](https://opensearch.org/docs/latest/dashboards/index/) useful. +To retrieve archives with the Python library, use [`download_logs`][hsml.deployment.Deployment.download_logs]. -Once in the OpenSearch Dashboards, you can search for keywords, apply multiple filters and sort the records by timestamp. +### Configuring disk logging -??? info "Available filters" +Disk logging controls whether a component archives its output to HopsFS, and how many of its instances do so. +It is configured per component, under `Disk logging` in the advanced options of the deployment form. - | Filter | Description | - | -------------- | -------------------------------------------------------------------------------------------------------- | - | component | Name of the deployment component (i.e., predictor or transformer) | - | container_name | Name of the container within a component (i.e., kserve-container, storage-initializer, inference-logger) | - | serving_name | Name of the deployment | - | model_name | Name of the model being served | - | model_version | Version of the model being served | - | timestamp | Timestamp when the record was reported | +| Setting | Behaviour | +| ------- | --------- | +| Disabled | No archives are written and no HopsFS sidecar is attached to the component. | +| One replica | A single elected instance archives its output. Chosen when one representative log per deployment is enough. | +| All replicas | Every instance archives its own output to a separate file, distinguished by pod name. | + +Python and scikit-learn predictors, including agent deployments, have disk logging set to one replica by default. +TensorFlow Serving, vLLM and transformers have it disabled by default, because attaching the HopsFS sidecar to them is only worth its cost when you actually want the archives. + +!!! note + Enabling disk logging attaches a HopsFS sidecar to the component and starts a new deployment revision. + Kubernetes gives every instance of a revision the same pod template, so the sidecar is attached to all of them even in one-replica mode, where only the elected instance writes archives. ## Code @@ -160,6 +166,28 @@ Once in the OpenSearch Dashboards, you can search for keywords, apply multiple f deployment.get_logs(component="predictor|transformer", tail=10) ``` +To follow a running deployment instead of taking a single snapshot, use `tail_logs`. +It blocks and prints new lines as they arrive, skipping what it has already shown. + +=== "Python" + + ```python + deployment.tail_logs(component="predictor") + ``` + +### Step 5: Download historical logs + +=== "Python" + + ```python + local_paths = deployment.download_logs(latest=True) + for local_path in local_paths: + with open(local_path) as archive: + print(archive.read()) + ``` + +Omit `latest` to download every archive the deployment has kept. + ### API Reference [`Deployment`][hsml.deployment.Deployment] diff --git a/docs/user_guides/projects/python-deployment/troubleshooting.md b/docs/user_guides/projects/python-deployment/troubleshooting.md index 2c1702a9d2..0355c56cdf 100644 --- a/docs/user_guides/projects/python-deployment/troubleshooting.md +++ b/docs/user_guides/projects/python-deployment/troubleshooting.md @@ -66,55 +66,63 @@ In those cases, you can explore the deployments logs in search of the cause of t ### Step 3: Explore transient logs Each deployment is composed of several components depending on its configuration. -Transient logs refer to component-specific logs that are directly retrieved from the component itself. -Therefore, these logs can only be retrieved as long as the deployment components are reachable. +Transient logs refer to component-specific logs that are read directly from the running component. +Therefore, these logs can only be retrieved as long as the deployment components are running. !!! info "" Transient logs are informative and fast to retrieve, facilitating the troubleshooting of deployment components at a glance Transient logs are convenient when access to the most recent logs of a deployment is needed. +To follow them in the UI, click the `Logs` button at the top of the deployment overview page. +The pane tails the selected component every two seconds and lets you search, copy and download what it has buffered. +You can also read them with the Hopsworks Machine Learning Python library, as shown in [Step 4](#step-4-explore-transient-logs) of the code section. + !!! info When a deployment is in idle state, there are no components running (i.e., scaled to zero) and, thus, no transient logs are available. + Use historical logs to inspect an instance that is already gone. !!! note - In the current version of Hopsworks, transient logs can only be accessed using the Hopsworks Machine Learning Python library. - See [an example](#step-4-explore-transient-logs). + Standard output and standard error arrive as a single interleaved stream. + Kubernetes merges them at the container runtime, so the two cannot be separated after the fact. ### Step 4: Explore historical logs -Transient logs are continuously collected and stored in OpenSearch, where they become historical logs accessible using the integrated OpenSearch Dashboards. -Therefore, historical logs contain the same information than transient logs. -However, there might be cases where transient logs could not be collected in time for a specific component and, thus, not included in the historical logs. +Historical logs are archives that each instance writes to the project's `Logs` dataset from inside its own container. +An instance archives its output when it exits, is restarted, or is stopped, which means an instance removed by scale-to-zero or replaced by a new deployment revision still leaves its logs behind. !!! info "" - Historical logs are persisted transient logs that can be queried, filtered and sorted using OpenSearch Dashboards, facilitating a more sophisticated exploration of past records. + Historical logs are convenient when a deployment fails occasionally, or when the instance you need to inspect is no longer running -Historical logs are convenient when a deployment fails occasionally, either at runtime or without a clear reason. -In this case, narrowing the inspection of component-specific logs at a concrete point in time and searching for keywords can be helpful. +Archives are written to `Logs/Serving//` and named `__.log`, one file per instance run. +Browse them under the `Logs` section of the deployment overview page, or in the `Logs` dataset, and open one to read it. -To access the OpenSearch Dashboards, click on the `See logs` button at the top of the deployment overview page. +Historical logs are only written for components that have disk logging enabled. +See [configuring disk logging](#configuring-disk-logging) below, and note that Python predictors have it on by default while every other model server has it off. -

-

- See logs button -
Access to historical logs of a deployment
-
-

+!!! warning + The number of archives kept per deployment is capped by the `log_history_limit` cluster variable, which defaults to 30. + Once the cap is reached, the oldest archive is deleted each time a new one is written, so long-lived deployments do not fill the project with logs. -!!! note - In case you are not familiar with the interface, you may find the [official documentation](https://opensearch.org/docs/latest/dashboards/index/) useful. +To retrieve archives with the Python library, use [`download_logs`][hsml.deployment.Deployment.download_logs]. -Once in the OpenSearch Dashboards, you can search for keywords, apply multiple filters and sort the records by timestamp. +### Configuring disk logging -??? info "Available filters" +Disk logging controls whether a component archives its output to HopsFS, and how many of its instances do so. +It is configured per component, under `Disk logging` in the advanced options of the deployment form. - | Filter | Description | - | -------------- | ---------------------------------------- | - | component | Name of the deployment component | - | container_name | Name of the container within a component | - | serving_name | Name of the deployment | - | timestamp | Timestamp when the record was reported | +| Setting | Behaviour | +| ------- | --------- | +| Disabled | No archives are written and no HopsFS sidecar is attached to the component. | +| One replica | A single elected instance archives its output. Chosen when one representative log per deployment is enough. | +| All replicas | Every instance archives its own output to a separate file, distinguished by pod name. | + +Python and scikit-learn predictors, including agent deployments, have disk logging set to one replica by default. +TensorFlow Serving, vLLM and transformers have it disabled by default, because attaching the HopsFS sidecar to them is only worth its cost when you actually want the archives. + +!!! note + Enabling disk logging attaches a HopsFS sidecar to the component and starts a new deployment revision. + Kubernetes gives every instance of a revision the same pod template, so the sidecar is attached to all of them even in one-replica mode, where only the elected instance writes archives. ## Code @@ -155,9 +163,31 @@ Once in the OpenSearch Dashboards, you can search for keywords, apply multiple f === "Python" ```python - deployment.get_logs(tail=10) + deployment.get_logs(component="predictor|transformer", tail=10) + ``` + +To follow a running deployment instead of taking a single snapshot, use `tail_logs`. +It blocks and prints new lines as they arrive, skipping what it has already shown. + +=== "Python" + + ```python + deployment.tail_logs(component="predictor") ``` +### Step 5: Download historical logs + +=== "Python" + + ```python + local_paths = deployment.download_logs(latest=True) + for local_path in local_paths: + with open(local_path) as archive: + print(archive.read()) + ``` + +Omit `latest` to download every archive the deployment has kept. + ### API Reference [`Deployment`][hsml.deployment.Deployment] From 49aedfc8ca4eda3659c3eaf1294b00164fbdbac0 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 7 Aug 2026 13:43:28 +0200 Subject: [PATCH 2/5] [HWORKS-2954] Drop the one-replica disk-logging mode from the docs https://hopsworks.atlassian.net/browse/HWORKS-2954 Disk logging is now a plain per-component checkbox: either no instance has the HopsFS sidecar, or all of them do and all of them archive. The three-way table and the one-replica default go with it. The note explaining that the sidecar lands on every instance regardless is kept and turned around, because it is now the reason there is no single-instance option rather than a caveat about one. All instances of a deployment share a pod template, so electing one writer left the others paying for a sidecar they never used. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- .../mlops/serving/troubleshooting.md | 23 ++++++++++--------- .../python-deployment/troubleshooting.md | 22 +++++++++--------- 2 files changed, 23 insertions(+), 22 deletions(-) diff --git a/docs/user_guides/mlops/serving/troubleshooting.md b/docs/user_guides/mlops/serving/troubleshooting.md index a4ff4bd5d2..bdb02c21f3 100644 --- a/docs/user_guides/mlops/serving/troubleshooting.md +++ b/docs/user_guides/mlops/serving/troubleshooting.md @@ -99,6 +99,7 @@ Browse them under the `Logs` section of the deployment overview page, or in the Historical logs are only written for components that have disk logging enabled. See [configuring disk logging](#configuring-disk-logging) below, and note that Python predictors have it on by default while every other model server has it off. +Every instance of the component writes its own archive, distinguished by pod name. !!! warning The number of archives kept per deployment is capped by the `log_history_limit` cluster variable, which defaults to 30. @@ -108,21 +109,21 @@ To retrieve archives with the Python library, use [`download_logs`][hsml.deploym ### Configuring disk logging -Disk logging controls whether a component archives its output to HopsFS, and how many of its instances do so. -It is configured per component, under `Disk logging` in the advanced options of the deployment form. +Disk logging controls whether a component archives its output to HopsFS. +It is a per-component checkbox under `Disk logging` in the advanced options of the deployment form. -| Setting | Behaviour | -| ------- | --------- | -| Disabled | No archives are written and no HopsFS sidecar is attached to the component. | -| One replica | A single elected instance archives its output. Chosen when one representative log per deployment is enough. | -| All replicas | Every instance archives its own output to a separate file, distinguished by pod name. | +When it is off, no archives are written and no HopsFS sidecar is attached to the component. +When it is on, a HopsFS sidecar is attached and every instance archives its own output to a separate file, distinguished by pod name. -Python and scikit-learn predictors, including agent deployments, have disk logging set to one replica by default. -TensorFlow Serving, vLLM and transformers have it disabled by default, because attaching the HopsFS sidecar to them is only worth its cost when you actually want the archives. +Python predictors, including agent deployments, have disk logging on by default. +TensorFlow Serving, vLLM and transformers have it off by default, because attaching the HopsFS sidecar to them is only worth its cost when you actually want the archives. !!! note - Enabling disk logging attaches a HopsFS sidecar to the component and starts a new deployment revision. - Kubernetes gives every instance of a revision the same pod template, so the sidecar is attached to all of them even in one-replica mode, where only the elected instance writes archives. + There is no single-instance mode. + All instances of a deployment share one pod template, so the sidecar cannot be attached to a subset of them: electing one writer would leave every other instance paying for a sidecar it never used. + +!!! note + Changing disk logging starts a new deployment revision, because it changes the pod template. ## Code diff --git a/docs/user_guides/projects/python-deployment/troubleshooting.md b/docs/user_guides/projects/python-deployment/troubleshooting.md index 0355c56cdf..0515bcf379 100644 --- a/docs/user_guides/projects/python-deployment/troubleshooting.md +++ b/docs/user_guides/projects/python-deployment/troubleshooting.md @@ -108,21 +108,21 @@ To retrieve archives with the Python library, use [`download_logs`][hsml.deploym ### Configuring disk logging -Disk logging controls whether a component archives its output to HopsFS, and how many of its instances do so. -It is configured per component, under `Disk logging` in the advanced options of the deployment form. +Disk logging controls whether a component archives its output to HopsFS. +It is a per-component checkbox under `Disk logging` in the advanced options of the deployment form. -| Setting | Behaviour | -| ------- | --------- | -| Disabled | No archives are written and no HopsFS sidecar is attached to the component. | -| One replica | A single elected instance archives its output. Chosen when one representative log per deployment is enough. | -| All replicas | Every instance archives its own output to a separate file, distinguished by pod name. | +When it is off, no archives are written and no HopsFS sidecar is attached to the component. +When it is on, a HopsFS sidecar is attached and every instance archives its own output to a separate file, distinguished by pod name. -Python and scikit-learn predictors, including agent deployments, have disk logging set to one replica by default. -TensorFlow Serving, vLLM and transformers have it disabled by default, because attaching the HopsFS sidecar to them is only worth its cost when you actually want the archives. +Python predictors, including agent deployments, have disk logging on by default. +TensorFlow Serving, vLLM and transformers have it off by default, because attaching the HopsFS sidecar to them is only worth its cost when you actually want the archives. !!! note - Enabling disk logging attaches a HopsFS sidecar to the component and starts a new deployment revision. - Kubernetes gives every instance of a revision the same pod template, so the sidecar is attached to all of them even in one-replica mode, where only the elected instance writes archives. + There is no single-instance mode. + All instances of a deployment share one pod template, so the sidecar cannot be attached to a subset of them: electing one writer would leave every other instance paying for a sidecar it never used. + +!!! note + Changing disk logging starts a new deployment revision, because it changes the pod template. ## Code From 4bf800df3689e11e82e8fe25ed7dd1d786959966 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sat, 8 Aug 2026 08:03:10 +0200 Subject: [PATCH 3/5] [HWORKS-2954] Restrict disk logging to Python deployments in the docs https://hopsworks.atlassian.net/browse/HWORKS-2954 Review of the code PRs restricted disk logging to Python deployments: the HopsFS sidecar it attaches runs privileged, and only Python serving pods carry the Kyverno policy exception that admits it on clusters enforcing the restricted pod security standards. Both troubleshooting pages now say TensorFlow Serving and vLLM do not support the setting and that the API rejects it, instead of describing it as an opt-in whose deployment a hardened cluster would then refuse. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/user_guides/mlops/serving/troubleshooting.md | 6 +++--- .../projects/python-deployment/troubleshooting.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/user_guides/mlops/serving/troubleshooting.md b/docs/user_guides/mlops/serving/troubleshooting.md index bdb02c21f3..843e815047 100644 --- a/docs/user_guides/mlops/serving/troubleshooting.md +++ b/docs/user_guides/mlops/serving/troubleshooting.md @@ -98,7 +98,7 @@ Archives are written to `Logs/Serving//` and named `/` and named ` Date: Fri, 21 Aug 2026 15:07:50 +0200 Subject: [PATCH 4/5] [HWORKS-2954] Replace livelogs pipeline with direct Kubernetes pod-log reading and HopsFS deployment log archives https://hopsworks.atlassian.net/browse/HWORKS-2954 Disk logging uploads to the project's Logs dataset at the end of an instance's life rather than writing through a privileged HopsFS sidecar. Support follows the serving image, so a KServe Python deployment with no predictor script is excluded alongside TensorFlow Serving and vLLM. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/user_guides/mlops/serving/troubleshooting.md | 15 +++++++++------ .../projects/python-deployment/troubleshooting.md | 15 +++++++++------ 2 files changed, 18 insertions(+), 12 deletions(-) diff --git a/docs/user_guides/mlops/serving/troubleshooting.md b/docs/user_guides/mlops/serving/troubleshooting.md index 843e815047..59df48f24d 100644 --- a/docs/user_guides/mlops/serving/troubleshooting.md +++ b/docs/user_guides/mlops/serving/troubleshooting.md @@ -109,18 +109,21 @@ To retrieve archives with the Python library, use [`download_logs`][hsml.deploym ### Configuring disk logging -Disk logging controls whether a component archives its output to HopsFS. +Disk logging controls whether a component archives its output to the project's `Logs` dataset. It is a per-component checkbox under `Disk logging` in the advanced options of the deployment form. -When it is off, no archives are written and no HopsFS sidecar is attached to the component. -When it is on, a HopsFS sidecar is attached and every instance archives its own output to a separate file, distinguished by pod name. +When it is on, each instance keeps its output on local disk while it runs and uploads it when it stops, to a separate file distinguished by pod name. +This covers stops the platform initiates on its own, such as scale-to-zero and revision replacement, not only stops a user asks for. +When it is off, nothing is written. -Disk logging is only available for Python deployments, agent deployments included, and their transformers; Python predictors have it on by default. -TensorFlow Serving and vLLM do not support it: the HopsFS sidecar it attaches runs privileged, and on clusters enforcing the restricted pod security policies only Python serving pods carry the policy exception that admits it. The API rejects the setting for those runtimes rather than deploying something a hardened cluster would refuse. +Disk logging is only available for deployments whose serving container runs a Hopsworks inference pipeline image, because the upload runs the Hopsworks Python library from inside that container. +That means Python deployments, agent deployments included, and their transformers; Python predictors have it on by default. +TensorFlow Serving and vLLM do not support it, and neither does a KServe Python deployment with no predictor script, which runs the sklearnserver runtime image. +The API rejects the setting for those rather than deploying something that cannot archive. !!! note There is no single-instance mode. - All instances of a deployment share one pod template, so the sidecar cannot be attached to a subset of them: electing one writer would leave every other instance paying for a sidecar it never used. + All instances of a deployment share one pod template, so they either all archive or none do. !!! note Changing disk logging starts a new deployment revision, because it changes the pod template. diff --git a/docs/user_guides/projects/python-deployment/troubleshooting.md b/docs/user_guides/projects/python-deployment/troubleshooting.md index 9a6a7c8cfc..da69f479fa 100644 --- a/docs/user_guides/projects/python-deployment/troubleshooting.md +++ b/docs/user_guides/projects/python-deployment/troubleshooting.md @@ -108,18 +108,21 @@ To retrieve archives with the Python library, use [`download_logs`][hsml.deploym ### Configuring disk logging -Disk logging controls whether a component archives its output to HopsFS. +Disk logging controls whether a component archives its output to the project's `Logs` dataset. It is a per-component checkbox under `Disk logging` in the advanced options of the deployment form. -When it is off, no archives are written and no HopsFS sidecar is attached to the component. -When it is on, a HopsFS sidecar is attached and every instance archives its own output to a separate file, distinguished by pod name. +When it is on, each instance keeps its output on local disk while it runs and uploads it when it stops, to a separate file distinguished by pod name. +This covers stops the platform initiates on its own, such as scale-to-zero and revision replacement, not only stops a user asks for. +When it is off, nothing is written. -Disk logging is only available for Python deployments, agent deployments included, and their transformers; Python predictors have it on by default. -TensorFlow Serving and vLLM do not support it: the HopsFS sidecar it attaches runs privileged, and on clusters enforcing the restricted pod security policies only Python serving pods carry the policy exception that admits it. The API rejects the setting for those runtimes rather than deploying something a hardened cluster would refuse. +Disk logging is only available for deployments whose serving container runs a Hopsworks inference pipeline image, because the upload runs the Hopsworks Python library from inside that container. +That means Python deployments, agent deployments included, and their transformers; Python predictors have it on by default. +TensorFlow Serving and vLLM do not support it, and neither does a KServe Python deployment with no predictor script, which runs the sklearnserver runtime image. +The API rejects the setting for those rather than deploying something that cannot archive. !!! note There is no single-instance mode. - All instances of a deployment share one pod template, so the sidecar cannot be attached to a subset of them: electing one writer would leave every other instance paying for a sidecar it never used. + All instances of a deployment share one pod template, so they either all archive or none do. !!! note Changing disk logging starts a new deployment revision, because it changes the pod template. From d3f6feefedd6703f898367fcc885b8b810b4408d Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sat, 22 Aug 2026 16:06:30 +0200 Subject: [PATCH 5/5] [HWORKS-2954] Replace livelogs pipeline with direct Kubernetes pod-log reading and HopsFS deployment log archives https://hopsworks.atlassian.net/browse/HWORKS-2954 Review round 16.10: the tail_logs example called the generator without iterating it, so the snippet as written did nothing; both troubleshooting pages now iterate and print the yielded chunks, and the prose says it returns a generator rather than that it prints. Signed-off-by: Jim Dowling Co-Authored-By: Claude Fable 5 --- docs/user_guides/mlops/serving/troubleshooting.md | 5 +++-- .../projects/python-deployment/troubleshooting.md | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/user_guides/mlops/serving/troubleshooting.md b/docs/user_guides/mlops/serving/troubleshooting.md index 59df48f24d..5b494abd6a 100644 --- a/docs/user_guides/mlops/serving/troubleshooting.md +++ b/docs/user_guides/mlops/serving/troubleshooting.md @@ -171,12 +171,13 @@ The API rejects the setting for those rather than deploying something that canno ``` To follow a running deployment instead of taking a single snapshot, use `tail_logs`. -It blocks and prints new lines as they arrive, skipping what it has already shown. +It returns a generator that yields new lines as they arrive, skipping what it has already yielded. === "Python" ```python - deployment.tail_logs(component="predictor") + for chunk in deployment.tail_logs(component="predictor"): + print(chunk, end="") ``` ### Step 5: Download historical logs diff --git a/docs/user_guides/projects/python-deployment/troubleshooting.md b/docs/user_guides/projects/python-deployment/troubleshooting.md index da69f479fa..b395313156 100644 --- a/docs/user_guides/projects/python-deployment/troubleshooting.md +++ b/docs/user_guides/projects/python-deployment/troubleshooting.md @@ -170,12 +170,13 @@ The API rejects the setting for those rather than deploying something that canno ``` To follow a running deployment instead of taking a single snapshot, use `tail_logs`. -It blocks and prints new lines as they arrive, skipping what it has already shown. +It returns a generator that yields new lines as they arrive, skipping what it has already yielded. === "Python" ```python - deployment.tail_logs(component="predictor") + for chunk in deployment.tail_logs(component="predictor"): + print(chunk, end="") ``` ### Step 5: Download historical logs