diff --git a/docs/user_guides/mlops/serving/troubleshooting.md b/docs/user_guides/mlops/serving/troubleshooting.md index 673dddcc5a..5b494abd6a 100644 --- a/docs/user_guides/mlops/serving/troubleshooting.md +++ b/docs/user_guides/mlops/serving/troubleshooting.md @@ -66,57 +66,67 @@ In those cases, you can explore the deployments logs in search of the cause of t ### Step 3: Explore transient logs Each deployment is composed of several components depending on its configuration and the model being served. -Transient logs refer to component-specific logs that are directly retrieved from the component itself. -Therefore, these logs can only be retrieved as long as the deployment components are reachable. +Transient logs refer to component-specific logs that are read directly from the running component. +Therefore, these logs can only be retrieved as long as the deployment components are running. !!! info "" Transient logs are informative and fast to retrieve, facilitating the troubleshooting of deployment components at a glance Transient logs are convenient when access to the most recent logs of a deployment is needed. +To follow them in the UI, click the `Logs` button at the top of the deployment overview page. +The pane tails the selected component every two seconds and lets you search, copy and download what it has buffered. +You can also read them with the Hopsworks Machine Learning Python library, as shown in [Step 4](#step-4-explore-transient-logs) of the code section. + !!! info When a deployment is in idle state, there are no components running (i.e., scaled to zero) and, thus, no transient logs are available. + Use historical logs to inspect an instance that is already gone. !!! note - In the current version of Hopsworks, transient logs can only be accessed using the Hopsworks Machine Learning Python library. - See [an example](#step-4-explore-transient-logs). + Standard output and standard error arrive as a single interleaved stream. + Kubernetes merges them at the container runtime, so the two cannot be separated after the fact. ### Step 4: Explore historical logs -Transient logs are continuously collected and stored in OpenSearch, where they become historical logs accessible using the integrated OpenSearch Dashboards. -Therefore, historical logs contain the same information than transient logs. -However, there might be cases where transient logs could not be collected in time for a specific component and, thus, not included in the historical logs. +Historical logs are archives that each instance writes to the project's `Logs` dataset from inside its own container. +An instance archives its output when it exits, is restarted, or is stopped, which means an instance removed by scale-to-zero or replaced by a new deployment revision still leaves its logs behind. !!! info "" - Historical logs are persisted transient logs that can be queried, filtered and sorted using OpenSearch Dashboards, facilitating a more sophisticated exploration of past records. + Historical logs are convenient when a deployment fails occasionally, or when the instance you need to inspect is no longer running -Historical logs are convenient when a deployment fails occasionally, either at inference time or without a clear reason. -In this case, narrowing the inspection of component-specific logs at a concrete point in time and searching for keywords can be helpful. +Archives are written to `Logs/Serving//` and named `__.log`, one file per instance run. +Browse them under the `Logs` section of the deployment overview page, or in the `Logs` dataset, and open one to read it. -To access the OpenSearch Dashboards, click on the `See logs` button at the top of the deployment overview page. +Historical logs are only written for components that have disk logging enabled. +See [configuring disk logging](#configuring-disk-logging) below, and note that only Python deployments support it: Python predictors have it on by default. +Every instance of the component writes its own archive, distinguished by pod name. -

-

- See logs button -
Access to historical logs of a deployment
-
-

+!!! warning + The number of archives kept per deployment is capped by the `log_history_limit` cluster variable, which defaults to 30. + Once the cap is reached, the oldest archive is deleted each time a new one is written, so long-lived deployments do not fill the project with logs. -!!! note - In case you are not familiar with the interface, you may find the [official documentation](https://opensearch.org/docs/latest/dashboards/index/) useful. +To retrieve archives with the Python library, use [`download_logs`][hsml.deployment.Deployment.download_logs]. -Once in the OpenSearch Dashboards, you can search for keywords, apply multiple filters and sort the records by timestamp. +### Configuring disk logging -??? info "Available filters" +Disk logging controls whether a component archives its output to the project's `Logs` dataset. +It is a per-component checkbox under `Disk logging` in the advanced options of the deployment form. - | Filter | Description | - | -------------- | -------------------------------------------------------------------------------------------------------- | - | component | Name of the deployment component (i.e., predictor or transformer) | - | container_name | Name of the container within a component (i.e., kserve-container, storage-initializer, inference-logger) | - | serving_name | Name of the deployment | - | model_name | Name of the model being served | - | model_version | Version of the model being served | - | timestamp | Timestamp when the record was reported | +When it is on, each instance keeps its output on local disk while it runs and uploads it when it stops, to a separate file distinguished by pod name. +This covers stops the platform initiates on its own, such as scale-to-zero and revision replacement, not only stops a user asks for. +When it is off, nothing is written. + +Disk logging is only available for deployments whose serving container runs a Hopsworks inference pipeline image, because the upload runs the Hopsworks Python library from inside that container. +That means Python deployments, agent deployments included, and their transformers; Python predictors have it on by default. +TensorFlow Serving and vLLM do not support it, and neither does a KServe Python deployment with no predictor script, which runs the sklearnserver runtime image. +The API rejects the setting for those rather than deploying something that cannot archive. + +!!! note + There is no single-instance mode. + All instances of a deployment share one pod template, so they either all archive or none do. + +!!! note + Changing disk logging starts a new deployment revision, because it changes the pod template. ## Code @@ -160,6 +170,29 @@ Once in the OpenSearch Dashboards, you can search for keywords, apply multiple f deployment.get_logs(component="predictor|transformer", tail=10) ``` +To follow a running deployment instead of taking a single snapshot, use `tail_logs`. +It returns a generator that yields new lines as they arrive, skipping what it has already yielded. + +=== "Python" + + ```python + for chunk in deployment.tail_logs(component="predictor"): + print(chunk, end="") + ``` + +### Step 5: Download historical logs + +=== "Python" + + ```python + local_paths = deployment.download_logs(latest=True) + for local_path in local_paths: + with open(local_path) as archive: + print(archive.read()) + ``` + +Omit `latest` to download every archive the deployment has kept. + ### API Reference [`Deployment`][hsml.deployment.Deployment] diff --git a/docs/user_guides/projects/python-deployment/troubleshooting.md b/docs/user_guides/projects/python-deployment/troubleshooting.md index 2c1702a9d2..b395313156 100644 --- a/docs/user_guides/projects/python-deployment/troubleshooting.md +++ b/docs/user_guides/projects/python-deployment/troubleshooting.md @@ -66,55 +66,66 @@ In those cases, you can explore the deployments logs in search of the cause of t ### Step 3: Explore transient logs Each deployment is composed of several components depending on its configuration. -Transient logs refer to component-specific logs that are directly retrieved from the component itself. -Therefore, these logs can only be retrieved as long as the deployment components are reachable. +Transient logs refer to component-specific logs that are read directly from the running component. +Therefore, these logs can only be retrieved as long as the deployment components are running. !!! info "" Transient logs are informative and fast to retrieve, facilitating the troubleshooting of deployment components at a glance Transient logs are convenient when access to the most recent logs of a deployment is needed. +To follow them in the UI, click the `Logs` button at the top of the deployment overview page. +The pane tails the selected component every two seconds and lets you search, copy and download what it has buffered. +You can also read them with the Hopsworks Machine Learning Python library, as shown in [Step 4](#step-4-explore-transient-logs) of the code section. + !!! info When a deployment is in idle state, there are no components running (i.e., scaled to zero) and, thus, no transient logs are available. + Use historical logs to inspect an instance that is already gone. !!! note - In the current version of Hopsworks, transient logs can only be accessed using the Hopsworks Machine Learning Python library. - See [an example](#step-4-explore-transient-logs). + Standard output and standard error arrive as a single interleaved stream. + Kubernetes merges them at the container runtime, so the two cannot be separated after the fact. ### Step 4: Explore historical logs -Transient logs are continuously collected and stored in OpenSearch, where they become historical logs accessible using the integrated OpenSearch Dashboards. -Therefore, historical logs contain the same information than transient logs. -However, there might be cases where transient logs could not be collected in time for a specific component and, thus, not included in the historical logs. +Historical logs are archives that each instance writes to the project's `Logs` dataset from inside its own container. +An instance archives its output when it exits, is restarted, or is stopped, which means an instance removed by scale-to-zero or replaced by a new deployment revision still leaves its logs behind. !!! info "" - Historical logs are persisted transient logs that can be queried, filtered and sorted using OpenSearch Dashboards, facilitating a more sophisticated exploration of past records. + Historical logs are convenient when a deployment fails occasionally, or when the instance you need to inspect is no longer running -Historical logs are convenient when a deployment fails occasionally, either at runtime or without a clear reason. -In this case, narrowing the inspection of component-specific logs at a concrete point in time and searching for keywords can be helpful. +Archives are written to `Logs/Serving//` and named `__.log`, one file per instance run. +Browse them under the `Logs` section of the deployment overview page, or in the `Logs` dataset, and open one to read it. -To access the OpenSearch Dashboards, click on the `See logs` button at the top of the deployment overview page. +Historical logs are only written for components that have disk logging enabled. +See [configuring disk logging](#configuring-disk-logging) below, and note that only Python deployments support it: Python predictors have it on by default. -

-

- See logs button -
Access to historical logs of a deployment
-
-

+!!! warning + The number of archives kept per deployment is capped by the `log_history_limit` cluster variable, which defaults to 30. + Once the cap is reached, the oldest archive is deleted each time a new one is written, so long-lived deployments do not fill the project with logs. -!!! note - In case you are not familiar with the interface, you may find the [official documentation](https://opensearch.org/docs/latest/dashboards/index/) useful. +To retrieve archives with the Python library, use [`download_logs`][hsml.deployment.Deployment.download_logs]. -Once in the OpenSearch Dashboards, you can search for keywords, apply multiple filters and sort the records by timestamp. +### Configuring disk logging -??? info "Available filters" +Disk logging controls whether a component archives its output to the project's `Logs` dataset. +It is a per-component checkbox under `Disk logging` in the advanced options of the deployment form. - | Filter | Description | - | -------------- | ---------------------------------------- | - | component | Name of the deployment component | - | container_name | Name of the container within a component | - | serving_name | Name of the deployment | - | timestamp | Timestamp when the record was reported | +When it is on, each instance keeps its output on local disk while it runs and uploads it when it stops, to a separate file distinguished by pod name. +This covers stops the platform initiates on its own, such as scale-to-zero and revision replacement, not only stops a user asks for. +When it is off, nothing is written. + +Disk logging is only available for deployments whose serving container runs a Hopsworks inference pipeline image, because the upload runs the Hopsworks Python library from inside that container. +That means Python deployments, agent deployments included, and their transformers; Python predictors have it on by default. +TensorFlow Serving and vLLM do not support it, and neither does a KServe Python deployment with no predictor script, which runs the sklearnserver runtime image. +The API rejects the setting for those rather than deploying something that cannot archive. + +!!! note + There is no single-instance mode. + All instances of a deployment share one pod template, so they either all archive or none do. + +!!! note + Changing disk logging starts a new deployment revision, because it changes the pod template. ## Code @@ -155,9 +166,32 @@ Once in the OpenSearch Dashboards, you can search for keywords, apply multiple f === "Python" ```python - deployment.get_logs(tail=10) + deployment.get_logs(component="predictor|transformer", tail=10) + ``` + +To follow a running deployment instead of taking a single snapshot, use `tail_logs`. +It returns a generator that yields new lines as they arrive, skipping what it has already yielded. + +=== "Python" + + ```python + for chunk in deployment.tail_logs(component="predictor"): + print(chunk, end="") + ``` + +### Step 5: Download historical logs + +=== "Python" + + ```python + local_paths = deployment.download_logs(latest=True) + for local_path in local_paths: + with open(local_path) as archive: + print(archive.read()) ``` +Omit `latest` to download every archive the deployment has kept. + ### API Reference [`Deployment`][hsml.deployment.Deployment]