From 18773452c3fb070253b9bc8522ff0c8a4eeff20e Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Tue, 25 Aug 2026 14:24:54 +0200 Subject: [PATCH 1/3] [HWORKS-2895] Global Hopsworks Dashboard using Superset https://hopsworks.atlassian.net/browse/HWORKS-2895 Record the history of archived tag attachments, so the dashboards can ask how long an artifact sat in each state rather than only where it is now. The backend writes one immutable row per transition of one key of one tag on one artifact, in the same transaction as the tag write; this is the surrounding work that makes it readable and usable. Document the archive flag and what it produces. The two facts a reader has to have before turning it on are the ones they cannot recover from getting wrong: history begins at the moment the flag is set, because the live tag keeps only its current value, and clearing the flag stops recording without deleting what was already recorded. The table stores transitions rather than intervals, so the page gives the query that turns one into the other and says what an empty added_on or removed_at means: still in this state, or attached before Hopsworks recorded attachment times. Both are left empty rather than guessed, and a reader averaging over the column needs to know which. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/user_guides/fs/tags/tags.md | 59 +++++++++++++++++++++++++++++++- 1 file changed, 58 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/fs/tags/tags.md b/docs/user_guides/fs/tags/tags.md index 6fc582829c..f9fc47c6e9 100644 --- a/docs/user_guides/fs/tags/tags.md +++ b/docs/user_guides/fs/tags/tags.md @@ -135,7 +135,64 @@ From there you can select the tag schema of the tag you want to attach and popul

-## Step 3: Search +## Step 3: Archive tag history + +By default a tag records only its current value: reading it tells you where an artifact is now, not +where it has been. Turning on **Archive tag history** for a schema makes Hopsworks additionally +record every change to that tag's values, so you can ask how long an artifact spent in each state. + +The flag is set per schema, in `Cluster settings` > `Tag schemas`, either when the schema is created +or afterwards. It applies to every artifact the tag is attached to: feature groups, feature views, +training datasets, jobs, models and deployments. + +Two things are worth knowing before you turn it on: + +- **History starts when you turn it on.** Changes made before that are not recoverable, because the + live tag keeps only its current value. Attachments that already exist are backfilled with the + state they are in, timed from when they were attached. +- **Turning it off stops recording but keeps what was recorded.** The rows already written are still + true, and the tag is still attached, so nothing is deleted. + +History is recorded per key of the schema, not per tag. Changing one key of a multi-key tag records +a change to that key alone and leaves the others untouched, so a correction to one field does not +make every other field look like it changed at the same moment. + +### Reading the history + +The history is stored in the `tag_history` table of the Hopsworks metadata database, one row per +transition: the value became current, or it stopped being current. A value change writes both at the +same instant, so one interval's end is the next one's start. + +It is read with SQL rather than through the tag APIs, which continue to return the current value. +On a cluster with the `hopsworks_analytics` project enabled, its Superset connection can query the +table directly, and +`create_tag_history_dashboard.py` in the `okr-dashboards` repository builds a "Tag Lifecycle" +dashboard over it: time spent in each state, whether that is increasing, what is in each state now, +and what has been in one state longest. + +The table stores events rather than intervals. To get `added_on` and `removed_at`, take the next +event's time for the same artifact, tag and key: + +```sql +SELECT artifact_type, artifact_id, tag_name, tag_key, tag_value, + event_time AS added_on, + LEAD(event_time) OVER ( + PARTITION BY artifact_type, artifact_id, tag_name, tag_key + ORDER BY event_time, id + ) AS removed_at +FROM hopsworks.tag_history +WHERE event_type = 'OPENED' +``` + +A `removed_at` of `NULL` means the artifact is still in that state. An `added_on` of `NULL` means the +tag was attached before Hopsworks began recording attachment times, so the start is unknown; it is +left empty rather than filled with a guess. + +The history outlives what it describes. Deleting the artifact, the tag schema or the project closes +the open intervals and keeps the rows, so a report over a past quarter still returns what was true +then. + +## Step 4: Search Hopsworks indexes the tags attached to feature groups, feature views and training datasets. The tags will then be searchable using the free text search box located at the top of the UI. From 57e665bd93684770bf63bd0072ca4138c3b17ae1 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Tue, 25 Aug 2026 17:12:35 +0200 Subject: [PATCH 2/3] [HWORKS-2895] Document archiving tag history https://hopsworks.atlassian.net/browse/HWORKS-2895 The guide said the flag could be set when a schema is created or afterwards, in Cluster settings. There is no afterwards in the UI: the front-end wires the flag only in the create form, and there is no schema edit page. Toggling later is an admin REST call, which the guide now gives. Also documents what turning it off does. It ends the intervals still open, at that moment, and keeps everything already recorded. Without that the reader has no way to know the last value of every artifact would otherwise read as current forever, since nothing closes an interval once recording stops. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/user_guides/fs/tags/tags.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/user_guides/fs/tags/tags.md b/docs/user_guides/fs/tags/tags.md index 0c9b9b5ba2..9e1e5463ed 100644 --- a/docs/user_guides/fs/tags/tags.md +++ b/docs/user_guides/fs/tags/tags.md @@ -252,9 +252,16 @@ By default a tag records only its current value: reading it tells you where an a where it has been. Turning on **Archive tag history** for a schema makes Hopsworks additionally record every change to that tag's values, so you can ask how long an artifact spent in each state. -The flag is set per schema, in `Cluster settings` > `Tag schemas`, either when the schema is created -or afterwards. It applies to every artifact the tag is attached to: feature groups, feature views, -training datasets, jobs, models and deployments. +The flag is set per schema, in `Cluster settings` > `Tag schemas`, when the schema is created. To +turn it on or off for a schema that already exists, a cluster administrator calls +`PUT /hopsworks-api/api/tags/{name}/archive?value=true`. It applies to every artifact the tag is +attached to: feature groups, feature views, training datasets, jobs, models and deployments. + +Turning it off ends every interval the tag still has open, at the moment you turn it off, and keeps +everything already recorded. The recorded rows stay because they are still true; the open intervals +have to be ended because nothing would ever end them once recording stops, and the last value of +every artifact would otherwise read as current forever. Turning it back on starts a fresh interval +at that moment rather than pretending the gap was observed. Two things are worth knowing before you turn it on: From 8eb00080277ab0bf1715c011676d7844848695cc Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Thu, 27 Aug 2026 07:18:11 +0200 Subject: [PATCH 3/3] [HWORKS-2895] Document archiving tag history https://hopsworks.atlassian.net/browse/HWORKS-2895 Review round 5: the published interval query was the broken one. The dashboards and the design note were corrected in round 4; this page was not, so the query readers are most likely to copy was the only copy still wrong in both ways. It filtered to OPENED in the same query that evaluates LEAD. SQL applies WHERE before window functions, so the window never saw a CLOSED row and anything ending without a successor, a detached tag or a deleted artifact, read as still current with its duration growing against the clock forever. It also ordered by id, which cannot break the tie between the two halves of a value change: they share one event_time by design and are not written in the order they were built. Both are now shown correctly, with a note on each, because both produce numbers that look entirely reasonable. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/user_guides/fs/tags/tags.md | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/docs/user_guides/fs/tags/tags.md b/docs/user_guides/fs/tags/tags.md index 9e1e5463ed..ab5371e16a 100644 --- a/docs/user_guides/fs/tags/tags.md +++ b/docs/user_guides/fs/tags/tags.md @@ -292,16 +292,32 @@ The table stores events rather than intervals. To get `added_on` and `removed_at event's time for the same artifact, tag and key: ```sql -SELECT artifact_type, artifact_id, tag_name, tag_key, tag_value, - event_time AS added_on, - LEAD(event_time) OVER ( - PARTITION BY artifact_type, artifact_id, tag_name, tag_key - ORDER BY event_time, id - ) AS removed_at -FROM hopsworks.tag_history -WHERE event_type = 'OPENED' +SELECT e.artifact_type, e.artifact_id, e.tag_name, e.tag_key, e.tag_value, + e.added_on, e.removed_at +FROM ( + SELECT artifact_type, artifact_id, tag_name, tag_key, tag_value, event_type, + event_time AS added_on, + LEAD(event_time) OVER ( + PARTITION BY artifact_type, artifact_id, tag_name, tag_key + ORDER BY event_time, + CASE WHEN event_type = 'CLOSED' THEN 0 ELSE 1 END, + id + ) AS removed_at + FROM hopsworks.tag_history +) e +WHERE e.event_type = 'OPENED' ``` +Two details in that query are easy to get wrong and produce numbers that look reasonable: + +- The `OPENED` filter has to be in the outer query. SQL applies `WHERE` before window functions, so + filtering inside would hide every `CLOSED` row from `LEAD`, and anything that ended without a + successor, a detached tag or a deleted artifact, would report as still current with its duration + growing forever. +- The ordering has to put `CLOSED` before `OPENED` at the same timestamp. Both halves of a value + change share one `event_time` by design, so the ordering needs a tie-break, and `id` is not one: + rows are not written in the order the two halves were built. + A `removed_at` of `NULL` means the artifact is still in that state. An `added_on` of `NULL` means the tag was attached before Hopsworks began recording attachment times, so the start is unknown; it is left empty rather than filled with a guess.