[HWORKS-2895] Document archiving tag history

docs/user_guides/fs/tags/tags.md

@@ -246,6 +246,86 @@ Re-attaching a tag to change its value keeps the original attachment time, so th
 `created_on` is `None` when the attachment time is unknown rather than recent.
 That happens for tags attached before the cluster recorded attachment times, and for legacy per-file dataset tags, which are stored as HopsFS extended attributes and carry no timestamp.
 
+## Archive tag history
+
+By default a tag records only its current value: reading it tells you where an artifact is now, not
+where it has been. Turning on **Archive tag history** for a schema makes Hopsworks additionally
+record every change to that tag's values, so you can ask how long an artifact spent in each state.
+
+The flag is set per schema, in `Cluster settings` > `Tag schemas`, when the schema is created. To
+turn it on or off for a schema that already exists, a cluster administrator calls
+`PUT /hopsworks-api/api/tags/{name}/archive?value=true`. It applies to every artifact the tag is
+attached to: feature groups, feature views, training datasets, jobs, models and deployments.
+
+Turning it off ends every interval the tag still has open, at the moment you turn it off, and keeps
+everything already recorded. The recorded rows stay because they are still true; the open intervals
+have to be ended because nothing would ever end them once recording stops, and the last value of
+every artifact would otherwise read as current forever. Turning it back on starts a fresh interval
+at that moment rather than pretending the gap was observed.
+
+Two things are worth knowing before you turn it on:
+
+- **History starts when you turn it on.** Changes made before that are not recoverable, because the
+  live tag keeps only its current value. Attachments that already exist are backfilled with the
+  state they are in, timed from when they were attached.
+- **Turning it off stops recording but keeps what was recorded.** The rows already written are still
+  true, and the tag is still attached, so nothing is deleted.
+
+History is recorded per key of the schema, not per tag. Changing one key of a multi-key tag records
+a change to that key alone and leaves the others untouched, so a correction to one field does not
+make every other field look like it changed at the same moment.
+
+### Reading the history
+
+The history is stored in the `tag_history` table of the Hopsworks metadata database, one row per
+transition: the value became current, or it stopped being current. A value change writes both at the
+same instant, so one interval's end is the next one's start.
+
+It is read with SQL rather than through the tag APIs, which continue to return the current value.
+On a cluster with the `hopsworks_analytics` project enabled, its Superset connection can query the
+table directly, and
+`create_tag_history_dashboard.py` in the `okr-dashboards` repository builds a "Tag Lifecycle"
+dashboard over it: time spent in each state, whether that is increasing, what is in each state now,
+and what has been in one state longest.
+
+The table stores events rather than intervals. To get `added_on` and `removed_at`, take the next
+event's time for the same artifact, tag and key:
+
+```sql
+SELECT e.artifact_type, e.artifact_id, e.tag_name, e.tag_key, e.tag_value,
+       e.added_on, e.removed_at
+FROM (
+  SELECT artifact_type, artifact_id, tag_name, tag_key, tag_value, event_type,
+         event_time AS added_on,
+         LEAD(event_time) OVER (
+           PARTITION BY artifact_type, artifact_id, tag_name, tag_key
+           ORDER BY event_time,
+                    CASE WHEN event_type = 'CLOSED' THEN 0 ELSE 1 END,
+                    id
+         ) AS removed_at
+  FROM   hopsworks.tag_history
+) e
+WHERE e.event_type = 'OPENED'
+```
+
+Two details in that query are easy to get wrong and produce numbers that look reasonable:
+
+- The `OPENED` filter has to be in the outer query. SQL applies `WHERE` before window functions, so
+  filtering inside would hide every `CLOSED` row from `LEAD`, and anything that ended without a
+  successor, a detached tag or a deleted artifact, would report as still current with its duration
+  growing forever.
+- The ordering has to put `CLOSED` before `OPENED` at the same timestamp. Both halves of a value
+  change share one `event_time` by design, so the ordering needs a tie-break, and `id` is not one:
+  rows are not written in the order the two halves were built.
+
+A `removed_at` of `NULL` means the artifact is still in that state. An `added_on` of `NULL` means the
+tag was attached before Hopsworks began recording attachment times, so the start is unknown; it is
+left empty rather than filled with a guess.
+
+The history outlives what it describes. Deleting the artifact, the tag schema or the project closes
+the open intervals and keeps the rows, so a report over a past quarter still returns what was true
+then.
+
 ## Step 3: Search
 
 Hopsworks indexes the tags attached to feature groups, feature views, training datasets, jobs, models and deployments.