docs(learn): add Administration production reliability track - #671
Draft
Ethan-Arrowood wants to merge 7 commits into
Draft
Ethan-Arrowood wants to merge 7 commits into
Ethan-Arrowood wants to merge 7 commits into
Conversation
Ten Learn > Administration guides derived from Jeff's "Operating Reliable Harper Applications" field guide, drafted by Aleks Haugom with Claude and moved into the repo largely as delivered. Replaces the Administration "Coming Soon" placeholder and redirects its old URL to the first guide. Guides, in sidebar order: 1. How Harper Runs in Production 2. Health Checks and Traffic Admission 3. Sizing a Harper Cluster 4. Operating Replication 5. Monitoring and Triage 6. Safe Deployments and Rollback 7. Backup and Recovery 8. Engineering RPO, RTO, and Uptime 9. Production Readiness Checklist 10. Reliability Plan Template Only content change from the delivered drafts: the get_components link in guide 1 pointed at an anchor that does not exist on the operations page; it now points at components/applications#get_components. Co-Authored-By: Aleks Haugom <aleks@harperdb.io> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Ethan-Arrowood
added
the
content
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive set of administration guides and templates under the learn/administration/ directory, covering topics such as backup and recovery, replication, health checks, monitoring, and safe deployments, alongside updating the corresponding redirect rules. The review feedback focuses on improving documentation formatting and consistency, specifically recommending that critical caveats and limitations be wrapped in :::caution blocks, mid-sentence version badges be replaced with plain text, and API response fields in code examples use standard snake_case (e.g., renaming writtenAt to written_at).
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
Technical verification pass over the ten Administration guides against harper and harper-pro at origin/main (v5.2.12 current). Most of the track held up; these are the claims that did not, plus the reference fixes they turned up. Guides: - Rewrite "What restart actually does" for released 5.2 behavior. The awaited `restart: true` is v5.3.0 only, so the section now describes the fire-and-forget behavior operators actually have, with the wait moved to its own subsection badged `changed in v5.3.0`. - Warn that a replicated deploy with `restart: true` restarts every node at once, and prefer `rolling` on a cluster. Matches the guidance already in the shipped CI/CD guide. - Document that `replicated` is opt-out: omitting it replicates. - Correct the new-node row. A joining node requests a full copy from every peer it has no resume cursor for, and established peers full-copy from it in turn, so a join into N peers is N inbound plus N outbound transfers. - `system` is in replication scope by default, so users and roles already propagate unless someone narrowed it. - `drop_database` and `drop_table` replicate by default; `drop_attribute` does not. - `get_backup target=` writes a file; it does not clone a database. - Blob snapshots are hard-linked, so they are near-free on the default layout and a full copy only across filesystems. The transaction-log snapshot is the part that genuinely costs disk every backup. - `verify_backup` checks that a blob snapshot is present, not that blob contents are intact. - The availability flag is already `replicate: false` in both mechanisms, so the guidance is to preserve that rather than to go establish it. - `/status` returns a body as well as a status code, and the component entry belongs in the root harper-config.yaml, not an application's config.yaml, where it would replace the default component config. - Readiness sample: a missing record resolves rather than throws, so check the result; import from 'harper'; and note the route is unauthenticated because a static handler bypasses the authorization gate. - Logging sample: there is no node-name environment variable, so use server.hostname. `context.requestId` needs http.logging.id, which is off by default. Harper's logger renders text via Node's Console, not JSON. - Badge `write-transaction-queue-depth` v5.2.0, and note the Prometheus exporter does not cover the saturation, commit-time, or storage metrics. Reference: - State the `replicated` default across the component operations; it was the only optional parameter whose default went unstated. - `drop_database`/`drop_table` replicate by default; only `drop_attribute` is per-node. - Users and roles do propagate by default, which contradicted both the code and this file's own statement that all databases replicate by default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
Follow-on from the Administration verification pass. Each of these was checked against harper/harper-pro at origin/main and is wrong on the live site today. - `set_status` / `get_status` / `clear_status` are not in-memory. They persist to the `hdb_status` table in the `system` database, which is declared `replicate: false`, so values survive a restart and stay node-local. Adds what `restartRequired` on a bare `get_status` actually tracks. - `system_information` no longer advertises a `replication` attribute, which does not exist. Lists the ten real attribute names and notes that unrecognized names are dropped silently rather than rejected, so a typo returns a smaller response instead of an error. - The `cluster_status` sample showed `lastSendTime`, a field present in neither repo, and `thread_id` where the code emits `threadId`. Also notes that the timing fields hold the literal string "Copying" during an initial full copy, which breaks anything parsing them as dates. - `logging.console` defaults to `false`, not `true`. The two reference pages disagreed; the runtime default is `logOptions.console ?? false`. Core's own config schema description carries the same stale "Default: true" and should be fixed separately. - `get_backup target=` reads a snapshot from another node into a local file. It does not clone that node's database onto the current one. - `verify_backup` confirms a recorded blob snapshot is present and fails the backup as corrupt when it is missing, but does not check individual blob files, so verification can pass on damaged blob contents. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
Read Jeff's "Operating Reliable Harper Applications" doc and compared it against the ten guides. Coverage was broadly good, but three documented capabilities were dropped entirely and one CLI claim was wrong. - `get_backup` has no offline CLI form. Guide 7 told readers every backup operation works with the server stopped; `get_backup` is absent from the offline command switch and fails as an unknown command. That matters because a stopped node is exactly when an operator reaches for it. - `storage.writeAsync` was missing from the RPO guide. It defaults off, and turning it on disables fsync so acknowledged commits can be lost on power loss or an OS crash. Nothing else in the track surfaces it: backup timestamps still look right and no alert fires, so a stated recovery point can be quietly false. Now framed as an RPO decision with the cases where enabling it is defensible. - `threads.preload` and `threads.preloadRequire` were missing from the observability guide. They are how an APM or tracing agent loads ahead of the code it instruments. Includes the split-entry case, which fails confusingly: with only the loader hooks preloaded a tracer emits no-op spans and exports nothing, so instrumentation looks installed while the collector stays empty. - `HARPER_SAFE_MODE` was missing from the backup guide. Guide 7 explains that a component holding a database open blocks an online restore but never mentioned the escape hatch. Safe mode starts Harper without loading components, so the restore can run, and it is also how you reach a node a broken component stopped. Also corrects the safe mode example in the configuration reference, which invoked `harperdb` rather than `harper`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
Three of the seven review comments were worth taking. - Move `<VersionBadge>` out of prose and list items in guide 6, per CONTRIBUTING's "reserve VersionBadge for standalone placement after headings". Uses the repo's own inline form, `(Added in: vX.Y.Z)`, rather than the bare `(v5.2.0)` the review suggested. Also applies it to the two list-item badges the review did not flag, so the guide is internally consistent. Table-cell badges are left alone, which is established practice across the reference. - Promote the two managed-backup limitations to an admonition, since a database on per-table storage paths silently has no backup strategy at all. Uses `:::warning` rather than `:::caution` to match the dominant type in this repo. The other four were declined; see the PR threads for reasoning. In short: the snake_case suggestion would have made the sentinel example inconsistent with every shipped Learn guide, and the three remaining suggestions wrapped multi-paragraph body prose in admonitions, one of them quoting text that had already been corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
Ran the PR diff through Gemini (gemini-3.1-pro-high) file by file. Of 52 findings, 12 held up against the harper/harper-pro sources; the rest were contradicted by code I had already verified. The ones that held up cluster almost entirely in internal consistency, which is what a diff-only reviewer can actually check. Bugs this pass caught, several of my own making: - `HARPER_SAFE_MODE=1 harper` does not start the server. Bare `harper` falls through to the CLI-operations branch; starting it needs `harper run`. I introduced this when correcting the reference's `harperdb`. - The drain sequence checked peer capacity at step 3, after the node was already drained and its traffic had moved. Moved the check before the drain and added the abort path, which was missing entirely. - The readiness sample could never pass. After I changed it to test the sentinel record, nothing told the reader to create one, so the probe would return 503 forever. Added the seeding step. - Guide 8 contradicted itself on recovery time: one place measured from detection, another counted a detection gap inside the total. Settled on recovery time starting when the journey breaks, detection included. - The availability flag does not stop replication, so the restore sequence implied an isolation it never provided. An online restore on a drained but still-connected node exposes rolled-back data to peers before validation. Isolation is now its own step with a danger note. - `system_information` reports node and npm versions, not Harper's. The service inventory now points at `registration_info`. - The sizing sample requested neither disk nor network while the prose told the reader to watch both. - A Prettier wrap left a link orphaned as its own paragraph, stranding the sentence that followed it. - Smaller: "point-in-time restore" implied continuous PITR rather than restoring a snapshot; two checklist items still named `config.yaml` after the prose moved to the root `harper-config.yaml`; the status types are fixed while only their values are application-defined; and one leftover opt-in phrasing sat next to the corrected opt-out default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
The ten guides arrived hard-wrapped at 80 columns, which nothing else in the repo does: existing Learn and reference pages run one line per paragraph, with prose lines routinely past 500 characters. Prettier is configured without proseWrap, so it preserves whatever it is given and never enforced either style. Wrapping is undone across all ten. Paragraphs and list items are now one logical line each. Code fences, tables, frontmatter, JSX, admonitions, headings, blockquotes, and import lines are untouched, and a content fingerprint was compared before and after each rewrite to confirm nothing but line breaks changed. Verified per file: identical structural element counts, identical token streams, and a clean build. Also carries an edit to the opening paragraph of how-harper-runs-in-production.mdx that was made by hand in the worktree, not by the rewrap. It is the one token-level change in the diff. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ten new guides for Learn > Administration, replacing the "Coming Soon" placeholder that has sat there since the Learn tab launched.
The content is derived from a google doc created by Jeff. Aleks ran that guide through Claude to produce these drafts, and I have moved them into the repo as close to his delivered files as possible so we can iterate here rather than in a zip. Aleks is a co-author on the commit.
This is a draft on purpose. The drafts are AI-written and the code samples are unverified. It needs a technical pass from someone who knows the material before it merges.
The guides
In sidebar order (set via
sidebar_position; the Administration category is alreadyautogenerated, so no sidebar edit was needed):Guides 1 to 8 each end with a scoped
## Readiness checklist; guide 9 aggregates them and guide 10 is the fill-in document that holds the answers.What changed relative to Aleks's drop
learn/administration/coming-soon.md./learn/administration/coming-soonto the first guide, matching the existing/learn/developers/coming-soonrule.get_componentslink pointed atoperations-api/operations#get_components, which does not exist. It now points atcomponents/applications#get_components.Nothing else in the prose or code samples was touched.
Verification
npm run format:checkis clean (the drafts were already formatted with the repo's Prettier config).npm run buildpasses with zero broken links and zero broken anchors.Open questions for reviewers
These come from Aleks's review notes and from Nathan's code-level review of the original field guide in the Slack thread. The drafts were written to sidestep them ("verify on your own cluster" rather than asserting a default), but the docs still need one answer for each.
systemdatabase replicate by default?reference/replication/overview.mdsays users and roles are not replicated by default becausesystemis out of the default scope. Nathan's code evidence sayssystemreplicates by default and only the tables inNON_REPLICATING_SYSTEM_TABLESopt out, which does not includehdb_userorhdb_role. Guides 1 and 4 tell the reader to checkcluster_statusrather than assert either.9933a real replication port default? The reference says so; Nathan says it appears in core only in a schema description and a DockerfileEXPOSE, and the live fallback isREPLICATION_SECUREPORT ?? OPERATIONSAPI_NETWORK_SECUREPORT. Guide 1 presents it as a documented default to confirm, not a guarantee.set_statusdurability. The reference describes it as in-memory and not persisted across restarts. Nathan's evidence says it writes to a realsystem.hdb_statustable withreplicate: false. Guide 2 omits any persistence claim and steers traffic admission to@harperdb/status-checkinstead.Readinessresource in guide 2, the convergence sentinel in guide 4, and thelogger.infocall in guide 5. Treat every code block as a draft until someone has run it.Follow-ups outside this PR
Two things in
reference/operations-api/operations.mdappear to be wrong on the live site today and should be filed as Bug issues independently of this track:set_statusis described as "(in-memory)" and "not persisted across restarts" (around lines 1346 and 1376). Per the code it persists tosystem.hdb_status.system_informationexample lists"replication"as an attribute (around line 1368). There is no such attribute, and unknown names are silently dropped, so the example returns less than it appears to.Sidebar ordering.
plans/learn-section-outline/plans ten per-feature Administration guides (Running Locally, Containers, Logging, Certificates, and so on) that have not been created yet. This track takessidebar_position1 to 10. When those land they should take 20 and up, or this track could move into anadministration/production/subdirectory with its own_category_.jsonif a nested "Running in Production" group reads better.🤖 Generated with Claude Code