feat(semaphore-sync): two-tier tag sync + scheduler mislog fix - #2480
Merged
Merged
Conversation
…e) + fix mislog
Rework the semaphore-tag-sync applier so a weekly run adopts the existing
tag corpus without importing all ~20k SES terms, and picks up genuinely
new tags only under an explicit class allowlist.
Tier 1 (adopt-only): match each SES term by semaphoreId, then by name.
Backfill taxonomy fields (label/name/titlePath/semaphoreId) in place when
they differ; NEVER overwrite editorial flags (isActualTag/isInterestItem).
Tier 2 (gated intake): INSERT an unmatched term only when its SES class is
in semaphore.sync.intakeClasses (case-insensitive, URI-leaf tolerant).
New rows land inert (flags forced false), awaiting editor curation. Empty
allowlist = intake OFF (adopt-only), the safe default.
Dry-run summary now carries a pre-formatted classHistogram string so the
real SES class distribution can be read from an in-app dry run and used to
choose semaphore.sync.filter / intakeClasses from data. dryRun defaults TRUE.
Also fixes a scheduler mislog: a job fn RETURNING { ok:false } (rather than
throwing) was recorded as SUCCESS. The chassis now inspects the return value,
logs a FAILED PipelineLog row, records JobLastRun.lastErrorAt, and raises an
alert. Covered by new admin-job-controls tests.
Config keys (all ImsConfig, DB-driven -- no env vars):
semaphore.sync.intakeClasses (new), .dryRun, .filter, .model, .lang,
.actualTagClasses, .interestItemClasses
Refs #2184, #2478
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two-tier rework of the
semaphore-tag-synccron (#2184) so the weekly run adopts the existing tag corpus without importing all ~20k SES terms, and picks up genuinely new tags only under an explicit class allowlist. Plus a scheduler mislog fix (#2478).Tier 1 — adopt-only (existing rows)
semaphoreId, then byname(adopts legacy rows lacking asemaphoreId).label/name/titlePath/semaphoreId) in place when they differ.isActualTag/isInterestItem).Tier 2 — gated intake (unmatched rows)
semaphore.sync.intakeClasses(case-insensitive, URI-leaf tolerant).false, awaiting editor curation.Dry-run histogram
Dry-run summary now carries a pre-formatted
classHistogramstring (e.g.SoftwareProduct=1, IndustryCluster=1).formatJobSummaryonly renders scalar fields, so it's a string on purpose. This is the point of the first in-app dry run: read the real SES class distribution and choosesemaphore.sync.filter/intakeClassesfrom it.dryRunstill defaults TRUE.Scheduler mislog fix (#2478)
A job fn that returns
{ ok:false }(rather than throwing) was recorded as SUCCESS — the "no new tags, but green" symptom. The chassis now inspects the return value: logs a FAILEDPipelineLogrow, recordsJobLastRun.lastErrorAt+ message, and raises an alert.Config (all
ImsConfig, DB-driven — no env vars)semaphore.sync.intakeClasses(new),.dryRun,.filter,.model,.lang,.actualTagClasses,.interestItemClassesTests
48 passing across 3 suites (
semaphore-sync-applier,semaphore-tag-sync-job,admin-job-controls), including new coverage for the mislog branch (ok:false→lastErrorAtset / FAILED log; generic message fallback;ok:true→ SUCCESS) and the dry-run histogram + gated intake.Not in this PR (tracked in #2479, Tom to drive)
Run an in-app dry run against prod SES → read histogram → decide server FILTER vs
intakeClasses+ which class(es) → set config in prod DB → validate with a second dry run → flipdryRun=false. Rotate the compromised API key.Refs #2184, #2478