Skip to content

fix(preprocess): recover markdown-wrapped ticker hints#109

Merged
keli-wen merged 2 commits into
masterfrom
codex/fix-news-ticker-markup
Jul 15, 2026
Merged

fix(preprocess): recover markdown-wrapped ticker hints#109
keli-wen merged 2 commits into
masterfrom
codex/fix-news-ticker-markup

Conversation

@keli-wen

@keli-wen keli-wen commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Normalize Markdown link and */** emphasis decoration on a scan-only copy before extracting exchange-qualified ticker hints.
  • Preserve stored Markdown, content_hash, and document identity semantics.
  • Add deterministic markup fixtures, including the dominant multiline (NASDAQ:\n\n[PODD](...)) representation, plus a bounded live recall control that samples up to 25 PR Newswire articles with an independent oracle.
  • Keep multi-symbol list grammar and exchange-whitelist expansion out of scope; follow-up discussion is tracked in bug: exchange-qualified multi-symbol lists are only partially captured #108.

Root Cause and Controls

The extractor tests only covered ideal plain text while the article boundary supplies cleaned Markdown. The existing live verifier stopped after RSS and listing discovery, so it never exercised article representation or ticker-hint recall.

This change adds two executable controls: an offline fixture gate requiring full recall for single-line links, multiline links, bold, and italic forms without mutating stored content; and a daily live gate requiring 100% recall for the first supported exchange-coded symbol in each sampled group. A successfully parsed sample with zero supported symbols reports SKIP and passes neutrally, avoiding false failures on thin-volume windows.

Related Issue

Closes #107

Verification

  • bash scripts/verify.sh — 291 tests passed; 88.55% coverage; format, lint, typing, and import contracts passed.
  • python scripts/verify_news_e2e.py — 1,102 observations discovered; 25/25 articles parsed; 15/15 expected ticker hints recovered (100% recall).
  • python examples/preprocess/01_news_pr_wire.py — stored Markdown retained and CCL extracted.

Checklist

  • The title uses English Conventional Commit format: type(scope): summary.
  • The related issue or design discussion is linked when applicable.
  • bash scripts/verify.sh passes.
  • Every applicable live component gate passes, or this PR states why none applies.
  • Public behavior has focused tests, an example, and documentation where applicable.
  • The PR is complete, small, and contains no unrelated changes.

@keli-wen keli-wen added type: bug Existing behavior violates its contract or documented expectation area: preprocess Deterministic acquisition, parsing, cleaning, and source handling area: harness Contributor and agent controls, CI, hooks, skills, and rulesets impact: live-network Changes or depends on real public-network behavior or smoke tests labels Jul 15, 2026
@keli-wen keli-wen self-assigned this Jul 15, 2026
@keli-wen
keli-wen marked this pull request as ready for review July 15, 2026 02:33
@keli-wen
keli-wen merged commit aafa890 into master Jul 15, 2026
3 checks passed
@keli-wen
keli-wen deleted the codex/fix-news-ticker-markup branch July 15, 2026 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: harness Contributor and agent controls, CI, hooks, skills, and rulesets area: preprocess Deterministic acquisition, parsing, cleaning, and source handling impact: live-network Changes or depends on real public-network behavior or smoke tests type: bug Existing behavior violates its contract or documented expectation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: markdown-link-wrapped exchange tickers (e.g. (NYSE: [CCL](#financial-modal))) are silently dropped by extract_exchange_ticker_hints

1 participant