fix(news): derive titles from URL slug for MSN/aggregator articles#61
Merged
Conversation
MSN.com pages are JS SPAs whose server-side HTML <title> is literally 'MSN'. Both DDG and trafilatura return 'MSN' as the title for all msn.com articles. Changes: - ddg_news.py: add derive_title_from_slug() + is_generic_title() helpers - pipeline.py: _enrich_one now guards against generic fetched titles (Client Challenge, MSN, etc.) — won't overwrite a good DDG title with a bad fetched one, and derives from URL slug when both are generic - backfill_news_titles.py: one-shot script to fix 156 existing MSN titles The URL slug is the only source of the real headline for MSN articles (e.g. 'venezuela-quakes-caused-196-billion' -> 'Venezuela Quakes Caused 196 Billion'). Imperfect formatting but far more informative than 'MSN'.
nullhack
added a commit
that referenced
this pull request
Jul 24, 2026
Dashboard data regenerated with fixed news titles. 156 MSN-titled articles now show headline derived from URL slug instead of 'MSN'.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
155 of ~1700 stored news items have
title: MSN— the single most common title. MSN.com is a JS SPA whose server-side<title>tag is literally"MSN". Both DDG and trafilatura return this generic site name instead of the actual headline. Also 3 items havetitle: Client Challenge(Cloudflare bot-detection pages overwriting good DDG titles during enrichment).Root cause
Neither a capture failure nor a wrong-field issue — the headline exists only in the URL slug (e.g.
.../venezuela-quakes-caused-196-billion-in-damage/...), never in any title metadata field.Fix
ddg_news.py:derive_title_from_slug(url)extracts the headline slug from MSN-style URLs (/slug/ar-XXXor/slug/vi-XXX) and converts to title case.is_generic_title(title)detects known bad titles (MSN, Client Challenge, etc.).pipeline.py:_enrich_one: rewritten title resolution — prefers good fetched title, rejects generic fetched titles (won't overwrite a good DDG title with 'Client Challenge'), derives from slug when both DDG and fetched titles are generic.backfill_news_titles.py: one-shot backfill script for the 156 existing MSN-titled YAML files.Samples (before → after)
Known imperfections: decimals flatten (7.3→73), currency drops ($5M→5m), proper nouns lose casing (DRC→Drc). Still far more informative than 'MSN'.
Tests
7 new tests (3
derive_title_from_slug+ 4is_generic_title). All gates green: ruff, pyright, mypy.stubtest, full suite.