Fix FilterFullFeed to handle HTTPS/HTTP scheme mismatches and encoding - #149
Merged
Conversation
The filter did nothing on almost every feed, and where it did something it could make the item worse than it found it. Three separate causes, all of them consequences of the shipped siteinfo being a 2013 snapshot of a database that can no longer be refreshed. A link was matched against a record's pattern exactly as the feed gave it. 3,448 of the 3,504 usable records anchor on a scheme and all but twenty of those say `^http://`, while the sites they name have since moved to HTTPS, so an ordinary feed today matched nothing at all and every item was logged as "Fulltext SITEINFO not found". A record describes a site's layout, not how it is transported, so the link is now tried under either scheme; only the match is rewritten, and the page is still fetched from the link the feed gave. Against the shipped database, https links to gigazine.net, blog.livedoor.jp and webcre8.jp reach their records again instead of missing. Where a record did match but its XPath selected nothing -- a site redesigned since the XPath was written -- the empty result was assigned anyway, so the item lost the summary it arrived with and the feed went out with an empty body. The summary is now kept, and the miss is logged at warn with the XPath that missed. Page encoding was read from a decoded string. `Automatic::Http.read` returns what open-uri has already applied an encoding to whether or not the response declared one -- a page served as `text/html` with no charset comes back tagged UTF-8 because that is the fallback, not because the page said so -- and a parser given that string believes the tag and never looks at the meta charset underneath it, which is a whole article in mojibake for the many sites in this database that declare their charset only in a meta tag. The page is now handed to the parser as a stream, through `Automatic::Http.open`, so the parser detects the encoding for itself. A record's own `enc` is the fallback for a page that declares nothing anywhere; 1,186 records carry one, mostly EUC-JP and Shift_JIS, and an `enc` naming an encoding Ruby does not have is ignored rather than raised. The result is converted to UTF-8 with invalid bytes replaced as well as undefined ones, because converting UTF-8 to UTF-8 leaves invalid bytes alone unless they are named, and what leaves here goes on to a publish plugin that cannot recover from a string it cannot encode. Records are now compiled when the file is loaded rather than per item, which was several thousand `Regexp.new` calls for every link in every feed, and a record with no URL pattern, no XPath or a pattern that is not a regular expression is dropped there -- an empty pattern matches every link, so such a record would have put its own XPath over the whole feed. The spec reached the network for the one case that mattered and pointed at a host that has not served what it expected for years, so none of the above was covered. Sixteen local examples cover it now: they write a siteinfo file into a temporary HOME and stand in for open-uri, and each of the fixes fails at least one of them when reverted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RSLxro7DeKVdunYdYMyjvY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Significantly improves the FilterFullFeed plugin to work with modern HTTPS feeds against a 2013-era siteinfo database, properly handle character encodings, and gracefully handle edge cases where XPath selectors no longer match redesigned sites.
Key Changes
Scheme-agnostic matching: Records anchored on
http://now matchhttps://links and vice versa, since the database describes site layouts rather than transport protocols. 3,448 of 3,504 records anchor on schemes, nearly all^http://, but feeds now deliver HTTPS links.Encoding handling improvements:
encfield is used as a fallback for pages declaring no charset anywhereGraceful XPath failures: When an XPath selector matches nothing (site redesigned), the original feed summary is preserved instead of replacing it with an empty description. Misses are logged at warn level.
Performance optimization: Siteinfo records are now compiled into
Entrystructs with pre-compiled regex patterns, eliminating thousands ofRegexp.newcalls per feed run.Data validation: Records with missing URL patterns, missing XPath, or invalid regex patterns are filtered out during load rather than failing at match time. Empty patterns would otherwise match every link.
API enhancement: Added
Automatic::Http.open(url)to complementHttp.read(), allowing callers that need stream-level control (like HTML parsers) to avoid pre-decoded strings that interfere with charset detection.Implementation Details
Entrystruct holds compiled patterns and metadata for efficient matchingmatch()method tries links under both HTTP and HTTPS schemesdocument()method carefully manages encoding detection: response charset > meta tag > record fallbackbody()method returns nil for empty XPath results, allowing callers to distinguish "nothing found" from "found empty content"https://claude.ai/code/session_01RSLxro7DeKVdunYdYMyjvY