Remove unsupported plugins and centralize HTTP fetching - #146
Merged
Conversation
The plugin set shipped with the gem was written between 2012 and 2015 and had not been reviewed against the services and libraries it talks to. Each of the forty-five plugins was checked against the current state of its service, its API and its gem, and the set was reduced to what still has a use and made to work on Ruby 3.3 through 4.0. Removed eleven plugins whose service or API no longer exists: the three Twitter plugins, both Pocket plugins, HipChat, Google Calendar, livedoor Weather, So-net G-Guide and Chan-Toru, and Google News link rewriting. Their specs, integration recipes and optional dependencies went with them. A Recipe naming one now fails at load rather than at runtime, and doc/PLUGINS.md section 8 records each reason. Migrated PublishAmazonS3 and the S3 path of StoreFile from AWS SDK for Ruby v1 to aws-sdk-s3, keeping the Recipe keys and adding an optional region; both now also accept the SDK's own credential chain, so a bucket can be reached without a secret in the Recipe. Dropped xml-simple from CustomFeedSVNLog in favour of REXML, which the framework already depends on, and nkf from FilterDescriptionLink, whose parser detects an encoding for itself. PublishHatenaBookmark remains Needs rework -- the service and its API are current, the WSSE interface it speaks is not -- but no longer sends a password digest over plain HTTP. Corrected defects the review turned up: `interval` was never waited, because the line meant to do it assigned to a local variable named sleep; FilterDescriptionLink read neither of its settings, because it tested for Hash and a Recipe produces a Hashie::Mash; FilterAbsoluteURI treated an https link as relative; PublishMemcached could not take a port written as a number; FilterTumblrResize and FilterImage did not recognise the URL shapes those services now use. Added Automatic::Http, the one way in for what plugins fetch, so that the scheme allowlist, timeouts and redirect limit are decided once rather than by omission in seven plugins. A link in a pipeline item comes from a feed, and URI.open on such a string would read a local file as readily as an article. Every Supported plugin now has deterministic local tests that reach no network and need no credential, and a spec holds doc/PLUGINS.md and README to the files in plugins/, so a catalogue entry or a count cannot drift. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PpGaCj366pPvFCa3JX4Dh5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change removes plugins for services that no longer exist or have deprecated APIs, and introduces a centralized HTTP fetching utility to improve security and consistency across the codebase.
Summary
The branch modernizes the plugin ecosystem by removing 11 plugins that depend on defunct services or deprecated APIs, while introducing
Automatic::Httpas a single point for all HTTP fetching in plugins. This improves security by restricting URL schemes to HTTP/HTTPS and enforcing timeouts.Key Changes
Removed Plugins (Services No Longer Available):
PublishGoogleCalendar- Google Calendar GData API v2 and ClientLogin were withdrawnPublishTwitter,SubscriptionTwitter,SubscriptionTwitterSearch- Twitter API changesPublishPocket,SubscriptionPocket- Service deprecatedPublishHipchat- Service shutdownSubscriptionGGuide,SubscriptionChanToru,SubscriptionWeather- Services no longer availableFilterGoogleNews- Google News API deprecatedNew HTTP Utility:
lib/automatic/http.rbwithAutomatic::Httpclass providing:Http.read(url)- fetch URL body with validationHttp.uri(url)- parse and validate URIHttp.fetchable?(url)- check if URL can be fetched without raisingfile://attacks)Updated Plugins to Use New HTTP Utility:
FilterFullFeed,FilterImageSource,FilterDescriptionLink- removed directopen-uricallsSubscriptionTumblr,SubscriptionFeed,SubscriptionLink,SubscriptionXml- use centralized HTTPCustomFeedSVNLog- replacedxml-simpledependency withrexmlPublishInstapaper,PublishHatenaBookmark- useNet::Httpdirectly with proper HTTPSDocumentation Updates:
doc/PLUGINS.mdwith new section 3.8.1 explaining HTTP fetching best practicesxml-simple/xmlsimpletoactiverecord/active_recorddoc/POLICY.mdto reflect removal of unsupported pluginsTest Updates:
spec/doc/plugins_catalogue_spec.rbto verify plugin documentation consistencyDependency Changes:
nkfgem dependency (no longer needed)xml-simplegem dependency (replaced withrexml)rexmlandrssas explicit dependenciesNotable Implementation Details
Automatic::Httputility escapes and re-parses URLs with invalid characters (spaces, non-ASCII) rather than raising, improving robustnessfile:///etc/passwdwhen processing untrusted feed linksPublishHatenaBookmarkmarked as "Needs rework" but transport corrected to HTTPS to prevent password digest exposurePublishAmazonS3updated to handle S3 schemes (s3,s3n) separately from HTTPhttps://claude.ai/code/session_01PpGaCj366pPvFCa3JX4Dh5