Remove unsupported plugins and centralize HTTP fetching

Gemfile

@@ -54,11 +54,6 @@ group :plugins, :sanitize, optional: true do
   gem 'sanitize'
 end
 
-# FilterDescriptionLink, which normalizes a fetched page's encoding with it.
-group :plugins, :nkf, optional: true do
-  gem 'nkf'
-end
-
 # The autodiscovery and inspect subcommands. No plugin and no Recipe uses it.
 group :plugins, :autodiscovery, optional: true do
   gem 'feedbag', '>= 1.0', '< 2.0'
@@ -77,10 +72,9 @@ group :fluentd, optional: true do
   gem 'fluent-logger'     # PublishFluentd and ProvideFluentd, with a Fluentd instance
 end
 
-group :svn_log, optional: true do
-  gem 'xml-simple'        # CustomFeedSVNLog, with the svn command
+group :s3, optional: true do
+  gem 'aws-sdk-s3'        # PublishAmazonS3 and the s3 path of StoreFile, with a bucket
 end
 
-# PublishAmazonS3 and the s3n:// path of StoreFile call AWS::S3, which only
-# AWS SDK for Ruby v1 provided. No currently published gem satisfies them, so
-# there is no group to select; they need rework. See doc/PLUGINS.md.
+# CustomFeedSVNLog needs the svn command and no gem: it reads `svn log --xml`
+# with REXML, which is a runtime dependency of the framework already.

README.md

@@ -93,15 +93,16 @@ second user.
 
 The project began in February 2012 and this is the first release since 2015. The
 core, the Recipe format and the plugin contract are unchanged; what has changed
-is that it runs on a current Ruby, installs from a current RubyGems, and says
-plainly which of its plugins still work. See [`doc/VERSIONS`](doc/VERSIONS).
+is that it runs on a current Ruby, installs from a current RubyGems, and ships
+a plugin set every part of which still has somewhere to talk to. See
+[`doc/VERSIONS`](doc/VERSIONS).
 
 ## 2. Features
 
 - **Recipes in YAML.** A job is a file, not a program. No Ruby is written to
   wire a pipeline together.
-- **Forty-five plugins** across seven categories: subscribe, custom feed,
-  filter, store, provide, notify, publish.
+- **34 plugins** across seven categories: subscribe, custom feed, filter,
+  store, provide, notify, publish — and every one of them has a current use.
 - **Markdown out of the box.** `PublishMarkdown` writes the result as a plain
   Markdown document, to a file or to standard output, with no service and no
   credential behind it. It is the natural end of a new Recipe.
@@ -118,9 +119,10 @@ plainly which of its plugins still work. See [`doc/VERSIONS`](doc/VERSIONS).
   the framework: `gem install automatic` brings four pure-Ruby gems and the
   command, and installs neither an HTML parser nor a database — let alone an
   AWS SDK.
-- **Honest about what is broken.** Every plugin is classified, with its reason,
-  in [`doc/PLUGINS.md`](doc/PLUGINS.md). Nothing dead is stubbed into looking
-  alive.
+- **No museum.** Every plugin is classified, with its reason, in
+  [`doc/PLUGINS.md`](doc/PLUGINS.md). Nothing dead is stubbed into looking
+  alive, and an integration whose service has gone is removed rather than
+  kept as a fossil.
 
 ## 3. Architecture
 
@@ -367,26 +369,29 @@ like a shipped plugin replaces it.
 
 ### Which plugins still work
 
-Forty-five plugins ship with the gem, most of them written between 2012 and
-2015. Several talk to services that have since shut down. Every one is
-classified in [`doc/PLUGINS.md`](doc/PLUGINS.md) section 6, with its settings
-and the reason for its status:
+34 plugins ship with the gem. Every one is classified in
+[`doc/PLUGINS.md`](doc/PLUGINS.md) section 6, with its settings and the reason
+for its status:
 
 | Status | Count | Meaning |
 | --- | --- | --- |
 | **Supported** | 23 | Works on the supported Rubies with current dependencies |
-| **Supported (external)** | 9 | Works, but needs something you provide: a service, a command, a data file |
-| **Needs rework** | 3 | The service exists; this plugin speaks a replaced interface |
-| **Unsupported** | 10 | The service has shut down |
+| **Supported (external)** | 10 | Works, but needs something you provide: a service, a command, a credential, a data file |
+| **Needs rework** | 1 | The service exists; this plugin speaks a replaced interface |
 
-Restoring one of the three in **Needs rework** — `FilterGoogleNews`,
-`PublishAmazonS3`, `PublishHatenaBookmark` — is self-contained work and a good
-first contribution.
+Eleven plugins were removed in this release rather than kept as history: each
+talked to a service that has shut down, or through an API that has been
+withdrawn with no replacement. They are listed with their reasons in
+[`doc/PLUGINS.md`](doc/PLUGINS.md) section 8, and Git history holds the code.
+A Recipe naming one of them now fails at load, before anything runs.
+
+Restoring the one in **Needs rework** — `PublishHatenaBookmark` — is
+self-contained work and a good first contribution.
 
 No plugin here is stubbed, mocked or simulated to make a test pass. Where a
 plugin's gem is not installed its spec is skipped and says which gem is
 missing; where the plugin still loads, its spec covers what does not need the
-service. A dead integration is never made to look alive.
+service. A dead integration is never made to look alive — it is removed.
 
 The contract, a worked example, and how to test a plugin are in
 [`doc/PLUGINS.md`](doc/PLUGINS.md) sections 3 and 4.
@@ -523,7 +528,7 @@ COVERAGE=on bundle exec rake spec
   longer exists is never stubbed into passing.
 - `test/integration/` holds Recipes for exercising plugins against real
   services. They are run by hand, are not part of the suite, and are never run
-  in CI. Most need a credential, a dead service, or both — read one before
+  in CI. Most need a credential or a service you run — read one before
   running it.
 
 The required check installs the bundle, builds the gem, loads the library, runs
@@ -547,7 +552,7 @@ bundle exec rake
 bundle exec bin/automatic -c config/feed2console.yml
 ```
 
-Contributions are welcome — a new plugin, or reviving one of the three that need
+Contributions are welcome — a new plugin, or reviving the one that needs
 rework, most of all.
 
 1. Fork the repository.
@@ -565,8 +570,8 @@ Two rules worth knowing before you start:
 
 - **A gem needed by one plugin is not a dependency of the framework.** Require
   it at the top of the plugin's own file.
-- **Nothing dead is faked.** A plugin whose service has shut down is classified
-  as unsupported, not stubbed into passing a test.
+- **Nothing dead is faked.** A plugin whose service has shut down is removed,
+  not stubbed into passing a test.
 
 - Repository: <https://github.com/id774/automaticruby>
 - Issues: <https://github.com/id774/automaticruby/issues>
@@ -586,6 +591,7 @@ Two rules worth knowing before you start:
 │       ├── log.rb           Levelled logging to standard output
 │       ├── feed_maker.rb    Builds pipeline values from plain data
 │       ├── feed_parser.rb   Fetches and parses feeds
+│       ├── http.rb          The one way in for what plugins fetch
 │       ├── opml.rb          OPML parser, for the opmlparser subcommand
 │       ├── environment.rb   Bundler setup for a source checkout
 │       └── version.rb
@@ -619,7 +625,7 @@ this repository. No document here defers to another repository.
 | [`doc/PLUGIN_DEVELOPMENT.md`](doc/PLUGIN_DEVELOPMENT.md) | A complete user plugin and practical testing guidance |
 | [`doc/REQUIREMENTS.md`](doc/REQUIREMENTS.md) | What the system is for, what it guarantees, where its responsibility ends |
 | [`doc/BASIC_DESIGN.md`](doc/BASIC_DESIGN.md) | How it is composed: the parts, their responsibilities, the flow of a run |
-| [`doc/PLUGINS.md`](doc/PLUGINS.md) | The Recipe format, the plugin contract, and the catalogue of all 45 plugins |
+| [`doc/PLUGINS.md`](doc/PLUGINS.md) | The Recipe format, the plugin contract, and the catalogue of every shipped plugin |
 | [`doc/POLICY.md`](doc/POLICY.md) | How a change is made and judged: style, dependencies, tests, versioning |
 | [`doc/DEPLOYMENT.md`](doc/DEPLOYMENT.md) | Installing, scheduling, operating, and what to do when it fails |
 | [`doc/RELEASING.md`](doc/RELEASING.md) | For maintainers: building, verifying and publishing the gem |

automatic.gemspec

@@ -98,10 +98,10 @@ Gem::Specification.new do |spec|
   # groups, doc/DEPLOYMENT.md and doc/POLICY.md section 9.1.
   #
   # rexml and rss left the standard library and became gems over the 3.x
-  # series, and nkf followed after 3.3. Each gem listed here is listed because
-  # a file in lib/ requires it, and a library's move out of the standard
-  # library is not by itself a reason to declare it: nkf is a plugin's
-  # dependency and is in the Gemfile's optional groups instead.
+  # series. Each gem listed here is listed because a file in lib/ requires it,
+  # and a library's move out of the standard library is not by itself a reason
+  # to declare it: a library only a plugin needs goes in the Gemfile's optional
+  # groups instead.
   spec.add_dependency 'activesupport', '>= 7.1', '< 9.0'   # plugin loader, XML subscription
   spec.add_dependency 'hashie',        '>= 4.0', '< 6.0'   # Recipe
   spec.add_dependency 'rexml',         '>= 3.2', '< 4.0'   # OPML parser

config/feed2console.yml

@@ -1,14 +1,20 @@
+# Print a public feed to the terminal.
+#
+#   automatic -c ~/.automatic/config/example/feed2console.yml
+#
+# The same pipeline as feed2markdown.yml with a different last plugin, which
+# is the plugin to end a Recipe with while you are still writing it. Nothing
+# here needs an account, a credential or a gem beyond the ones
+# `gem install automatic` brings.
+
 global:
-  timezone: Asia/Tokyo
-  cache:
-    base: /tmp
   log:
     level: info
 
 plugins:
   - module: SubscriptionFeed
     config:
       feeds:
-        - http://blog.id774.net/post/feed/
+        - https://www.ruby-lang.org/en/feeds/news.rss
 
   - module: PublishConsole

doc/BASIC_DESIGN.md

@@ -63,7 +63,7 @@ lib/automatic/recipe.rb  lib/automatic/pipeline.rb
         |                   |                  ~/.automatic/plugins/<category>/<name>.rb
         v                   v
 lib/automatic/log.rb   lib/automatic/feed_maker.rb
-        |
+        |              lib/automatic/http.rb
         v
    standard output
 ```
@@ -74,8 +74,8 @@ Dependency points downward, and there is no edge back up:
 - `Automatic::CLI` knows the framework. Nothing in the framework knows the CLI.
 - `Automatic::Pipeline` knows how to find and call a plugin. It knows no plugin.
 - A plugin knows `Automatic::Log`, `Automatic::FeedMaker`,
-  `Automatic::FeedParser` and its own libraries. It knows no other plugin, with
-  one deliberate exception noted in section 4.9.
+  `Automatic::FeedParser`, `Automatic::Http` and its own libraries. It knows no
+  other plugin.
 - `Automatic::Log` and `Automatic::FeedMaker` are leaves. They depend on nothing
   in this repository.
 
@@ -226,9 +226,10 @@ It is a module with state rather than an injected object. That is a consequence
 of plugins calling `Automatic::Log` directly, which keeps a plugin's signature
 to `(config, pipeline)`.
 
-### 4.8 `lib/automatic/feed_maker.rb` and `feed_parser.rb`
+### 4.8 `lib/automatic/feed_maker.rb`, `feed_parser.rb` and `http.rb`
 
-The adapters between "some data" and the pipeline shape.
+The adapters between "some data" and the pipeline shape, and the one way in for
+what is fetched.
 
 - `FeedParser.get_url(url)` fetches a URL and parses it as a feed.
 - `FeedParser.parse_html(html)` builds a feed whose items are the page's links,
@@ -240,8 +241,20 @@ The adapters between "some data" and the pipeline shape.
   arbitrary payload in `content_encoded`, which is the route by which the XML
   subscription plugin feeds the Fluentd provide plugin.
 
-Both use Ruby's bundled `rss` library. That is the reason the pipeline value has
-the shape it has.
+The first two use Ruby's bundled `rss` library. That is the reason the pipeline
+value has the shape it has.
+
+- `Http.read(url)` fetches a URL and returns the body; `Http.uri(url)` returns
+  a validated URI and `Http.fetchable?(url)` answers whether there is one.
+
+`Automatic::Http` exists because the decisions a fetch implies — which schemes
+are allowed, how long to wait, how many redirects to follow, what to send as a
+User-Agent — were being made separately by every plugin that fetched, mostly by
+omission. It is a helper of about twenty lines and not a client: a plugin that
+wants something else calls Ruby directly. The scheme allowlist is the part that
+earns it a file of its own, because a link in a pipeline item comes from a feed
+and `URI.open` on such a string will read a local file as readily as an
+article.
 
 ### 4.9 `plugins/` — `Automatic::Plugin::*`
 
@@ -301,19 +314,15 @@ Its specification — what it writes for each field, how HTML in a body is
 reduced, where the output goes — is in [`PLUGINS.md`](PLUGINS.md) section 6.7,
 because it is a plugin's specification and not a property of the design.
 
-Two shared pieces sit inside `plugins/` rather than in `lib/`, because they are
-plugin implementation and the framework does not use them:
+One shared piece sits inside `plugins/` rather than in `lib/`, because it is
+plugin implementation and the framework does not use it:
 
 - `plugins/store/database.rb` — the `Automatic::Plugin::Database` mixin: opens
   the SQLite database named in the Recipe, creates the table from the including
   class's `column_definition` when it is absent, and provides
   `for_each_new_feed`, which yields only items whose key is not already stored.
   `StorePermalink` and `StoreFullText` are this mixin plus a model and a column
   list.
-- `plugins/subscription/chan_toru.rb` requires `g_guide.rb` and delegates to it.
-  This is the one plugin-to-plugin dependency, it is explicit, and it is not a
-  pattern to copy.
-
 ### 4.10 `db/`, `config/`, `assets/`
 
 Fallbacks inside the installation, used when the corresponding part of the user