feat(odpt): import Toei bus and train feeds absent from the portal API

[ianktc]

Summary:

This PR was created due to the two Toei (Bus and Train) feeds that were absent in the ODPT Integration. This is due to the Toei organization not being included in the ODPT Metadata API, and instead hosted at its own namespace. Both feeds are CC BY 4.0 and should be included in the ODPT integration by statically declaring them.

I referred back to this particular file shared with us by ODPT, and it calls out Toei Bus as an exception, I later found Toei Subway to also follow this exception case:

odpt_metadata_info.txt

Discovered after trying to deprecate mdb feeds with no odpt counterparts: MobilityData/mobility-database-catalogs#1633

Investigated with the help of Claude:

1. Toei is not in the portal response at all. Querying with format=gtfs and license (odpt|ccby4|cc0|other_auth|other_noauth) returns zero mentions of Toei:

   licence	orgs returned	Toei present
   odpt	38	no
   ccby4	34	no
   cc0	3	no
   other_auth	0	no
   other_noauth	3	no

2. The URL template cannot express Toei's path even if it were listed. _fetch_feeds discards the API's own dataresource[].url values and rebuilds each URL from PUBLIC_GTFS_ENDPOINT:

https://api-public.odpt.org/api/v4/files/ odpt/{org_label}/{dataset_label}.zip?date=current

All 528 dataresource URLs across all 75 orgs match that shape, so the template is correct for portal feeds. Toei's is a different namespace:

https://api-public.odpt.org/api/v4/files/ Toei/data/Toei-Train-GTFS.zip

with no odpt segment, a literal data/ element, a filename that is not a dataset label, and no date parameter. No (org_label, dataset_label) pair produces it.

Changes:

• New STATIC_FEEDS constant holding the two Toei entries as ordinary feed items with a literal gtfs_endpoint, bypassing PUBLIC_GTFS_ENDPOINT
• _fetch_feeds's result is bound to portal_feeds and merged in _import_odpt
• The stale-feed sweep now gates on portal_feeds rather than feeds_list. This is the one genuine hazard in the change: STATIC_FEEDS makes the merged list permanently non-empty, which would have silently disabled the existing empty-response guard and allowed a single empty-but-successful portal response to deprecate the entire odpt- catalog except the static feeds

Expected behavior:

An import run creates 2 schedule feeds and 4 GTFS-RT sub-feeds:

stable_id	producer_url
odpt-Toei-ToeiBus	.../files/Toei/data/ToeiBus-GTFS.zip
odpt-Toei-ToeiTrain	.../files/Toei/data/Toei-Train-GTFS.zip
odpt-Toei-ToeiBus-vp	.../gtfs/realtime/ToeiBus
odpt-Toei-ToeiTrain-vp	.../gtfs/realtime/toei_odpt_train_vehicle
odpt-Toei-ToeiTrain-tu	.../gtfs/realtime/toei_odpt_train_trip_update
odpt-Toei-ToeiTrain-sa	.../gtfs/realtime/toei_odpt_train_alert

Subsequent runs are no-ops for these feeds — the existing fingerprint diffing applies to them unchanged. They are never deprecated by the sweep despite being permanently absent from the portal.

Testing tips:

scripts/api-tests.sh --folder functions-python/tasks_executor

Result: all tests pass, branch coverage 84% (threshold 80%). The ODPT file alone is 12 passing tests.

Four new cases in TestStaticFeeds:

• test_static_feeds_are_well_formed — literal gtfs_endpoint, resolvable license_type, unique org/dataset pairs (that pair becomes the stable id), and any declared RT url is a real URL.
• test_static_feeds_are_imported_with_their_literal_urls — driven with an empty portal response so the only feeds processed are static ones; asserts the URLs survive untouched, plus RT rows' producer_url, entity type, and back-link to the schedule feed.
• test_static_feeds_are_marked_as_seen_for_the_stale_sweep — schedule and RT stable ids reach processed_stable_ids; RT sub-feeds match the same odpt- prefix and would otherwise be swept.
• test_static_feeds_do_not_defeat_the_empty_fetch_guard — regression guard for the sweep-gating hazard above: seeds a live feed, runs with an empty portal response, asserts it survives and deprecated == 0.

---

• Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
• Add or update any needed documentation to the repo — no doc changes needed; no public API surface or DB schema touched
• Format the title like "feat: [new feature short description]"
• Linked all relevant issues — no tracking issue known; please link if one exists
• Include screenshot(s) — not applicable, backend import task with no UI surface

odpt_metadata_info.txt

[cka-y]

[question] from my understand these feeds are independent of the api imports (i.e. they need to be imported only once) - any reason they have to apart of the monthly import here?

[ianktc]

hmm yes its true they are imported only once, this approach is only necessary if the endpoint exception

https://api-public.odpt.org/api/v4/files/{org_label}/data/{dataset_label}.zip comes up again with other organizations so we can track them moving forward. It also depends on whether we want to manually import an odpt series feed in the catalogs repo then?

In that case we are deciding to wait until more than 1 organization besides Toei occurs before we separately add this endpoint in this monthly import?

easier (now): just manually import the Toei feeds in the catalogs repo as odpt series feeds

plan for future: include in the monthly import as done in this PR in case we have future organizations that fall under this new namespace