fix(bootstrap): two-step encoding for procedures and deployments + NWS canonical refactor#6
Merged
Merged
Conversation
ensure_procedure and ensure_deployment now mirror ensure_system: POST a
geo+json stub (uid/name/description/geometry/featureType/validTime),
then optionally PUT a SensorML body with Content-Type:
application/sml+json against /resource/{id}.
Also adds _warn_if_sml_fields_in_stub: a closed-set guardrail that warns
(or raises, when OS4CSAPI_STRICT_BOOTSTRAP=1) if a stub body still
carries SensorML-only fields under properties.
force_sml=True now applies to procedures and deployments as well as
systems, allowing in-place recovery for records that were created with
the old single-POST shape.
Background: pre-strict CSAPI servers returned HTTP 201 and silently
dropped SensorML metadata on procedures/deployments. Strict upstream
(connected-systems-go after a467aba) returns HTTP 400. Either way, the
bug was on the client.
Refs: #5
Replaces PROCEDURE_BODY (single mixed-encoding dict) with _procedure_stub and _procedure_sml; strips documentation arrays from _deploy_root and _deploy_group stubs and adds matching _deploy_root_sml / _deploy_group_sml; threads force_sml through procedure and deployment create calls so --force-sml now repairs them in place. NWS is the canonical example for the same refactor that needs to land across the other 9 publishers (E.2 follow-up). Refs: #5
Four offline tests cover SML_ONLY_FIELDS membership and the _warn_if_sml_fields_in_stub guardrail in both lenient and strict modes. Two network-gated tests POST a procedure / deployment with marker keywords + identifiers, GET them back as application/sml+json, and assert the marker fields survive. Enable by setting OS4CSAPI_TEST_BASE_URL / OS4CSAPI_TEST_USER / OS4CSAPI_TEST_PASS. Verified: 6/6 passing against https://129-80-248-53.sslip.io/csapi-go-upstream/. Refs: #5
11-section report covering symptom, discovery, root cause, evidence, the fix, verification, recovery operations, lessons, cross-references, and timeline. Refs: #5
Sam-Bolling
added a commit
that referenced
this pull request
May 10, 2026
…atalog
Honest assessment of strict-parsing migration work to date:
- What is genuinely improving (portability, OGC 23-001 conformance,
empirical schema documented, helper infrastructure validated).
- What is NOT improving (metadata loss on the wire, no automated tests,
copy-paste mechanical pattern, inline workaround comments).
- Comprehensive catalog of 4 P1/P2 upstream csapi-go-v2 defects + 2
informational items, each with reproducer, current workaround, and
expiry condition:
#1 P1 procedure SML asymmetric typo (c2ab201 incomplete)
#2 P1 system SML rejects characteristics/capabilities
#3 P2 system SML returns 500 on typeOf
#4 P2 datastream rejects SWE Time referenceTime
#5 P3 datastream rejects top-level uid
#6 informational deployment parent@link encoding
Recommendations: extract sml_builders helper, add contract tests, file
upstream issues, track metadata loss explicitly, choose OSH posture.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5.
What this PR delivers (E.1 vertical slice)
publishers/bootstrap_helpers.pyensure_procedureandensure_deploymentnow mirrorensure_system: POST geo+json stub → optional PUTapplication/sml+jsonbody.force_smlextended to both. New_warn_if_sml_fields_in_stubguardrail (lenient warn / strictRuntimeErrorviaOS4CSAPI_STRICT_BOOTSTRAP=1).publishers/nws/bootstrap_nws.pyPROCEDURE_BODYsplit into_procedure_stub+_procedure_sml;_deploy_root/_deploy_groupdocumentation arrays moved into matching_deploy_root_sml/_deploy_group_smlcompanions;force_smlplumbed into all three create calls.tests/test_bootstrap_roundtrip.pydocs/research/Silent_SensorML_Field_Loss_Engineering_Report_2026-05-06.mdVerification
pytest tests/test_bootstrap_roundtrip.pyagainsthttps://129-80-248-53.sslip.io/csapi-go-upstream/→ 6 passed. Markerkeywordsandidentifiersround-trip end-to-end on both procedures and deployments.OS4CSAPI_STRICT_BOOTSTRAP=1strict-mode smoke: NWS stubs are clean, SML bodies have expected SensorML keys.Out of scope — tracked as E.2 follow-up
The other 9 publishers (
aviation_wx,coops,iss,ndbc,opensky,usgs_eq,usgs_nims,usgs_water) still use the single-POST pattern for procedures and deployments. They are unblocked by this PR — helpers are backward-compatible: omitsml_bodyand behavior is unchanged — and will be refactored in a follow-up batch using NWS as the reference template.Recovery for already-deployed environments
After this lands:
will PUT the (now correct) SensorML bodies against existing procedure / deployment records without rebuilding identity, links, or datastreams.
References
ensure_procedureandensure_deploymentsilently lose all SensorML metadata (keywords, identifiers, classifiers, …) — POSTs use wrong content-type and payload shape #5docs/research/Silent_SensorML_Field_Loss_Engineering_Report_2026-05-06.mdOS4CSAPI/connected-systems-go@a467aba