[0.4.0] validate OSW input and output, preserve zero-length edges and co-located nodes, remove warnings - #57

CHANGELOG.md

@@ -1,5 +1,19 @@
 # Change log
 
+### 0.4.0
+- Validate OSW input datasets with `python-osw-validation` before OSW→OSM conversion starts, returning the validator issues on `Response.error` with `status=False` instead of converting an invalid dataset. Each reported issue names the file and feature index it came from, and repeated messages are collapsed.
+- Validate the OSW dataset generated by OSM→OSW conversion before reporting success, returning the validator issues on `Response.error` with `status=False`. The check runs with the same `coordinate_precision` and `allow_zero_length_lines` used to produce the output, and can be skipped with `validate_output=False`.
+- Add formatter configuration for `validate_input` (default `True`) to skip the check for datasets known to be non-compliant.
+- Reject OSM input files whose node coordinates carry more decimal places than `coordinate_precision`, before any conversion work starts, for both XML and PBF inputs. This replaces the previous coordinate precision warning: conversion never adds precision, so gating the input keeps generated output within the limit.
+- Return a readable error when an OSM input file cannot be parsed, instead of printing a traceback and surfacing the raw parser message. The error says the file is corrupted and gives the line and column, with parser jargon kept off the message; it reads the same whether the failure surfaces during input validation or during conversion.
+- Report unreadable OSW archives in plain language -- missing file, not a zip, or no `.geojson` files inside -- instead of passing through OS error numbers and internal directory wording, and echo only the file name rather than its full path.
+- Restore OSW edge endpoints from `_u_id`/`_v_id` when converting to OSM, so an edge between two different nodes that share a location keeps running between them instead of collapsing onto whichever node was created first.
+- Preserve zero-length edges across both conversion directions when `allow_zero_length_lines` is set: an OSW edge whose endpoints are the same node becomes an OSM way with that node referenced twice, and such a way converts back to one node and one edge starting and ending at it. With the setting off, the edge collapses to a single point instead.
+- Keep two OSW point features at the same location as two OSM nodes when `allow_zero_length_lines` is set, even with identical tags. Way vertices still merge onto the node features at their endpoints, so edges stay connected and kerb nodes stay attached.
+- Apply `coordinate_precision` and `allow_zero_length_lines` to OSW input validation, matching the rules conversion runs with.
+- **Breaking:** change the `allow_zero_length_lines` default to `True`, matching the validator default. Set it to `False` to drop zero-length lines or collapse them to points.
+- **Breaking:** remove the warning subsystem. `Response.warnings` and `helpers/warnings.py` are gone, and conversion no longer emits non-blocking warnings.
+
 ### 0.3.7
 - [BUG-4066](https://dev.azure.com/TDEI-UW/TDEI/_workitems/edit/4066/) - Clean zero-length and collapsed coordinate geometries in both OSM→OSW and OSW→OSM conversion paths.
 - Preserve collapsed OSM edge/way data as valid custom OSW point features with original way tags stored under `ext:*`, avoiding invalid `nodes.geojson` properties.

README.md

@@ -93,8 +93,10 @@ The formatter supports optional conversion configuration through `FormatterConfi
 
 | Option | Default | Description |
 |--------|---------|-------------|
-| `coordinate_precision` | `7` | Maximum coordinate decimal places before a non-blocking warning is returned. Output coordinates are not modified by this setting. |
-| `allow_zero_length_lines` | `False` | Allows zero-length LineString geometries for line-based datasets when set to `True`. This does not apply to polygons or zones, which must still have valid non-zero-area geometry. |
+| `coordinate_precision` | `7` | Decimal places allowed in input coordinates. OSM input carrying more precise coordinates is rejected, and the same limit is applied when an OSW input dataset is validated. |
+| `allow_zero_length_lines` | `True` | Keeps zero-length LineString geometries for line-based datasets. Set to `False` to drop them, or collapse them to points where possible. This does not apply to polygons or zones, which must still have valid non-zero-area geometry. The same setting is applied when an OSW input dataset is validated. |
+| `validate_input` | `True` | Validates the input before conversion starts: an OSW dataset with `python-osw-validation`, an OSM file against `coordinate_precision`. Set to `False` to convert inputs that are known to be non-compliant. |
+| `validate_output` | `True` | Validates the OSW dataset generated by OSM → OSW conversion with `python-osw-validation`. Set to `False` to keep output that is known to be non-compliant. |
 
 Conversion returns a `Response` object:
 
@@ -103,12 +105,63 @@ Conversion returns a `Response` object:
 | `status` | `True` when conversion succeeds, `False` when conversion fails. |
 | `generated_files` | Output file path or list of output file paths. |
 | `error` | Error message when `status` is `False`. |
-| `warnings` | Non-blocking warning text. Warnings do not change generated data or conversion status. |
 
-Warnings may be returned when input coordinates exceed the configured precision or when duplicate/collapsed coordinate geometry is found. For duplicate or collapsed geometry, the formatter may remove repeated coordinate vertices, omit geometries that cannot form a valid line or polygon, preserve zero-length LineStrings only when `allow_zero_length_lines=True`, or convert collapsed features to point output when possible.
+Duplicate or collapsed coordinate geometry is cleaned during conversion: repeated coordinate vertices are removed, geometries that cannot form a valid line or polygon are omitted, zero-length LineStrings are preserved unless `allow_zero_length_lines=False`, and collapsed features are converted to point output when possible.
 
 Conversion returns `status=False` when no output files are generated, or when OSW → OSM generates an OSM XML file with no `node`, `way`, or `relation` elements.
 
+### OSM input validation
+
+OSM → OSW conversion checks every node coordinate in the input before any conversion work is done. A file carrying coordinates more precise than `coordinate_precision` is rejected outright rather than silently reduced. Conversion never invents precision — coordinates pass through unchanged — so a file that clears this check produces output within the limit:
+
+```python
+result = await Formatter(workdir=<OUTPUT_DIR>, file_path=<OSM_INPUT_FILE>).osm2osw()
+if not result.status:
+    print(result.error)
+    # invalid input file, the file has GPS locations with higher than 7-digits
+    # precision that TDEI doesn't allow. Please clean your dataset and resubmit
+```
+
+A file that cannot be parsed at all is reported the same way, naming the line and column rather than the parser's own message:
+
+```
+invalid input file, the OSM file is corrupted and could not be read.
+The problem is at line 3, column 65. Please fix the file and resubmit
+```
+
+Both `.osm`/`.xml` and `.pbf` inputs are checked. XML coordinates are read as exact decimal strings; PBF stores coordinates as integers in units of 1e-7 degrees, so it can only exceed a limit below 7. Pass `validate_input=False` to skip the check.
+
+### OSW output validation
+
+OSM → OSW conversion validates the dataset it generates before reporting success. If the validator rejects it, `status` is `False` and the issues come back on `error`, each naming the generated file and feature:
+
+```python
+result = await Formatter(workdir=<OUTPUT_DIR>, file_path=<OSM_INPUT_FILE>).osm2osw()
+if not result.status:
+    print(result.error)
+    # Generated OSW dataset is not valid.
+    # - out.graph.edges.geojson (feature 0): Invalid value at 'width': 'NaN' . Acceptable datatype is number ; provide a valid value and retry
+```
+
+The validator runs with the formatter's own `coordinate_precision` and `allow_zero_length_lines`, so output is judged by the rules it was produced with. Pass `validate_output=False` to skip the check.
+
+### OSW input validation
+
+OSW → OSM conversion validates the input archive before any conversion work is done. If the OSW validator rejects the dataset, no output is generated and the validator issues are returned to the caller on the `Response`. Each issue names the file and feature it came from:
+
+```python
+result = Formatter(workdir=<OUTPUT_DIR>, file_path=<OSW_INPUT_FILE>).osw2osm()
+if not result.status:
+    print(result.error)
+    # Input is not a valid OSW dataset.
+    # - nodes.geojson (feature 1): "_id" is a required property (at: features[1].properties)
+    # - edges.geojson (feature 0): Invalid value at 'width': 'NaN' . Acceptable datatype is number ; provide a valid value and retry
+```
+
+The validator runs with the formatter's own `coordinate_precision` and `allow_zero_length_lines` settings, so input is judged by the same rules the formatter converts with. Up to 20 issues are reported, and repeated messages are collapsed. Pass `validate_input=False` to skip the check.
+
+Sample datasets for both outcomes live in [`fixtures/`](fixtures/README.md): `valid_osw.zip` passes validation and converts, `invalid_osw.zip` fails with one deliberate defect in each of the six OSW files.
+
 
 ## Starting a new project with template  
 
@@ -123,7 +176,9 @@ from osm_osw_reformatter import Formatter, FormatterConfig
 async def osm_convert():
     config = FormatterConfig(
         coordinate_precision=7,
-        allow_zero_length_lines=False,
+        allow_zero_length_lines=True,
+        validate_input=True,
+        validate_output=True,
     )
     f = Formatter(workdir=<OUTPUT_DIR>, file_path=<OSM_INPUT_FILE>, config=config)
     return await f.osm2osw()
@@ -144,8 +199,6 @@ if __name__ == '__main__':
         print(results.generated_files)
     else:
         print(results.error)
-    if results.warnings:
-        print(results.warnings)
     osw_convert()  
 ```  
 

requirements.txt

@@ -5,4 +5,4 @@ shapely~=2.0.2
 pyproj~=3.6.1
 coverage~=7.5.1
 ogr2osm==1.2.0
-python-osw-validation==0.4.0
+python-osw-validation==0.5.0

setup.py

@@ -32,7 +32,8 @@
         'networkx~=3.2',
         'shapely~=2.0.2',
         'pyproj~=3.6.1',
-        'ogr2osm==1.2.0'
+        'ogr2osm==1.2.0',
+        'python-osw-validation==0.5.0'
     ],
     packages=find_packages(where='src'),
     classifiers=[

src/example.py

@@ -6,8 +6,8 @@
 ROOT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 
 OUTPUT_DIR = f'{ROOT_DIR}/output'
-OSM_INPUT_FILE = f'{ROOT_DIR}/input/wedgewood_output.osm.pbf'
-OSW_INPUT_FILE = f'{ROOT_DIR}/input/wa.seattle.zip'
+OSM_INPUT_FILE = f'{ROOT_DIR}/fixtures/zero_length_way_only.xml'
+OSW_INPUT_FILE = f'{ROOT_DIR}/fixtures/invalid_osw.zip'
 
 is_exists = os.path.exists(OUTPUT_DIR)
 if not is_exists:
@@ -28,14 +28,15 @@ async def osm_convert():
 def osw_convert():
     f = Formatter(workdir=OUTPUT_DIR, file_path=OSW_INPUT_FILE)
     results = f.osw2osm()
-    print(results.warnings)
+    if not results.status:
+        print(results.error)
     # Uncomment below line to clean up the generated files
     # f.cleanup()
 
 
-# if __name__ == '__main__':
-#     asyncio.run(osm_convert())
-#     osw_convert()
+if __name__ == '__main__':
+    asyncio.run(osm_convert())
+    # osw_convert()
 
 def main():
     parser = argparse.ArgumentParser(description='Convert between OSM and OSW')

src/osm_osw_reformatter/__init__.py

@@ -5,6 +5,8 @@
 from .config import (
     DEFAULT_ALLOW_ZERO_LENGTH_LINES,
     DEFAULT_COORDINATE_PRECISION,
+    DEFAULT_VALIDATE_INPUT,
+    DEFAULT_VALIDATE_OUTPUT,
     FormatterConfig,
 )
 from .helpers.response import Response
@@ -24,6 +26,8 @@ def __init__(
         config: FormatterConfig = None,
         coordinate_precision: int = None,
         allow_zero_length_lines: bool = None,
+        validate_input: bool = None,
+        validate_output: bool = None,
     ):
         is_exists = os.path.exists(workdir)
         if not is_exists:
@@ -42,6 +46,16 @@ def __init__(
                     if allow_zero_length_lines is None
                     else allow_zero_length_lines
                 ),
+                validate_input=(
+                    DEFAULT_VALIDATE_INPUT
+                    if validate_input is None
+                    else validate_input
+                ),
+                validate_output=(
+                    DEFAULT_VALIDATE_OUTPUT
+                    if validate_output is None
+                    else validate_output
+                ),
             )
         self.workdir = workdir
         self.file_path = file_path

src/osm_osw_reformatter/config.py

@@ -2,7 +2,9 @@
 
 
 DEFAULT_COORDINATE_PRECISION = 7
-DEFAULT_ALLOW_ZERO_LENGTH_LINES = False
+DEFAULT_ALLOW_ZERO_LENGTH_LINES = True
+DEFAULT_VALIDATE_INPUT = True
+DEFAULT_VALIDATE_OUTPUT = True
 
 
 @dataclass(frozen=True)
@@ -11,6 +13,8 @@ class FormatterConfig:
 
     coordinate_precision: int = DEFAULT_COORDINATE_PRECISION
     allow_zero_length_lines: bool = DEFAULT_ALLOW_ZERO_LENGTH_LINES
+    validate_input: bool = DEFAULT_VALIDATE_INPUT
+    validate_output: bool = DEFAULT_VALIDATE_OUTPUT
 
     def __post_init__(self) -> None:
         if isinstance(self.coordinate_precision, bool) or not isinstance(
@@ -21,3 +25,7 @@ def __post_init__(self) -> None:
             raise ValueError("coordinate_precision must be zero or greater.")
         if not isinstance(self.allow_zero_length_lines, bool):
             raise TypeError("allow_zero_length_lines must be a boolean.")
+        if not isinstance(self.validate_input, bool):
+            raise TypeError("validate_input must be a boolean.")
+        if not isinstance(self.validate_output, bool):
+            raise TypeError("validate_output must be a boolean.")