diff --git a/CLAUDE.md b/CLAUDE.md index 205272c..33e63c4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,17 +29,17 @@ mkdocs serve `xml2db` maps an XSD schema to a relational database schema and loads XML files into it. The top-level flow is: -1. **`DataModel`** (`model.py`) reads an XSD file using `xmlschema` + `lxml`, traverses the schema tree, and builds a set of `DataModelTable` objects — one per XSD `complexType`. It then creates SQLAlchemy tables from those objects. +1. **`DataModel`** (`model.py`) reads an XSD file using `xmlschema` + `lxml`, traverses the schema tree, and builds a set of `DataModelTable` objects (one per XSD `complexType`). It then creates SQLAlchemy tables from those objects. 2. **`DataModel.parse_xml()`** returns a **`Document`** (`document.py`), which holds the parsed flat data ready for insertion. -3. **`XMLConverter`** (`xml_converter.py`) does the actual XML traversal, producing a nested "document tree" dict. Two strategies exist: iterative (`iterparse=True`) and recursive — tests assert they produce identical output. +3. **`XMLConverter`** (`xml_converter.py`) does the actual XML traversal, producing a nested "document tree" dict. Two strategies exist: iterative (`iterparse=True`) and recursive; tests assert they produce identical output. 4. **`Document.insert_into_target_tables()`** inserts the flat data into the database. **`Document.to_xml()`** converts it back. ### Table hierarchy (`table/`) Each XSD `complexType` becomes one of two concrete table classes: -- **`DataModelTableReused`** — deduplicates identical subtrees via a SHA-256 hash column (`xml2db_record_hash`). This is the default. Relationships between a reused child and multiple parents require an intermediate join table (`DataModelRelationN` + `DataModelTransformedTable`). -- **`DataModelTableDuplicated`** — stores rows without deduplication; parent FK lives directly in the child row. Set `"reuse": False` in `model_config` to use this per table. +- **`DataModelTableReused`**: deduplicates identical subtrees via a SHA-256 hash column (`xml2db_record_hash`). This is the default. Relationships between a reused child and multiple parents require an intermediate join table (`DataModelRelationN` + `DataModelTransformedTable`). +- **`DataModelTableDuplicated`**: stores rows without deduplication; parent FK lives directly in the child row. Set `"reuse": False` in `model_config` to use this per table. Relations are stored as `DataModelRelation1` (0..1 / 1..1) or `DataModelRelationN` (0..n / 1..n) in `DataModelTable.fields`. @@ -57,6 +57,12 @@ cd tests/sample_models && python models.py then commit the updated snapshot files alongside the code change. +## Writing style + +- After any code change, check whether docstrings, inline docs, or `docs/` pages need updating and update them as part of the same task. +- After updating `docs/`, verify it builds without errors: `python -m mkdocs build --strict`. In strict mode warnings are also errors, matching CI. +- Write docstrings and documentation concisely and directly. No em-dashes or en-dashes; use commas, colons, or plain sentences instead. Avoid AI-sounding phrasing such as bold lead-ins ("**Why this matters:**"), filler constructions ("it is worth noting", "this allows", "this ensures"), and verbose qualifiers. + ### Key configuration options (`model_config`) | Option | Effect | diff --git a/docs/configuring.md b/docs/configuring.md index 3351287..8fd5ba1 100644 --- a/docs/configuring.md +++ b/docs/configuring.md @@ -5,25 +5,47 @@ description: "Configure xml2db's data model via model_config: override column ty # Configuring your data model -The data model is derived automatically from an XSD file you provide. It is a set of tables linked by foreign key -relationships. Each `complexType` in the XSD corresponds to a table, named after the first element of that type (with -deduplication if needed). Columns correspond to `simpleType` elements and attributes within the complex type, named -after the XML element or attribute. +The data model is derived automatically from an XSD file. Each `complexType` becomes a table, columns come from `simpleType` elements and attributes, and `xml2db` applies a few simplifications by default to reduce complexity. -`xml2db` applies a few simplifications to the original data model by default, but they can also be opted-out or forced -through the configuration `dict` provided to the `DataModel` constructor. - -The column types can also be configured to override the default type mapping, using `sqlalchemy` types. +Options apply at three levels: model, table, and field. !!! tip - We recommend that you first build the data model without any configuration, visualize it as a text tree or ER - diagram (see the [Getting started](getting_started.md) page for directions on how to visualize data models) and - then adapt the configuration if need be. + Start without any configuration, visualize the data model, then add options as needed. The `xml2db serve` command opens an interactive browser explorer with a live-updating ERD, tree views, DDL, and a YAML config editor. See [Getting started](getting_started.md) for details. + +## Config file format + +The config can be written as a YAML file and passed with `--config` (CLI) or `load_config()` (Python API): + +``` yaml title="model_config.yml" +row_numbers: false +record_hash_column_name: record_hash +metadata_columns: + - name: input_file_path + type: String(256) +tables: + my_table: + reuse: false + choice_transform: false + fields: + my_column: + type: String(100) + rename: col_name + transform: skip +``` + +SQLAlchemy type names in YAML are strings like `String(256)`, `Integer`, `DateTime(timezone=True)`. The full list of supported names: `String`, `Text`, `Integer`, `BigInteger`, `SmallInteger`, `Float`, `Double`, `Numeric`, `Boolean`, `DateTime`, `Date`, `Time`, `LargeBinary`, `JSON`, `Uuid`. + +Keys that require Python callables (`document_tree_hook`, `document_tree_node_hook`, `record_hash_constructor`) cannot be set in a YAML file. Pass a Python dict directly in that case. + +## Python dict config + +For programmatic use, or when you need callable hooks or SQLAlchemy type instances, pass a dict to `DataModel`: -Options apply at three levels: model, table, and field. The general structure of the configuration dict is: +```py title="Model config general structure" linenums="1" +from xml2db import DataModel +import sqlalchemy -```py title="Model config general structure" linenums="1" -{ +model_config = { "document_tree_hook": None, "document_tree_node_hook": None, "row_numbers": False, @@ -36,13 +58,22 @@ Options apply at three levels: model, table, and field. The general structure of "as_columnstore": False, "fields": { "my_column": { - "type": None #default type - } + "type": None, # default type + } }, "extra_args": [], } - } + }, } + +data_model = DataModel(xsd_file="schema.xsd", model_config=model_config) +``` + +To load a YAML file into a Python dict, use `load_config`: + +``` py +from xml2db import load_config +model_config = load_config("model_config.yml") ``` ## Model configuration @@ -71,6 +102,7 @@ as dicts, the only required keys are `name` and `type` (a SQLAlchemy type object as keyword arguments to `sqlalchemy.Column`. Actual values need to be passed to [`DataModel.parse_xml`](api/data_model.md#xml2db.model.DataModel.parse_xml) for each parsed documents, as a `dict`, using the `metadata` argument. +* `transform` (`false` or `"auto"`): set to `false` to disable all automatic field transformations globally: no joining of multi-value columns, no elevation of child tables, no collapsing of choice groups. The default `"auto"` applies all of these where applicable. Per-field `transform` and per-table `choice_transform` still override the global setting. * `record_hash_column_name`: the column name to use to store records hash data (defaults to `xml2db_record_hash`). * `record_hash_constructor`: a function used to build a hash, with a signature similar to `hashlib` constructor functions (defaults to `hashlib.sha1`). @@ -81,8 +113,26 @@ functions (defaults to `hashlib.sha1`). These configuration options are defined for a specific field of a specific table. A "field" refers to a column in the table, or a child table. +### Source names vs target names + +Field names in `model_config` come from two different points in the processing pipeline. Which one to use depends on the config key: + +| Config key | Name to use | Where to look | +|---|---|---| +| `transform` | **Source name**: the original XSD element or relation name, before any simplification | **Source tree** tab | +| `type`, `rename` | **Target name**: the logical column name after elevation and prefixing | **Target tree** tab | + +Elevation (the default for small mandatory children) collapses a child relation into prefixed columns in the parent. For example, if `timeInterval` is elevated, the target model has `timeInterval_start` and `timeInterval_end`, and `timeInterval` itself no longer appears in the target tree. To opt out of elevation you configure `fields.timeInterval.transform: false` using the **source name**. To rename an elevated result you configure `fields.timeInterval_start.rename: "start"` using the **target name**. + +If a field is not elevated (a direct column or a kept relation), its source and target names are identical and there is no ambiguity. + +The browser explorer autocomplete (`xml2db serve`) offers both source and target field names and labels each accordingly. + ### Data types +!!! note "Uses target name" + Use the field name as it appears in the **Target tree** tab. + By default, the data type defined in the database table for each column is based on a mapping between the data type indicated in the XSD and a corresponding `sqlalchemy` type implemented in the following three methods: @@ -127,6 +177,9 @@ defined as `sqlalchemy` types and will be passed to the `sqlalchemy.Column` cons ### Renaming columns +!!! note "Uses target name" + Use the field name as it appears in the **Target tree** tab. For elevated fields, this is the prefixed name (e.g. `orderperson_name`), not the original child relation name. + The physical database column name for any field can be overridden while keeping the original XML element name as the internal logical key. This is useful when XSD element names are awkward, conflict with reserved SQL words, or need to follow a naming convention that differs from the source schema. @@ -165,12 +218,14 @@ Configuration: `"rename":` `"new_column_name"` (no default; omit to keep the ori ### Joining values for simple types +!!! note "Uses source name" + Use the field name as it appears in the **Source tree** tab. + By default, XML simple type elements with types in `["string", "date", "dateTime", "NMTOKEN", "time", "base64Binary", "decimal"]` and max occurrences >= 1 are joined in one column as comma separated values and optionally wrapped in double quotes if they contain commas (an Excel-like csv format, which can be queried with `LIKE` statements in SQL). -Configuration: `"transform":` `"join"` (default). It is not currently possible to use `False` to opt-out of an -automatically applied `join`, as it would require a complex process of adding a new table. +Configuration: `"transform":` `"join"` (default). Opting out of an automatically applied `join` is not supported, as it would require adding a new table. !!! example This config option is currently not very useful as it cannot be opted out. @@ -190,6 +245,9 @@ automatically applied `join`, as it would require a complex process of adding a ### Skipping fields +!!! note "Uses source name" + Use the field name as it appears in the **Source tree** tab. + Any field (column or relation) can be excluded from the data model entirely by setting its transform to `"skip"`. The field will be absent from the target table schema and all data for it will be silently dropped during XML parsing. This is useful for PII columns, large binary blobs, or fields that are irrelevant for analysis. @@ -220,6 +278,9 @@ Configuration: `"transform": "skip"` ### Elevate children to upper level +!!! note "Uses source name" + Use the child relation name as it appears in the **Source tree** tab (the parent's field pointing to the child, before any elevation). This name may not appear in the Target tree at all if default elevation has already collapsed it. + A mandatory child (min occurrences = 1, i.e. `[1, 1]`) is always elevated to its parent by default, as long as it is not involved in a 1-n relationship elsewhere in the schema. @@ -228,7 +289,7 @@ are not counted), and again only when it is not involved in a 1-n relationship e This behaviour can be disabled, or forced for larger children (more than 4 simple-type columns), using: -`"transform":` `"elevate"` (default) or `"elevate_wo_prefix"` or `False` (disable). +`"transform":` `"elevate"` (default) or `"elevate_wo_prefix"` or `False` (disable) or `"auto"` (explicit default). By default, the elevated field is prefixed with the child's name to clarify its origin and avoid name collisions. Use `"elevate_wo_prefix"` to skip the prefix. @@ -298,7 +359,7 @@ idOfMarketParticipant[1, 1] (choice): This simplification is applied automatically when there are more than 2 options of the same data type. It can be forced on or disabled explicitly with the following option: -`"choice_transform":` `True` (force on) or `False` (disable) +`"choice_transform":` `True` (force on) or `False` (disable) or `"auto"` (explicit default) !!! example Disable choice group simplification for a choice group: diff --git a/docs/getting_started.md b/docs/getting_started.md index b6e6b83..67d6daf 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -1,138 +1,146 @@ --- title: "Getting started" -description: "Step-by-step guide to installing xml2db, creating a DataModel from an XSD file, importing XML into a relational database, and visualizing the resulting data model." +description: "Step-by-step guide to installing xml2db, exploring an XSD schema with the CLI, importing XML into a relational database, and visualizing the resulting data model." --- # Getting started -This guide walks you through installing xml2db, creating a data model from an XSD schema, loading XML files into a relational database, and exporting data back to XML. +This guide walks you through installing xml2db, exploring and configuring a data model from an XSD schema, and loading XML files into a relational database. ## Installation -The package can be installed, preferably in a virtual environment, using `pip`: +Install the package, preferably in a virtual environment: ``` bash pip install xml2db ``` +You will also need a database driver for your backend (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for MySQL, `pyodbc` for SQL Server, `duckdb-engine` for DuckDB). See [How it works](how_it_works.md#bulk-loading) for which drivers enable native bulk loading. + !!! note - If you want to contribute to the development of `xml2db`, clone the git repository and then install it in your - virtual environment in editable mode. From the project's directory, run: - + To contribute to `xml2db` development, clone the repository and install in editable mode: + ``` bash pip install -e .[docs,tests] ``` -## Reading an XML schema +## Exploring the data model -Start from an XSD schema file that you will read with `xml2db` to create a [`DataModel`](api/data_model.md) object: +The quickest way to understand your schema and configure the data model is the interactive browser explorer: -``` py title="Create a DataModel object" linenums="1" -from xml2db import DataModel +``` bash +xml2db serve path/to/schema.xsd +``` -data_model = DataModel( - xsd_file="path/to/file.xsd", - db_schema="source_data", # the name of the database target schema - connection_string="postgresql+psycopg2://testuser:testuser@localhost:5432/testdb", - model_config={}, -) +This opens a browser with four tabs: + +- **ERD**: an entity-relationship diagram of the tables derived from the XSD +- **Target tree**: a text tree of the simplified data model that will be loaded into the database +- **Source tree**: a text tree of the raw XSD structure before simplification +- **DDL**: `CREATE TABLE` statements for the target schema + +The left panel is a YAML config editor with autocomplete for table names, field names, and all config options. Edit the config and the diagram updates automatically (with a short debounce). When the config looks right, click **Save** to write it to a file (default: `model_config.yml`). + +You can also render these representations directly to stdout or a file without the browser: + +``` bash +xml2db render schema.xsd --format erd +xml2db render schema.xsd --format target-tree +xml2db render schema.xsd --format source-tree +xml2db render schema.xsd --format ddl --db-type postgresql +``` + +See [Configuring your data model](configuring.md) for a full description of the available config options. + +## Importing XML files + +Once the data model looks right, import an XML file into the database: + +``` bash +xml2db import file.xml schema.xsd \ + --connection-string "postgresql+psycopg2://user:pw@host/db" \ + --config model_config.yml ``` -A connection string and database are not required at this stage, but both are needed to actually import data. You -may need to install a connector package (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for -MySQL, `pyodbc` for SQL Server, `duckdb-engine` for DuckDB). See -[How it works](how_it_works.md#bulk-loading) for which drivers enable native bulk loading. +On success, the command prints the number of rows inserted and already-existing (deduplicated), with per-phase timings. + +Key options: -The optional `model_config` controls schema simplifications, column types, and more. By default, some simplifications -are applied to reduce data model complexity. +- `--config FILE`: YAML model config file +- `--db-schema SCHEMA`: target database schema +- `--metadata KEY=VALUE`: values for `metadata_columns` (e.g. `--metadata source=file.xml`) +- `--validate`: validate the XML against the schema before importing +- `--recover`: attempt to parse malformed XML -## Visualizing the data model +## Using the Python API -When starting from a new XML schema, we recommend visualizing the data model first to decide whether any tweaking is -needed. Generate a Markdown file with a visual representation of your schema (`data_model` being the -[`DataModel`](api/data_model.md) object previously created): +The same operations are available programmatically. Create a [`DataModel`](api/data_model.md) from an XSD file: -``` py title="Write an Entity Relationship Diagram to a file" linenums="1" -with open(f"target_data_model_erd.md", "w") as f: - f.write(data_model.get_entity_rel_diagram()) +``` py title="Create a DataModel" linenums="1" +from xml2db import DataModel, load_config + +data_model = DataModel( + xsd_file="path/to/file.xsd", + db_schema="source_data", + connection_string="postgresql+psycopg2://user:pw@host/db", + model_config=load_config("model_config.yml"), # or a plain dict +) ``` -You can see an example of these diagrams on the [Introduction page](index.md). +A connection string is not required until you actually import data. + +### Visualizing the data model -The diagram uses [Mermaid](https://mermaid.js.org/syntax/entityRelationshipDiagram.html) to show the tables and their -relationships. [PyCharm](https://www.jetbrains.com/help/pycharm/markdown.html#diagrams) and GitHub both render Mermaid -diagrams natively. +``` py title="Write an ERD to a file" linenums="1" +with open("data_model_erd.md", "w") as f: + f.write(data_model.get_entity_rel_diagram()) +``` -You can also visualize your model in a tree-like text mode. In this format, you can visualize the raw, untouched XML -schema, as well as the simplified one (we call it "target" model): +The diagram uses [Mermaid](https://mermaid.js.org/syntax/entityRelationshipDiagram.html). PyCharm and GitHub both render Mermaid natively. -``` py title="Write source tree and target tree to a file" linenums="1" -with open(f"source_tree.txt", "w") as f: +``` py title="Write source and target trees to files" linenums="1" +with open("source_tree.txt", "w") as f: f.write(data_model.source_tree) -with open(f"target_tree.txt", "w") as f: +with open("target_tree.txt", "w") as f: f.write(data_model.target_tree) ``` -It will write something like this: +The tree format shows element names, data types, and cardinality (min/max occurrences): ``` -... -docStatus_value[0, 1]: NMTOKEN TimeSeries[0, None]: mRID[1, 1]: string businessType[1, 1]: NMTOKEN - quantity_Measure_Unit.name[1, 1]: NMTOKEN - curveType[1, 1]: NMTOKEN Available_Period[0, None]: timeInterval_start[1, 1]: string timeInterval_end[1, 1]: string resolution[1, 1]: duration - Point[1, None]: - position[1, 1]: integer - quantity[1, 1]: decimal - WindPowerFeedin_Period[0, None]: - timeInterval_start[1, 1]: string - timeInterval_end[1, 1]: string -... ``` -This gives you the elements names, data type and cardinality (min/max number of children elements). - -It is useful to visualize your data model in order to [configure it](configuring.md) to suit your needs. - -## Importing XML files - -Once the data model looks right, load XML files as follows: +### Importing XML files -``` py title="Parse an XML file" linenums="1" -document = data_model.parse_xml( - xml_file="path/to/file.xml", -) +``` py title="Parse and import an XML file" linenums="1" +document = data_model.parse_xml(xml_file="path/to/file.xml") document.insert_into_target_tables() ``` By default, XML files are not validated against the schema. Enable validation if you need to verify file integrity. -[`Document.insert_into_target_tables`](api/document.md#xml2db.document.Document.insert_into_target_tables) is all you -need to load data into the database. - -See the [How it works](how_it_works.md) page for a deeper explanation of the loading process. +[`Document.insert_into_target_tables`](api/document.md#xml2db.document.Document.insert_into_target_tables) handles creating tables, staging data, merging, and cleanup automatically. !!! note - `xml2db` can save metadata for each loaded XML file. These can be configured using the - [`metadata_columns` option](configuring.md#model-configuration) and create additional columns in the root table. - It can be used for instance to save file name or loading timestamp. - - Actual values need to be passed to [`DataModel.parse_xml`](api/data_model.md#xml2db.model.DataModel.parse_xml) for - each parsed documents, as a `dict`, using the `metadata` argument. + To attach metadata to each loaded file (e.g. filename or timestamp), configure [`metadata_columns`](configuring.md#model-configuration) and pass values via the `metadata` argument: -!!! note "Loading multiple XML files in one database operation" - By default, each `parse_xml` + `insert_into_target_tables` call is an independent database operation. When you have - many small XML files to load, you can instead accumulate all of them in memory first and insert them in a single - batch, which reduces the number of database round-trips. + ``` py + document = data_model.parse_xml( + xml_file="path/to/file.xml", + metadata={"input_file_path": "path/to/file.xml"}, + ) + ``` - Pass the `flat_data` from the previous document into the next `parse_xml` call to accumulate records: +!!! note "Loading multiple files in one database operation" + Accumulate records from multiple files in memory before a single insert to reduce database round-trips: ``` py flat_data = None @@ -146,15 +154,9 @@ See the [How it works](how_it_works.md) page for a deeper explanation of the loa document.insert_into_target_tables() ``` - Note that each file can carry its own `metadata` values (e.g. the file name or a loading timestamp), which will be - stored per root record in the columns defined by - [`metadata_columns`](configuring.md#model-configuration). - - - -## Getting back the data into XML +## Getting data back to XML -Data can be extracted from the database back to XML, primarily for round-trip testing. +Data can be extracted from the database back to XML, primarily for round-trip testing: ``` py title="Extract data back to XML" linenums="1" document = data_model.extract_from_database( diff --git a/docs/how_it_works.md b/docs/how_it_works.md index cb07b10..56d5b86 100644 --- a/docs/how_it_works.md +++ b/docs/how_it_works.md @@ -108,7 +108,7 @@ stored as deduplicated elements. In the end, the schema can have both `1-n` and `n-1` relationships, so table dependency order differs from the original tree structure. When processing tables in dependency order, the root table is no longer necessarily first. -## Caveats +### Caveats `xml2db` handles a variety of data models, but does not cover all possible schemas allowed by the [XML schema documents specification](https://en.wikipedia.org/wiki/XML_Schema_(W3C)). @@ -117,14 +117,14 @@ Known unsupported cases are described below. Other edge cases may also fail and thorough testing for unusual schemas. For example, you can implement round-trip tests (XML → database → XML) and compare the output against the original. -### Recursive XSD +#### Recursive XSD Recursive XML schemas are not fully supported, because they result in cycles in tables dependencies, which would make the process much more complex. Whenever a field which would introduce a dependency cycle is detected in the XSD, it is discarded with a warning, which means that the corresponding data in XML files will not be imported. The rest of the data should be processed correctly. -### Mixed content elements +#### Mixed content elements XML elements with mixed content can contain both text and children elements (tags). `xml2db` offers partial support for this: the text value will be stored in a specific column named `value`, but it will not record the specific sequence of diff --git a/docs/index.md b/docs/index.md index 7fe8a6d..0c823ad 100644 --- a/docs/index.md +++ b/docs/index.md @@ -15,44 +15,23 @@ description: "xml2db is a Python package that automatically maps an XSD schema t XML files into a relational model that stays close to the source data while remaining easy to query as flat database tables. -## How to load XML files into a database - -Loading XML files into a relational database with `xml2db` can be as simple as: - -``` py title="Loading XML into a database" linenums="1" -from xml2db import DataModel - -# Create a DataModel object from an XSD file -data_model = DataModel( - xsd_file="path/to/file.xsd", - connection_string="postgresql+psycopg2://testuser:testuser@localhost:5432/testdb", -) - -# Parse an XML file based on this XSD schema -document = data_model.parse_xml(xml_file="path/to/file.xml") - -# Load data into the database, creating target tables if need be -document.insert_into_target_tables() -``` - -The resulting data model closely follows the XSD schema. By default, `xml2db` applies a few simplifications to reduce -complexity and storage footprint. The above code works out of the box for most schemas. - The raw data can then be transformed using [DBT](https://www.getdbt.com/), SQL views, or stored procedures to produce more user-friendly tables. +See the [Getting started](getting_started.md) guide for installation, CLI usage, and the Python API. + +## Supported backends + Built on `sqlalchemy`, `xml2db` supports multiple database backends. Integration tests cover PostgreSQL, MySQL, MS SQL Server, and DuckDB. You may need to install a connector package (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB). See [How it works](how_it_works.md#bulk-loading) for which drivers enable native bulk loading. -## How to visualize your data model +## Visualising data models -`xml2db` can also generate visual diagrams of your data model directly from an XSD file, using +`xml2db` generates visual diagrams of your data model directly from an XSD file, using [Mermaid](https://mermaid.js.org/syntax/entityRelationshipDiagram.html) to represent tables and their relationships. -This is useful to review the data model before deciding whether any [configuration](./configuring.md) is needed. - It looks like this: ```mermaid diff --git a/pyproject.toml b/pyproject.toml index 92f5c64..f73a444 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,12 +20,16 @@ dependencies = [ "sqlalchemy>1.4", "xmlschema>=3.3.2", "lxml>=5.1.0", + "pyyaml>=6.0", ] [project.optional-dependencies] docs = ["mkdocs-material>=9.5.34", "mkdocstrings-python>=1.11.1"] tests = ["pytest>=7.0"] +[project.scripts] +xml2db = "xml2db.cli:main" + [project.urls] "Documentation" = "https://cre-dev.github.io/xml2db" "Repository" = "https://github.com/cre-dev/xml2db" diff --git a/src/xml2db/__init__.py b/src/xml2db/__init__.py index 07bd4ad..aafaf59 100644 --- a/src/xml2db/__init__.py +++ b/src/xml2db/__init__.py @@ -8,6 +8,7 @@ DataModelRelationN, DataModelRelation1, ) +from .config import ModelConfig, TableConfig, FieldConfig, load_config, parse_yaml_config __all__ = [ "DataModel", @@ -20,4 +21,9 @@ "DataModelColumn", "DataModelRelation1", "DataModelRelationN", + "ModelConfig", + "TableConfig", + "FieldConfig", + "load_config", + "parse_yaml_config", ] diff --git a/src/xml2db/cli.py b/src/xml2db/cli.py new file mode 100644 index 0000000..2b4c9fd --- /dev/null +++ b/src/xml2db/cli.py @@ -0,0 +1,669 @@ +"""Command-line interface for xml2db.""" +from __future__ import annotations + +import argparse +import html as _html +import json +import os +import threading +import webbrowser +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from typing import Optional + +from .config import load_config, parse_yaml_config +from .model import DataModel + + +# --------------------------------------------------------------------------- +# render command +# --------------------------------------------------------------------------- + +def _get_sa_dialect(db_type: str | None): + """Return a SQLAlchemy dialect instance for DDL compilation, or None for generic.""" + if db_type is None: + return None + from sqlalchemy.dialects import postgresql, mssql, mysql + _map = { + "postgresql": postgresql, + "mssql": mssql, + "mysql": mysql, + "mariadb": mysql, + } + module = _map.get(db_type) + return module.dialect() if module is not None else None + + +def cmd_import(args: argparse.Namespace) -> None: + config = load_config(args.config) if args.config else None + metadata = dict(kv.split("=", 1) for kv in args.metadata) if args.metadata else None + + model = DataModel( + xsd_file=args.xsd_file, + short_name=args.short_name, + model_config=config, + connection_string=args.connection_string, + db_schema=args.db_schema, + ) + doc = model.parse_xml( + xml_file=args.xml_file, + metadata=metadata, + skip_validation=not args.validate, + iterparse=not args.no_iterparse, + recover=args.recover, + ) + stats = doc.insert_into_target_tables() + print( + f"Imported {args.xml_file}: " + f"{stats.inserted} rows inserted, {stats.existing} rows already existed " + f"({stats.duration_temp_insert:.2f}s staging, " + f"{stats.duration_merge:.2f}s merge, " + f"{stats.duration_cleanup:.2f}s cleanup)" + ) + + +def cmd_render(args: argparse.Namespace) -> None: + config = load_config(args.config) if args.config else None + db_type = getattr(args, "db_type", None) + model = DataModel( + xsd_file=args.xsd_file, + short_name=args.short_name, + model_config=config, + db_type=db_type, + ) + fmt = args.format + if fmt == "erd": + sa_dialect = _get_sa_dialect(db_type) if args.db_names else None + output = model.get_entity_rel_diagram(text_context=False, use_db_names=args.db_names, sa_dialect=sa_dialect) + elif fmt == "target-tree": + output = model.target_tree + elif fmt == "source-tree": + output = model.source_tree + else: # ddl + sa_dialect = _get_sa_dialect(db_type) + parts = [str(s.compile(dialect=sa_dialect)) for s in model.get_all_create_table_statements()] + parts += [str(s.compile(dialect=sa_dialect)) + "\n\n" for s in model.get_all_create_index_statements()] + output = "".join(parts) + + if args.output: + with open(args.output, "w", encoding="utf-8") as f: + f.write(output) + print(f"Written to {args.output}") + else: + print(output) + + +# --------------------------------------------------------------------------- +# serve command: shared state +# --------------------------------------------------------------------------- + +class _State: + def __init__( + self, + xsd_file: str, + config_file: Optional[str], + short_name: str, + db_type: Optional[str], + ): + self.xsd_file = xsd_file + self.config_file = config_file if config_file else "model_config.yml" + self.short_name = short_name + self.db_type = db_type + self.lock = threading.Lock() + + self.config_yaml = "" + if config_file and os.path.exists(config_file): + with open(config_file, encoding="utf-8") as f: + self.config_yaml = f.read() + + self.outputs: dict = {"erd": "", "target_tree": "", "source_tree": "", "ddl": ""} + self.schema_info: dict = {} + self.build_error = "" + self._rebuild(self.config_yaml) + + def _rebuild(self, yaml_text: str) -> tuple[dict | None, str]: + """Build all outputs from yaml_text. On success updates state and returns (outputs, ""). + On failure updates build_error and returns (None, error).""" + try: + cfg = parse_yaml_config(yaml_text) if yaml_text.strip() else None + model = DataModel( + xsd_file=self.xsd_file, + short_name=self.short_name, + model_config=cfg, + db_type=self.db_type, + ) + sa_dialect = _get_sa_dialect(self.db_type) + ddl_parts = [ + str(s.compile(dialect=sa_dialect)) + for s in model.get_all_create_table_statements() + ] + ddl_parts += [ + str(s.compile(dialect=sa_dialect)) + "\n\n" + for s in model.get_all_create_index_statements() + ] + outputs = { + "erd": model.get_entity_rel_diagram(text_context=False), + "erd_db": model.get_entity_rel_diagram(text_context=False, use_db_names=True, sa_dialect=sa_dialect), + "target_tree": model.target_tree, + "source_tree": model.source_tree, + "ddl": "".join(ddl_parts), + } + schema_info = {} + for table in model.tables.values(): + target_fields = sorted( + list(table.columns.keys()) + + list(table.relations_1.keys()) + + list(table.relations_n.keys()) + ) + source_fields = sorted(set( + fn for (tn, fn) in model.fields_transforms + if tn == table.type_name + )) + schema_info[table.name] = { + "target": target_fields, + "source": source_fields, + } + with self.lock: + self.outputs = outputs + self.schema_info = schema_info + self.build_error = "" + self.config_yaml = yaml_text + return outputs, "" + except Exception as exc: + error = str(exc) + with self.lock: + self.build_error = error + return None, error + + def apply_yaml(self, text: str) -> tuple[dict, str]: + """Rebuild from new YAML text. Always returns (current_outputs, error) so the + browser keeps the last successful output visible when a build fails.""" + self._rebuild(text) + with self.lock: + return dict(self.outputs), self.build_error + + def save_config(self, text: str) -> str: + """Write text to config_file. Returns error string or "".""" + try: + with open(self.config_file, "w", encoding="utf-8") as f: + f.write(text) + return "" + except OSError as exc: + return str(exc) + + +# --------------------------------------------------------------------------- +# HTTP handler +# --------------------------------------------------------------------------- + +def _make_handler(state: _State): + class Handler(BaseHTTPRequestHandler): + def log_message(self, fmt, *args): # silence default stderr logging + pass + + def do_GET(self): + if self.path == "/": + self._serve_html() + else: + self.send_error(404) + + def do_POST(self): + length = int(self.headers.get("Content-Length", 0)) + body = self.rfile.read(length).decode("utf-8", errors="replace") + if self.path == "/config": + outputs, error = state.apply_yaml(body) + self._json({"outputs": outputs, "error": error}) + elif self.path == "/save": + error = state.save_config(body) + self._json({"error": error, "saved_to": state.config_file}) + else: + self.send_error(404) + + def _json(self, data: dict): + body = json.dumps(data).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def _serve_html(self): + body = _build_html(state).encode("utf-8") + self.send_response(200) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + return Handler + + +# --------------------------------------------------------------------------- +# HTML template +# --------------------------------------------------------------------------- + +_HTML = """\ + + + + +xml2db: TMPL_TITLE + + + + + +
+ xml2db + TMPL_HEADER +
+
+
+
model_config (YAML)
+
+
+ +
+ +
+ + + +""" + + +def _build_html(state: _State) -> str: + header = os.path.basename(state.xsd_file) + return ( + _HTML + .replace("TMPL_TITLE", _html.escape(header)) + .replace("TMPL_HEADER", _html.escape(header)) + .replace("TMPL_SAVE_PATH", _html.escape(state.config_file)) + .replace("TMPL_SCHEMA_INFO_JSON", json.dumps(state.schema_info)) + .replace("TMPL_CONFIG_YAML_JSON", json.dumps(state.config_yaml)) + .replace("TMPL_INITIAL_OUTPUTS", json.dumps(state.outputs)) + .replace("TMPL_INITIAL_ERROR", json.dumps(state.build_error)) + ) + + +# --------------------------------------------------------------------------- +# serve command +# --------------------------------------------------------------------------- + +def cmd_serve(args: argparse.Namespace) -> None: + print(f"Loading schema: {args.xsd_file}") + state = _State( + xsd_file=args.xsd_file, + config_file=args.config, + short_name=args.short_name, + db_type=getattr(args, "db_type", None), + ) + if state.build_error: + print(f"Warning: initial build error: {state.build_error}") + + server = ThreadingHTTPServer(("127.0.0.1", args.port), _make_handler(state)) + url = f"http://127.0.0.1:{args.port}" + print(f"Serving at {url} (Ctrl+C to stop)") + + if not args.no_browser: + threading.Timer(0.4, webbrowser.open, args=[url]).start() + + try: + server.serve_forever() + except KeyboardInterrupt: + server.shutdown() + + +# --------------------------------------------------------------------------- +# CLI entry point +# --------------------------------------------------------------------------- + +def main() -> None: + parser = argparse.ArgumentParser( + prog="xml2db", + description="xml2db: explore and configure XSD-to-database mappings", + ) + sub = parser.add_subparsers(dest="command", required=True) + + i = sub.add_parser("import", help="Parse an XML file and load it into a database") + i.add_argument("xml_file", help="Path to the XML file to import") + i.add_argument("xsd_file", help="Path to the XSD schema file") + i.add_argument("--connection-string", "-d", required=True, metavar="DSN", + help="SQLAlchemy connection string (e.g. postgresql+psycopg2://user:pw@host/db)") + i.add_argument("--config", "-c", metavar="FILE", help="YAML model config file") + i.add_argument("--short-name", default="DocumentRoot", metavar="NAME", + help="Data model short name (default: DocumentRoot)") + i.add_argument("--db-schema", metavar="SCHEMA", default=None, + help="Database schema to use") + i.add_argument("--metadata", "-m", nargs="*", metavar="KEY=VALUE", + help="Metadata values for root table metadata_columns (e.g. -m source=file.xml)") + i.add_argument("--validate", action="store_true", + help="Validate the XML against the schema before importing") + i.add_argument("--no-iterparse", action="store_true", + help="Use recursive parser instead of iterparse (higher memory usage)") + i.add_argument("--recover", action="store_true", + help="Attempt to parse malformed XML") + + r = sub.add_parser("render", help="Print ERD, tree or DDL to stdout or a file") + r.add_argument("xsd_file", help="Path to the XSD schema file") + r.add_argument("--config", "-c", metavar="FILE", help="YAML model config file") + r.add_argument( + "--format", "-f", + choices=["erd", "target-tree", "source-tree", "ddl"], + default="erd", + help="Output format (default: erd)", + ) + r.add_argument("--output", "-o", metavar="FILE", help="Write to file instead of stdout") + r.add_argument("--short-name", default="DocumentRoot", metavar="NAME", + help="Data model short name (default: DocumentRoot)") + r.add_argument("--db-type", metavar="BACKEND", default=None, + help="Database backend for DDL output (postgresql, mssql, mysql, …)") + r.add_argument("--db-names", action="store_true", + help="Use physical database identifiers in the ERD instead of logical names") + + s = sub.add_parser("serve", help="Launch an interactive schema explorer in the browser") + s.add_argument("xsd_file", help="Path to the XSD schema file") + s.add_argument("--config", "-c", metavar="FILE", + help="YAML model config file (loaded on startup; Save button writes it back)") + s.add_argument("--port", "-p", type=int, default=8765, metavar="PORT", + help="HTTP port (default: 8765)") + s.add_argument("--no-browser", action="store_true", + help="Do not open the browser automatically") + s.add_argument("--short-name", default="DocumentRoot", metavar="NAME", + help="Data model short name (default: DocumentRoot)") + s.add_argument("--db-type", metavar="BACKEND", default=None, + help="Database backend for DDL tab (postgresql, mssql, mysql, …)") + + args = parser.parse_args() + if args.command == "import": + cmd_import(args) + elif args.command == "render": + cmd_render(args) + else: + cmd_serve(args) + + +if __name__ == "__main__": + main() diff --git a/src/xml2db/config.py b/src/xml2db/config.py new file mode 100644 index 0000000..cc78022 --- /dev/null +++ b/src/xml2db/config.py @@ -0,0 +1,254 @@ +"""Typed config definitions and YAML loading utilities for xml2db.""" +from __future__ import annotations + +import re +from typing import Any, TypedDict + +import sqlalchemy as sa + +from .exceptions import DataModelConfigError + +# Keys that require Python callables; cannot be expressed in YAML +_CALLABLE_ONLY_KEYS: frozenset[str] = frozenset( + {"document_tree_hook", "document_tree_node_hook", "record_hash_constructor"} +) + +# --------------------------------------------------------------------------- +# SQLAlchemy type resolver +# --------------------------------------------------------------------------- + +_SA_TYPE_MAP: dict[str, type] = { + "String": sa.String, + "Text": sa.Text, + "Integer": sa.Integer, + "BigInteger": sa.BigInteger, + "SmallInteger": sa.SmallInteger, + "Float": sa.Float, + "Double": sa.Double, + "Numeric": sa.Numeric, + "Boolean": sa.Boolean, + "DateTime": sa.DateTime, + "Date": sa.Date, + "Time": sa.Time, + "LargeBinary": sa.LargeBinary, + "JSON": sa.JSON, + "Uuid": sa.Uuid, + "UUID": sa.Uuid, +} + +_TYPE_RE = re.compile(r"^(\w+)(?:\(([^)]*)\))?$") + + +def _parse_type_arg(s: str) -> Any: + s = s.strip() + if s in ("True", "true"): + return True + if s in ("False", "false"): + return False + try: + return int(s) + except ValueError: + pass + try: + return float(s) + except ValueError: + pass + if len(s) >= 2 and s[0] in ('"', "'") and s[-1] == s[0]: + return s[1:-1] + raise DataModelConfigError(f"Cannot parse type argument value: '{s}'") + + +def resolve_sa_type(type_spec: Any) -> Any: + """Resolve a SQLAlchemy type name string to a type instance, or pass through. + + Args: + type_spec: A string such as ``"String(100)"``, ``"Integer"``, + ``"DateTime(timezone=True)"``, or an existing SQLAlchemy type + instance / class (returned unchanged). + + Returns: + A SQLAlchemy type instance. + + Raises: + DataModelConfigError: If the string cannot be parsed or names an + unknown type. + """ + if not isinstance(type_spec, str): + return type_spec # already a SQLAlchemy type; pass through + + m = _TYPE_RE.match(type_spec.strip()) + if not m: + raise DataModelConfigError(f"Cannot parse SQLAlchemy type: '{type_spec}'") + + type_name, args_str = m.group(1), m.group(2) + if type_name not in _SA_TYPE_MAP: + raise DataModelConfigError( + f"Unknown SQLAlchemy type '{type_name}'. " + f"Supported: {', '.join(sorted(_SA_TYPE_MAP))}" + ) + + type_cls = _SA_TYPE_MAP[type_name] + if not args_str or not args_str.strip(): + return type_cls() + + args: list = [] + kwargs: dict = {} + for part in args_str.split(","): + part = part.strip() + if not part: + continue + if "=" in part: + k, _, v = part.partition("=") + kwargs[k.strip()] = _parse_type_arg(v) + else: + args.append(_parse_type_arg(part)) + + return type_cls(*args, **kwargs) + + +# --------------------------------------------------------------------------- +# TypedDicts +# --------------------------------------------------------------------------- + +class FieldConfig(TypedDict, total=False): + type: Any # str (e.g. "String(100)") or SQLAlchemy type (Python only when not a str) + rename: str + transform: Any # str | False + + +class IndexConfig(TypedDict, total=False): + name: str # Index name (required) + columns: list[str] # Column names (required) + unique: bool + + +class TableConfig(TypedDict, total=False): + reuse: bool + as_columnstore: bool + choice_transform: bool + # In YAML: list of IndexConfig dicts. In Python: list/tuple of SQLAlchemy + # schema objects, or a zero-argument callable returning such a list. + extra_args: list[IndexConfig] | list[Any] | Any + fields: dict[str, FieldConfig] + + +class MetadataColumnConfig(TypedDict, total=False): + name: str # Required: column name + type: Any # Required: str (e.g. "String(100)") or SQLAlchemy type + nullable: bool + default: Any + server_default: Any + comment: str + index: bool + unique: bool + + +class ModelConfig(TypedDict, total=False): + as_columnstore: bool + row_numbers: bool + transform: Any # False or "auto" (default) + document_tree_hook: Any # callable, Python only + document_tree_node_hook: Any # callable, Python only + record_hash_column_name: str + record_hash_constructor: Any # callable, Python only + record_hash_size: int + metadata_columns: list[MetadataColumnConfig] + tables: dict[str, TableConfig] + + +# --------------------------------------------------------------------------- +# YAML parsing + validation +# --------------------------------------------------------------------------- + +def _check_config(data: dict, *, from_yaml: bool) -> None: + """Validate a raw config dict, raising DataModelConfigError on Python-only values.""" + if not from_yaml: + return # dict configs are trusted; no checks needed + + # Model-level callable-only keys + for key in _CALLABLE_ONLY_KEYS: + if key in data: + raise DataModelConfigError( + f"'{key}' requires a Python callable and cannot be set in a YAML config " + "file. Pass a Python dict with the callable directly instead." + ) + + # metadata_columns: each entry's 'type' must be a string + for i, col_cfg in enumerate(data.get("metadata_columns", [])): + if not isinstance(col_cfg, dict): + raise DataModelConfigError( + f"metadata_columns[{i}] must be a mapping with at least 'name' and 'type'." + ) + if "type" in col_cfg and not isinstance(col_cfg["type"], str): + raise DataModelConfigError( + f"metadata_columns[{i}].type must be a string (e.g. 'String(100)') " + "in a YAML config file." + ) + + # Table- and field-level checks + for table_name, table_cfg in data.get("tables", {}).items(): + if not isinstance(table_cfg, dict): + continue + # extra_args must be a list (callable not possible from YAML) + if "extra_args" in table_cfg and callable(table_cfg["extra_args"]): + raise DataModelConfigError( + f"tables.{table_name}.extra_args is a callable and cannot be set in a " + "YAML config file. Pass a Python dict with the callable directly instead." + ) + for field_name, field_cfg in table_cfg.get("fields", {}).items(): + if not isinstance(field_cfg, dict): + continue + if "type" in field_cfg and not isinstance(field_cfg["type"], str): + raise DataModelConfigError( + f"tables.{table_name}.fields.{field_name}.type must be a string " + "(e.g. 'String(100)') in a YAML config file." + ) + + +def parse_yaml_config(text: str) -> ModelConfig: + """Parse a YAML string into a model config dict. + + Args: + text: YAML text representing a model config mapping. + + Returns: + A :class:`ModelConfig` dict. + + Raises: + DataModelConfigError: If the config contains keys that require Python + objects (callables or SQLAlchemy types) or if the top level is not + a mapping. + ImportError: If PyYAML is not installed. + """ + try: + import yaml + except ImportError: + raise ImportError( + "PyYAML is required to parse YAML config. Install with: pip install pyyaml" + ) + + data = yaml.safe_load(text) + if data is None: + return {} + if not isinstance(data, dict): + raise DataModelConfigError("Config must be a YAML mapping at the top level.") + + _check_config(data, from_yaml=True) + return data + + +def load_config(path: str) -> ModelConfig: + """Load a model config from a YAML file. + + Args: + path: Path to a YAML file containing the model config. + + Returns: + A :class:`ModelConfig` dict. + + Raises: + DataModelConfigError: If the file contains keys that require Python objects. + ImportError: If PyYAML is not installed. + """ + with open(path, encoding="utf-8") as f: + return parse_yaml_config(f.read()) diff --git a/src/xml2db/model.py b/src/xml2db/model.py index 79818f1..ba7bf0d 100644 --- a/src/xml2db/model.py +++ b/src/xml2db/model.py @@ -147,6 +147,16 @@ def _validate_config(self, cfg): ("metadata_columns", list, []), ] } + transform_raw = cfg.get("transform", "auto") + if transform_raw is False or transform_raw == "false": + model_config["transform"] = False + elif transform_raw is True or transform_raw == "auto": + model_config["transform"] = True + else: + raise DataModelConfigError( + f"Invalid 'transform' value: '{transform_raw}'. Use false to disable all" + " automatic field transformations, or auto (the default) to keep them." + ) return model_config @property @@ -568,21 +578,27 @@ def get_occurs(particle): return parent_table - def get_entity_rel_diagram(self, text_context: bool = True) -> str: + def get_entity_rel_diagram( + self, text_context: bool = True, use_db_names: bool = False, sa_dialect: object = None + ) -> str: """Build an entity relationship diagram for the data model The ERD syntax is used by mermaid.js to create a visual representation of the diagram, which is supported by Pycharm IDE or GitHub in markdown files, among others Args: - text_context: Should we add a title, a text explanation, etc. or just the ERD? + text_context: should we add a title, a text explanation, etc. or just the ERD? + use_db_names: if True, use the physical database identifier for table and + column names, and the compiled SQL type for column types. + sa_dialect: SQLAlchemy dialect instance for SQL type compilation when + use_db_names is True. Falls back to generic type names when None. Returns: A string representation of the ERD """ out = ["erDiagram"] for tb in self.fk_ordered_tables_reversed: - out += tb.get_entity_rel_diagram() + out += tb.get_entity_rel_diagram(use_db_names=use_db_names, sa_dialect=sa_dialect) if text_context: out = ( diff --git a/src/xml2db/table/column.py b/src/xml2db/table/column.py index 877874c..2eedc4d 100644 --- a/src/xml2db/table/column.py +++ b/src/xml2db/table/column.py @@ -3,6 +3,8 @@ from sqlalchemy import Column +from ..config import resolve_sa_type + if TYPE_CHECKING: from ..model import DataModel @@ -101,6 +103,10 @@ def get_sqlalchemy_column(self, temp: bool = False) -> Iterable[Column]: temp: temp table or target table ? """ field_config = self.config.get("fields", {}).get(self.name, {}) - column_type = field_config.get("type") or self.data_model.dialect.column_type(self, temp) + raw_type = field_config.get("type") + column_type = ( + resolve_sa_type(raw_type) if raw_type is not None + else self.data_model.dialect.column_type(self, temp) + ) db_col = self.data_model.dialect.db_identifier(field_config.get("rename", self.name)) yield Column(db_col, column_type, key=self.name) diff --git a/src/xml2db/table/reused_table.py b/src/xml2db/table/reused_table.py index a0a9fe3..a0ea178 100644 --- a/src/xml2db/table/reused_table.py +++ b/src/xml2db/table/reused_table.py @@ -12,6 +12,7 @@ from .column import DataModelColumn from .transformed_table import DataModelTableTransformed +from ..config import resolve_sa_type def shorten_str(x: str, max_len: int = 30) -> str: @@ -61,7 +62,7 @@ def get_col(temp=False): for metadata_col in self.data_model.model_config["metadata_columns"]: yield Column( metadata_col["name"], - metadata_col["type"], + resolve_sa_type(metadata_col["type"]), **{ k: v for k, v in metadata_col.items() diff --git a/src/xml2db/table/table.py b/src/xml2db/table/table.py index db64a99..735465e 100644 --- a/src/xml2db/table/table.py +++ b/src/xml2db/table/table.py @@ -14,6 +14,22 @@ logger = logging.getLogger(__name__) +def _resolve_extra_args(extra_args): + """Convert dict-form index specs to sqlalchemy.Index objects, pass others through.""" + if callable(extra_args): + return extra_args # callable, Python-only, unchanged + result = [] + for item in extra_args: + if isinstance(item, dict): + name = item.get("name") + columns = item.get("columns", []) + kwargs = {k: v for k, v in item.items() if k not in ("name", "columns")} + result.append(sqlalchemy.Index(name, *columns, **kwargs)) + else: + result.append(item) + return result + + class DataModelTable: """A class representing a database table translated from an XML schema complex type @@ -106,11 +122,14 @@ def _validate_config(self, cfg): or callable(cfg["extra_args"]) ): raise DataModelConfigError("extra_args must be a list, a tuple or callable") - config["extra_args"] = cfg.get("extra_args", []) + config["extra_args"] = _resolve_extra_args(cfg.get("extra_args", [])) if "choice_transform" in cfg: - config["choice_transform"] = check_type( - cfg, "choice_transform", bool, False - ) + if cfg["choice_transform"] == "auto": + config["choice_transform"] = "auto" + else: + config["choice_transform"] = check_type( + cfg, "choice_transform", bool, False + ) config["fields"] = cfg.get("fields", {}) config = self.data_model.dialect.validate_table_config(config) @@ -377,30 +396,68 @@ def drop_temp_tables(self, engine: sqlalchemy.engine.base.Engine) -> None: rel.temp_rel_table.drop(engine, checkfirst=True) self.temp_table.drop(engine, checkfirst=True) - def get_entity_rel_diagram(self) -> List: + def get_entity_rel_diagram(self, use_db_names: bool = False, sa_dialect=None) -> List: """Build ERD representation for a single table and its relationships The string representation is used by mermaid.js to create a visual diagram. + Args: + use_db_names: if True, use the physical database identifier for table and + column names, and the compiled SQL type for column types. + sa_dialect: SQLAlchemy dialect instance used to compile SQL types when + use_db_names is True. Falls back to generic SQL type names when None. + Returns: a list of strings (lines) """ + d = self.data_model.dialect + if use_db_names: + tname = d.db_identifier(self.name) + col_by_key = {col.key: col for col in self.table.c} + else: + tname = self.name + + def other_name(tb): + return d.db_identifier(tb.name) if use_db_names else tb.name + + def col_name(logical): + if use_db_names: + sa_col = col_by_key.get(logical) + return sa_col.name if sa_col is not None else logical.replace(".", "_") + return logical.replace(".", "_") + + def col_type(logical): + if use_db_names: + sa_col = col_by_key.get(logical) + if sa_col is not None: + return ( + sa_col.type.compile(dialect=sa_dialect) + if sa_dialect is not None + else str(sa_col.type) + ) + return self.columns[logical].data_type + + def col_type_suffix(logical): + if use_db_names: + return "" + return "-N" if self.columns[logical].occurs[1] is None else "" + out = ( [ - f"{self.name} ||--{'o' if rel.occurs[0] == 0 else '|'}| {rel.other_table.name} : " + f"{tname} ||--{'o' if rel.occurs[0] == 0 else '|'}| {other_name(rel.other_table)} : " f'"{rel.name}"' for rel in self.relations_1.values() ] + [ - f"{self.name} ||--{'o' if rel.occurs[0] == 0 else '|'}{{ {rel.other_table.name} : " + f"{tname} ||--{'o' if rel.occurs[0] == 0 else '|'}{{ {other_name(rel.other_table)} : " f"\"{rel.name}{'*' if rel.other_table.is_reused else ''}\"" for rel in self.relations_n.values() ] - + [f"{self.name} {{"] + + [f"{tname} {{"] + [ ( - f" {self.columns[field[1]].data_type}{'-N' if self.columns[field[1]].occurs[1] is None else ''} " - f"{field[1].replace('.', '_')}" + f" {col_type(field[1])}{col_type_suffix(field[1])} " + f"{col_name(field[1])}" ) for field in self.fields if field[0] == "col" diff --git a/src/xml2db/table/transformed_table.py b/src/xml2db/table/transformed_table.py index 65382dd..16fc806 100644 --- a/src/xml2db/table/transformed_table.py +++ b/src/xml2db/table/transformed_table.py @@ -36,7 +36,7 @@ def _is_table_choice_transform_applicable(self) -> bool: Returns: True if choice transform is to be applied, False otherwise. """ - if "choice_transform" in self.config: + if "choice_transform" in self.config and self.config["choice_transform"] != "auto": if isinstance(self.config["choice_transform"], bool): if self.config["choice_transform"]: if self._can_choice_transform_table(): @@ -53,7 +53,7 @@ def _is_table_choice_transform_applicable(self) -> bool: f"Unrecognized choice_transform value '{self.config['choice_transform']}'" f" for table '{self.name}'. Only boolean values True or False are allowed." ) - elif self._can_choice_transform_table() and len(self.columns) > 2: + elif self.data_model.model_config.get("transform", True) and self._can_choice_transform_table() and len(self.columns) > 2: # column number isn't reduced if the number of columns = 2, as it would be elevated to 2 columns then return True return False @@ -137,21 +137,21 @@ def _get_field_transform( The default transformation that should be applied """ field_config = self.config.get("fields", {}).get(field_name, {}) - if "transform" in field_config: - if field_config["transform"] is False: + field_transform = field_config.get("transform") + if field_transform is not None and field_transform != "auto": + if field_transform is False: return None - if field_config["transform"] == "skip": + if field_transform == "skip": return "skip" - if self._can_transform_field( - field_type, field_name, field_config["transform"] - ): - return field_config["transform"] + if self._can_transform_field(field_type, field_name, field_transform): + return field_transform else: raise DataModelConfigError( - f"Transform value '{field_config['transform']}' cannot be applied" + f"Transform value '{field_transform}' cannot be applied" f" to field '{field_name}' of table '{self.name}'." ) - else: + # "auto" at field level always runs auto-detection, overriding global transform:false + if field_transform == "auto" or self.data_model.model_config.get("transform", True): if field_type == "col": if self._can_transform_field("col", field_name, "join"): return "join" diff --git a/tests/test_bulk_insert.py b/tests/test_bulk_insert.py index 5bcd939..6848cdd 100644 --- a/tests/test_bulk_insert.py +++ b/tests/test_bulk_insert.py @@ -177,7 +177,7 @@ def test_duckdb_bulk_insert_quoted_csv_field_after_large_unquoted_sample(duckdb_ read_csv bypasses auto-detection and must always be present. """ table = _make_table(duckdb_engine, "quoted_field_test") - # 'vals' value that contains a comma — document.py's 'join' transform can produce + # 'vals' value that contains a comma; document.py's 'join' transform can produce # strings like '"val,ue",other' which csv.writer then wraps in outer quotes, # yielding a quoted CSV cell. problematic_value = '"val,ue",other_value' diff --git a/tests/test_bulk_insert_mssql.py b/tests/test_bulk_insert_mssql.py index c595eff..a583ae3 100644 --- a/tests/test_bulk_insert_mssql.py +++ b/tests/test_bulk_insert_mssql.py @@ -300,7 +300,7 @@ def test_mssql_bulk_load_true_raises_without_bcp(mssql_engine): import unittest.mock as mock if shutil.which("bcp") is None: - # BCP already absent — just confirm the error is raised directly. + # BCP already absent; just confirm the error is raised directly. table, meta = _make_table(mssql_engine, "mssql_bi_bulk_load_true_no_bcp") try: records = [{"id": i, "label": f"r{i}"} for i in range(_BCP_THRESHOLD)] @@ -311,7 +311,7 @@ def test_mssql_bulk_load_true_raises_without_bcp(mssql_engine): finally: _drop(meta, mssql_engine) else: - # BCP present — patch shutil.which to simulate absence. + # BCP present; patch shutil.which to simulate absence. table, meta = _make_table(mssql_engine, "mssql_bi_bulk_load_true_no_bcp") try: records = [{"id": i, "label": f"r{i}"} for i in range(_BCP_THRESHOLD)] diff --git a/tests/test_config.py b/tests/test_config.py new file mode 100644 index 0000000..d6333e5 --- /dev/null +++ b/tests/test_config.py @@ -0,0 +1,217 @@ +"""Tests for config.py (resolve_sa_type, parse_yaml_config) and model-level +transform option.""" +import os +import textwrap + +import pytest +import sqlalchemy as sa + +from xml2db import DataModel +from xml2db.config import parse_yaml_config, resolve_sa_type +from xml2db.exceptions import DataModelConfigError + +ORDERS_XSD = os.path.join("tests", "sample_models", "orders", "orders.xsd") + + +# --------------------------------------------------------------------------- +# resolve_sa_type +# --------------------------------------------------------------------------- + + +class TestResolveSaType: + def test_simple_name(self): + t = resolve_sa_type("Integer") + assert isinstance(t, sa.Integer) + + def test_string_with_length(self): + t = resolve_sa_type("String(100)") + assert isinstance(t, sa.String) + assert t.length == 100 + + def test_datetime_with_kwarg(self): + t = resolve_sa_type("DateTime(timezone=True)") + assert isinstance(t, sa.DateTime) + assert t.timezone is True + + def test_numeric_with_args(self): + t = resolve_sa_type("Numeric(10, 2)") + assert isinstance(t, sa.Numeric) + assert t.precision == 10 + assert t.scale == 2 + + def test_passthrough_sa_instance(self): + instance = sa.String(50) + assert resolve_sa_type(instance) is instance + + def test_passthrough_sa_class(self): + assert resolve_sa_type(sa.Integer) is sa.Integer + + def test_unknown_type_raises(self): + with pytest.raises(DataModelConfigError, match="Unknown SQLAlchemy type"): + resolve_sa_type("NotAType") + + def test_malformed_string_raises(self): + with pytest.raises(DataModelConfigError, match="Cannot parse"): + resolve_sa_type("String(") + + def test_uuid_alias(self): + assert isinstance(resolve_sa_type("UUID"), sa.Uuid) + + +# --------------------------------------------------------------------------- +# parse_yaml_config +# --------------------------------------------------------------------------- + + +class TestParseYamlConfig: + def test_empty_string_returns_empty_dict(self): + assert parse_yaml_config("") == {} + + def test_none_yaml_returns_empty_dict(self): + assert parse_yaml_config("# just a comment\n") == {} + + def test_valid_config(self): + yaml = textwrap.dedent("""\ + row_numbers: true + tables: + shiporder: + reuse: false + fields: + orderid: + type: String(50) + rename: order_id + """) + cfg = parse_yaml_config(yaml) + assert cfg["row_numbers"] is True + assert cfg["tables"]["shiporder"]["reuse"] is False + assert cfg["tables"]["shiporder"]["fields"]["orderid"]["rename"] == "order_id" + + def test_non_mapping_raises(self): + with pytest.raises(DataModelConfigError, match="mapping"): + parse_yaml_config("- item1\n- item2\n") + + def test_callable_key_rejected(self): + for key in ("document_tree_hook", "document_tree_node_hook", "record_hash_constructor"): + with pytest.raises(DataModelConfigError, match="callable"): + parse_yaml_config(f"{key}: something\n") + + def test_metadata_columns_type_must_be_string(self): + yaml = textwrap.dedent("""\ + metadata_columns: + - name: col1 + type: 123 + """) + with pytest.raises(DataModelConfigError, match="must be a string"): + parse_yaml_config(yaml) + + def test_metadata_columns_string_type_accepted(self): + yaml = textwrap.dedent("""\ + metadata_columns: + - name: col1 + type: String(256) + """) + cfg = parse_yaml_config(yaml) + assert cfg["metadata_columns"][0]["type"] == "String(256)" + + def test_field_type_must_be_string(self): + yaml = textwrap.dedent("""\ + tables: + t: + fields: + f: + type: 42 + """) + with pytest.raises(DataModelConfigError, match="must be a string"): + parse_yaml_config(yaml) + + +# --------------------------------------------------------------------------- +# Top-level transform option +# --------------------------------------------------------------------------- + + +class TestGlobalTransformOption: + def _make_model(self, config): + return DataModel(xsd_file=ORDERS_XSD, model_config=config) + + def test_default_elevates_orderperson(self): + m = self._make_model({}) + # With default (auto), orderperson is elevated into shipordertype + assert "orderperson" not in m.tables["shipordertype"].relations_1 + assert any(c.startswith("orderperson_") for c in m.tables["shipordertype"].columns) + + def test_transform_false_keeps_relations(self): + m = self._make_model({"transform": False}) + # With transform:false, orderperson stays as a child relation + assert "orderperson" in m.tables["shipordertype"].relations_1 + assert not any(c.startswith("orderperson_") for c in m.tables["shipordertype"].columns) + + def test_transform_false_keeps_more_tables(self): + m_auto = self._make_model({}) + m_off = self._make_model({"transform": False}) + # No elevation means child tables are not absorbed + assert len(m_off.tables) > len(m_auto.tables) + + def test_transform_auto_string_same_as_default(self): + m_default = self._make_model({}) + m_auto = self._make_model({"transform": "auto"}) + assert list(m_auto.tables["shipordertype"].columns.keys()) == \ + list(m_default.tables["shipordertype"].columns.keys()) + + def test_transform_true_same_as_default(self): + m_default = self._make_model({}) + m_true = self._make_model({"transform": True}) + assert list(m_true.tables["shipordertype"].columns.keys()) == \ + list(m_default.tables["shipordertype"].columns.keys()) + + def test_invalid_transform_value_raises(self): + with pytest.raises(DataModelConfigError, match="Invalid 'transform'"): + self._make_model({"transform": "none"}) + + def test_transform_false_per_field_auto_overrides_global(self): + # Global false but explicit per-field "auto" should restore default elevation + m_off = self._make_model({"transform": False}) + m_field_auto = self._make_model({ + "transform": False, + "tables": {"shiporder": {"fields": {"orderperson": {"transform": "auto"}}}}, + }) + # With global false, orderperson is a relation + assert "orderperson" in m_off.tables["shipordertype"].relations_1 + # With field-level auto override, orderperson is elevated again + assert "orderperson" not in m_field_auto.tables["shipordertype"].relations_1 + assert any( + c.startswith("orderperson_") + for c in m_field_auto.tables["shipordertype"].columns + ) + + +# --------------------------------------------------------------------------- +# Field-level transform: "auto" and choice_transform: "auto" +# --------------------------------------------------------------------------- + + +class TestAutoTransformValue: + def _make_model(self, config): + return DataModel(xsd_file=ORDERS_XSD, model_config=config) + + def test_field_transform_auto_same_as_omitted(self): + m_omit = self._make_model({}) + m_auto = self._make_model({ + "tables": {"shiporder": {"fields": {"orderperson": {"transform": "auto"}}}} + }) + assert list(m_auto.tables["shipordertype"].columns.keys()) == \ + list(m_omit.tables["shipordertype"].columns.keys()) + + def test_field_transform_auto_does_not_suppress_elevation(self): + m = self._make_model({ + "tables": {"shiporder": {"fields": {"orderperson": {"transform": "auto"}}}} + }) + assert "orderperson" not in m.tables["shipordertype"].relations_1 + + def test_choice_transform_auto_same_as_omitted(self): + m_omit = self._make_model({}) + m_auto = self._make_model({ + "tables": {"companyId": {"choice_transform": "auto"}} + }) + # Both should produce the same set of tables + assert set(m_auto.tables.keys()) == set(m_omit.tables.keys()) diff --git a/tests/test_roundtrip.py b/tests/test_roundtrip.py index a87c5a3..56a5600 100644 --- a/tests/test_roundtrip.py +++ b/tests/test_roundtrip.py @@ -39,7 +39,7 @@ def test_load_stats(conn_string): assert stats.duration_merge > 0 assert stats.duration_cleanup > 0 - # second load of same file — reused rows should be existing (backends that + # second load of same file; reused rows should be existing (backends that # report rowcount); on DuckDB both will be 0, which is also acceptable doc2 = model.parse_xml(xml_path, metadata={"src": "b"}) stats2 = doc2.insert_into_target_tables()