diff --git a/README.md b/README.md index c649b75..e9a6743 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,64 @@ # Loading XML files into a relational database -`xml2db` is a Python package which allows parsing and loading XML files into a relational database. It handles complex +`xml2db` is a Python package that parses and loads XML files into a relational database. It handles complex XML files which cannot be denormalized to flat tables, and works out of the box, without any custom mapping rules. -It can be used within an [Extract, Load, Transform](https://docs.getdbt.com/terms/elt) data pipeline pattern as it -allows loading XML files into a relational data model which is very close from the source data, yet easy to work with. +It fits naturally into an [Extract, Load, Transform](https://docs.getdbt.com/terms/elt) pipeline: it +loads XML files into a relational data model that stays close to the source data while remaining easy to query as +flat database tables. The raw data can then be transformed using [DBT](https://www.getdbt.com/), SQL views, or stored procedures to produce +more user-friendly tables. Starting from an XSD schema which represents a given XML structure, `xml2db` builds a data model, i.e. a set of database tables linked to each other by foreign keys relationships. Then, it allows parsing and loading XML files into the database, and getting them back from the database into XML format if needed. +This package uses `sqlalchemy` to interact with the database, so it should work with different database backends. +Automated integration tests run against PostgreSQL, MySQL, MS SQL Server and DuckDB. You may have to install additional +packages to connect to your database (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for +MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB). + +**Please read the [package documentation website](https://cre-dev.github.io/xml2db) for all the details!** + +## Installation + +The package can be installed, preferably in a virtual environment, using `pip`: + +``` bash +pip install xml2db +``` + +## CLI + +After installation, `xml2db` is available as a command-line tool with three subcommands. + +Explore your XSD schema and configure the data model interactively in a browser: + +```bash +xml2db serve path/to/schema.xsd +``` + +This opens a page with an Entity Relationship Diagram, source/target tree views, DDL output, and a live YAML config +editor with autocomplete. + +Import an XML file directly from the command line: + +```bash +xml2db import file.xml schema.xsd \ + --connection-string "postgresql+psycopg2://user:pw@host/db" \ + --config model_config.yml +``` + +Render the ERD, trees, or DDL to stdout or a file without starting a server: + +```bash +xml2db render schema.xsd --format erd +xml2db render schema.xsd --format ddl --db-type postgresql +``` + +See the [CLI reference](https://cre-dev.github.io/xml2db/cli/) for all options. + +## Python API + Loading XML files into a relational database with `xml2db` can be as simple as: ```python @@ -28,28 +77,6 @@ document = data_model.parse_xml( document.insert_into_target_tables() ``` -The data model created by `xml2db` will be close to the XSD schema. However, `xml2db` will perform a few systematic -simplifications aimed at limiting the complexity of the resulting data model and the storage footprint. The resulting -data model can be configured, but the above code will work out of the box, with reasonable defaults. - -The raw data loaded into the database can then be processed if need be, using for instance [DBT](https://www.getdbt.com/), -SQL views or stored procedures aimed at extracting, correcting and formatting the data into more user-friendly tables. - -This package uses `sqlalchemy` to interact with the database, so it should work with different database backends. -Automated integration tests run against PostgreSQL, MySQL, MS SQL Server and DuckDB. You may have to install additional -packages to connect to your database (e.g. `psycopg2` or `psycopg` for PostgreSQL, `pymysql` or `mysqlclient` for -MySQL, `pyodbc` for MS SQL Server, or `duckdb-engine` for DuckDB). - -**Please read the [package documentation website](https://cre-dev.github.io/xml2db) for all the details!** - -## Installation - -The package can be installed, preferably in a virtual environment, using `pip`: - -``` bash -pip install xml2db -``` - ## Testing Running the tests requires installing additional development dependencies, after cloning the repo, with: diff --git a/docs/api/overview.md b/docs/api/overview.md index 064b7ab..e1d9a29 100644 --- a/docs/api/overview.md +++ b/docs/api/overview.md @@ -38,7 +38,7 @@ for lower level steps. It can be useful for advanced use cases, for instance: * transforming the data in intermediate steps, * adding logging, -* limiting concurrent access to the database within a multiprocess setup, etc. +* limiting concurrent access to the database in a [multiprocessing context](../how_it_works.md#multiprocessing), etc. For those scenarios you can easily reimplement [`Document.insert_into_target_tables`](document.md/#xml2db.document.Document.insert_into_target_tables) to suit your @@ -59,74 +59,6 @@ flowchart TB end ``` -### Multiprocessing example - -XML parsing is CPU-bound and scales well across processes. Loading into the -database, however, must be coordinated to avoid conflicts on shared tables. -The right level of synchronisation depends on the backend: - -* **DuckDB (file-based)**: only one active writer is allowed at a time, so - all database I/O must be serialised. -* **PostgreSQL, MS SQL Server, …**: concurrent writes to *different* temp - tables are safe (each process gets a unique temp-table prefix), but the final - merge into the shared target tables should be serialised. - -The simplest approach (and the one shown below) is to serialise the entire -database phase with a `multiprocessing.Lock`, keeping only the parsing step -parallel. This works correctly for all backends. - -```python -import multiprocessing -from xml2db import DataModel - - -def load_one_file(xml_path, xsd_path, connection_string, lock): - # Each process creates its own DataModel with a unique temp_prefix. - model = DataModel( - xsd_file=xsd_path, - connection_string=connection_string, - ) - # XML parsing is CPU-bound and runs in parallel across all processes. - doc = model.parse_xml(xml_path) - - # Serialise all database I/O across processes. - with lock: - doc.insert_into_target_tables() - model.engine.dispose() - - -if __name__ == "__main__": - xsd_path = "schema.xsd" - connection_string = "duckdb:///data.duckdb" - xml_files = ["file1.xml", "file2.xml", "file3.xml"] - - lock = multiprocessing.Lock() - processes = [ - multiprocessing.Process( - target=load_one_file, - args=(xml_path, xsd_path, connection_string, lock), - ) - for xml_path in xml_files - ] - for p in processes: - p.start() - for p in processes: - p.join() - if p.exitcode != 0: - raise RuntimeError(f"Worker failed with exit code {p.exitcode}") -``` - -!!! Note - For backends that support concurrent writers, you can increase throughput - by splitting - [`Document.insert_into_target_tables`](document.md/#xml2db.document.Document.insert_into_target_tables) - into separate calls to - [`Document.insert_into_temp_tables`](document.md/#xml2db.document.Document.insert_into_temp_tables) - (run concurrently, since each process has a unique temp-table prefix, so - there are no collisions) and - [`Document.merge_into_target_tables`](document.md/#xml2db.document.Document.merge_into_target_tables) - (serialised via lock). - ## *Advanced use:* get data from the database back to XML The flow chart below presents data conversions used to get back data from the database into XML, showing the functions diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 0000000..9cd91a0 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,116 @@ +--- +title: "CLI usage" +description: "Reference for the xml2db command-line interface: import XML files, render ERDs and DDL, and launch the interactive browser explorer." +--- + +# CLI usage + +The `xml2db` CLI provides three subcommands: `import`, `render`, and `serve`. + +## xml2db import + +Parse an XML file and load it into a database. + +``` +xml2db import XML_FILE XSD_FILE --connection-string DSN [options] +``` + +**Positional arguments:** + +| Argument | Description | +|---|---| +| `XML_FILE` | Path to the XML file to import | +| `XSD_FILE` | Path to the XSD schema file | + +**Options:** + +| Option | Description | +|---|---| +| `--connection-string DSN`, `-d DSN` | SQLAlchemy connection string (required) | +| `--config FILE`, `-c FILE` | YAML model config file | +| `--db-schema SCHEMA` | Database schema to use | +| `--metadata KEY=VALUE`, `-m KEY=VALUE` | Metadata values for `metadata_columns` (repeatable) | +| `--short-name NAME` | Data model short name (default: `DocumentRoot`) | +| `--no-iterparse` | Use the recursive parser instead of iterparse (higher memory usage) | +| `--recover` | Attempt to parse malformed XML | +| `--validate` | Validate the XML against the schema before importing | + +**Example:** + +```bash +xml2db import file.xml schema.xsd \ + --connection-string "postgresql+psycopg2://user:pw@host/db" \ + --config model_config.yml \ + --metadata source=file.xml +``` + +On success, the command prints the number of rows inserted and already-existing (deduplicated), with per-phase timings. + +## xml2db render + +Print an ERD, source/target tree, or DDL to stdout or a file, without starting a server. + +``` +xml2db render XSD_FILE [options] +``` + +**Positional arguments:** + +| Argument | Description | +|---|---| +| `XSD_FILE` | Path to the XSD schema file | + +**Options:** + +| Option | Description | +|---|---| +| `--config FILE`, `-c FILE` | YAML model config file | +| `--db-names` | Use physical database identifiers in the ERD instead of logical names | +| `--db-type BACKEND` | Database backend for DDL output (`postgresql`, `mssql`, `mysql`, ...) | +| `--format FORMAT`, `-f FORMAT` | Output format: `erd` (default), `target-tree`, `source-tree`, or `ddl` | +| `--output FILE`, `-o FILE` | Write output to a file instead of stdout | +| `--short-name NAME` | Data model short name (default: `DocumentRoot`) | + +**Examples:** + +```bash +xml2db render schema.xsd --format erd +xml2db render schema.xsd --format target-tree +xml2db render schema.xsd --format source-tree +xml2db render schema.xsd --format ddl --db-type postgresql +xml2db render schema.xsd --format erd --output diagram.md +``` + +## xml2db serve + +Launch an interactive schema explorer in the browser. + +``` +xml2db serve XSD_FILE [options] +``` + +The explorer shows four tabs: ERD, target tree, source tree, and DDL. The left panel is a YAML config editor with autocomplete for table names, field names, and all config options. Edits trigger an automatic rebuild. The **Save** button writes the config back to disk. + +**Positional arguments:** + +| Argument | Description | +|---|---| +| `XSD_FILE` | Path to the XSD schema file | + +**Options:** + +| Option | Description | +|---|---| +| `--config FILE`, `-c FILE` | YAML model config file to load on startup; Save writes it back to this path (default: `model_config.yml`) | +| `--db-type BACKEND` | Database backend for the DDL tab (`postgresql`, `mssql`, `mysql`, ...) | +| `--no-browser` | Do not open the browser automatically | +| `--port PORT`, `-p PORT` | HTTP port (default: `8765`) | +| `--short-name NAME` | Data model short name (default: `DocumentRoot`) | + +**Example:** + +```bash +xml2db serve schema.xsd --config model_config.yml --db-type postgresql +``` + +See [Getting started](getting_started.md) for a walkthrough of the explorer. diff --git a/docs/configuring.md b/docs/configuring.md index 8fd5ba1..cf0ae91 100644 --- a/docs/configuring.md +++ b/docs/configuring.md @@ -80,6 +80,9 @@ model_config = load_config("model_config.yml") The following options can be passed as top-level keys of the model configuration `dict`: +* `as_columnstore` (`bool`): for MS SQL Server, create clustered columnstore indexes on all tables. This can be also set up at +the table level for each table. However, for `n-n` relationships tables, this option is the only way to configure the +clustered columnstore indexes. The default value is `False` (disabled). * `document_tree_hook` (`Callable`): sets a hook function which can modify the data extracted from the XML. It gives direct access to the underlying tree data structure just before it is extracted to be loaded to the database. This can be used, for instance, to prune or modify some parts of the document tree before loading it into the database. The document tree @@ -89,24 +92,21 @@ the declarative [`"transform": "skip"`](#skipping-fields) option is simpler. similar with `document_tree_hook`, but it is called as soon as a node is completed, not waiting for the entire parsing to finish. It is especially useful if you intend to filter out some nodes and reduce memory footprint while parsing. For straightforward field exclusion, see [`"transform": "skip"`](#skipping-fields). -* `row_numbers` (`bool`): adds `xml2db_row_number` columns either to `n-n` relationships tables, or directly to data tables when -deduplication of rows is opted out. This allows recording the original order of elements in the source XML, which is not -always respected otherwise. It was implemented primarily for round-trip tests, but could serve other purposes. The -default value is `False` (disabled). -* `as_columnstore` (`bool`): for MS SQL Server, create clustered columnstore indexes on all tables. This can be also set up at -the table level for each table. However, for `n-n` relationships tables, this option is the only way to configure the -clustered columnstore indexes. The default value is `False` (disabled). * `metadata_columns` (`list`): a list of extra columns that you want to add to the root table of your model. This is useful for instance to add the name of the file which has been parsed, or a timestamp, etc. Columns should be specified as dicts, the only required keys are `name` and `type` (a SQLAlchemy type object); other keys will be passed directly as keyword arguments to `sqlalchemy.Column`. Actual values need to be passed to [`DataModel.parse_xml`](api/data_model.md#xml2db.model.DataModel.parse_xml) for each parsed documents, as a `dict`, using the `metadata` argument. -* `transform` (`false` or `"auto"`): set to `false` to disable all automatic field transformations globally: no joining of multi-value columns, no elevation of child tables, no collapsing of choice groups. The default `"auto"` applies all of these where applicable. Per-field `transform` and per-table `choice_transform` still override the global setting. * `record_hash_column_name`: the column name to use to store records hash data (defaults to `xml2db_record_hash`). * `record_hash_constructor`: a function used to build a hash, with a signature similar to `hashlib` constructor functions (defaults to `hashlib.sha1`). * `record_hash_size`: the byte size of the record hash (defaults to 20, which is the size of a `sha-1` hash). +* `row_numbers` (`bool`): adds `xml2db_row_number` columns either to `n-n` relationships tables, or directly to data tables when +deduplication of rows is opted out. This allows recording the original order of elements in the source XML, which is not +always respected otherwise. It was implemented primarily for round-trip tests, but could serve other purposes. The +default value is `False` (disabled). +* `transform` (`false` or `"auto"`): set to `false` to disable all automatic field transformations globally: no joining of multi-value columns, no elevation of child tables, no collapsing of choice groups. The default `"auto"` applies all of these where applicable. Per-field `transform` and per-table `choice_transform` still override the global setting. ## Fields configuration diff --git a/docs/getting_started.md b/docs/getting_started.md index 67d6daf..60d0a9c 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -39,7 +39,8 @@ This opens a browser with four tabs: - **Source tree**: a text tree of the raw XSD structure before simplification - **DDL**: `CREATE TABLE` statements for the target schema -The left panel is a YAML config editor with autocomplete for table names, field names, and all config options. Edit the config and the diagram updates automatically (with a short debounce). When the config looks right, click **Save** to write it to a file (default: `model_config.yml`). +The left panel is a YAML config editor with autocomplete for table names, field names, and all config options. Edit the config and the diagram updates automatically. +When the config looks right, click **Save** to write it to a file (default: `model_config.yml`). You can also render these representations directly to stdout or a file without the browser: @@ -54,7 +55,7 @@ See [Configuring your data model](configuring.md) for a full description of the ## Importing XML files -Once the data model looks right, import an XML file into the database: +Once you are happy with the data model, import an XML file into the database (config is optional): ``` bash xml2db import file.xml schema.xsd \ @@ -96,7 +97,7 @@ with open("data_model_erd.md", "w") as f: f.write(data_model.get_entity_rel_diagram()) ``` -The diagram uses [Mermaid](https://mermaid.js.org/syntax/entityRelationshipDiagram.html). PyCharm and GitHub both render Mermaid natively. +The diagram uses [Mermaid](https://mermaid.js.org/syntax/entityRelationshipDiagram.html). Your IDE should be able to render Mermaid preview. ``` py title="Write source and target trees to files" linenums="1" with open("source_tree.txt", "w") as f: diff --git a/docs/how_it_works.md b/docs/how_it_works.md index 56d5b86..484da0a 100644 --- a/docs/how_it_works.md +++ b/docs/how_it_works.md @@ -230,6 +230,76 @@ can be broken down into lower-level calls if you need to separate the temporary- [`Document.insert_into_target_tables`](api/document.md#xml2db.document.Document.insert_into_target_tables) source code for how these steps fit together. +### Multiprocessing + +XML parsing is CPU-bound and scales well across processes. Loading into the +database, however, must be coordinated to avoid conflicts on shared tables. +The right level of synchronisation depends on the backend: + +* **DuckDB (file-based)**: only one active writer is allowed at a time, so + all database I/O must be serialised. +* **PostgreSQL, MS SQL Server, …**: concurrent writes to *different* temp + tables are safe (each process gets a unique temp-table prefix), but the final + merge into the shared target tables should be serialised. + +The simplest approach (and the one shown below) is to serialise the entire +database phase with a `multiprocessing.Lock`, keeping only the parsing step +parallel. This works correctly for all backends. + +``` py title="Speed up ingestion with multiprocessing" linenums="1" +import multiprocessing +from xml2db import DataModel + + +def load_one_file(xml_path, xsd_path, connection_string, lock): + # Each process creates its own DataModel with a unique temp_prefix. + model = DataModel( + xsd_file=xsd_path, + connection_string=connection_string, + ) + # XML parsing is CPU-bound and runs in parallel across all processes. + doc = model.parse_xml(xml_path) + + # Serialise all database I/O across processes. + with lock: + doc.insert_into_target_tables() + model.engine.dispose() + + +if __name__ == "__main__": + xsd_path = "schema.xsd" + connection_string = "duckdb:///data.duckdb" + xml_files = ["file1.xml", "file2.xml", "file3.xml"] + + lock = multiprocessing.Lock() + processes = [ + multiprocessing.Process( + target=load_one_file, + args=(xml_path, xsd_path, connection_string, lock), + ) + for xml_path in xml_files + ] + for p in processes: + p.start() + for p in processes: + p.join() + if p.exitcode != 0: + raise RuntimeError(f"Worker failed with exit code {p.exitcode}") +``` + +!!! Note + For backends that support concurrent writers, you can increase throughput + by splitting + [`Document.insert_into_target_tables`](api/document.md#xml2db.document.Document.insert_into_target_tables) + into separate calls to lower level methods + [`Document.insert_into_temp_tables`](api/document.md#xml2db.document.Document.insert_into_temp_tables) + (run concurrently, since each process has a unique temp-table prefix, so + there are no collisions) and + [`Document.merge_into_target_tables`](api/document.md#xml2db.document.Document.merge_into_target_tables) + (serialised via lock). + See the [API overview](api/overview.md#advanced-use-loading-data-into-the-database) for more details. + + ## Extracting the data back to XML Extracting data from the database and converting it back to XML follows similar steps, in reverse order. diff --git a/docs/index.md b/docs/index.md index 0c823ad..3a72127 100644 --- a/docs/index.md +++ b/docs/index.md @@ -9,7 +9,7 @@ description: "xml2db is a Python package that automatically maps an XSD schema t * it automatically maps an XSD schema to a set of tables in the database * it can handle complex XML files which cannot be denormalized into flat tables -* it works out of the box, without any custom mapping rules. +* it works out of the box, without any custom mapping rules, while still configurable. `xml2db` fits naturally into an ETL or [ELT (Extract, Load, Transform)](https://docs.getdbt.com/terms/elt) pipeline. It loads XML files into a relational model that stays close to the source data while remaining easy to query as flat database @@ -30,7 +30,8 @@ MS SQL Server, and DuckDB. You may need to install a connector package (e.g. `ps ## Visualising data models `xml2db` generates visual diagrams of your data model directly from an XSD file, using -[Mermaid](https://mermaid.js.org/syntax/entityRelationshipDiagram.html) to represent tables and their relationships. +[Mermaid](https://mermaid.js.org/syntax/entityRelationshipDiagram.html) to represent tables and their relationships. You can see this representation and others +through the interactive configuration panel in your browser (`xml2db serve`). It looks like this: diff --git a/mkdocs.yml b/mkdocs.yml index 5df29c8..1850517 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -2,7 +2,7 @@ site_name: xml2db site_url: https://cre-dev.github.io/xml2db repo_url: https://github.com/cre-dev/xml2db repo_name: cre-dev/xml2db -copyright: Copyright © 2024 Commission de régulation de l'énergie +copyright: Copyright © 2026 Commission de régulation de l'énergie site_description: xml2db is a Python package to load XML files into a relational database. It handles complex XML and works out of the box, without any custom mapping rules. @@ -11,6 +11,7 @@ nav: - Getting started: "getting_started.md" - Configuring: "configuring.md" - How it works: "how_it_works.md" + - CLI usage: "cli.md" - API: - Overview: "api/overview.md" - DataModel: "api/data_model.md" diff --git a/src/xml2db/cli.py b/src/xml2db/cli.py index 2b4c9fd..50ccae6 100644 --- a/src/xml2db/cli.py +++ b/src/xml2db/cli.py @@ -4,7 +4,9 @@ import argparse import html as _html import json +import logging import os +import sys import threading import webbrowser from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer @@ -52,9 +54,12 @@ def cmd_import(args: argparse.Namespace) -> None: recover=args.recover, ) stats = doc.insert_into_target_tables() + if stats.row_counts_available: + counts = f"{stats.inserted} rows inserted, {stats.existing} rows already existed" + else: + counts = "import complete (row counts not available for this backend)" print( - f"Imported {args.xml_file}: " - f"{stats.inserted} rows inserted, {stats.existing} rows already existed " + f"Imported {args.xml_file}: {counts} " f"({stats.duration_temp_insert:.2f}s staging, " f"{stats.duration_merge:.2f}s merge, " f"{stats.duration_cleanup:.2f}s cleanup)" @@ -241,8 +246,7 @@ def _serve_html(self): # HTML template # --------------------------------------------------------------------------- -_HTML = """\ - +_HTML = """ @@ -311,6 +315,15 @@ def _serve_html(self): background: #fafafa; flex-shrink: 0; } #erd-names.visible { display: flex; } #erd-names label { display: flex; align-items: center; gap: 3px; cursor: pointer; } + #zoom-controls { display: flex; align-items: center; gap: 4px; margin-left: auto; } + .zoom-btn { padding: 2px 8px; border: 1px solid #bbb; border-radius: 3px; cursor: pointer; + font-size: 13px; background: #f0f0f0; line-height: 1.4; } + .zoom-btn:hover { background: #e0e0e0; } + #zoom-level { font-size: 11px; color: #555; min-width: 36px; text-align: center; } + #content.erd-mode { overflow: hidden; padding: 0; cursor: grab; user-select: none; } + #content.erd-mode.dragging { cursor: grabbing; } + #content.erd-mode svg { max-width: none; } + #erd-canvas { transform-origin: 0 0; display: inline-block; padding: 14px; height: 100%; width: 100%; } @@ -336,6 +349,12 @@ def _serve_html(self): Names & types: +
+ + 100% + + +
@@ -385,7 +404,7 @@ def _serve_html(self): const li = text.length - s.length; if (li >= searchIndent) continue; // Match optional list marker ("- ") then a key name followed by ":" - const m = s.match(/^(?:-\s+)?(\S[^:#]*?):/); + const m = s.match(/^(?:-\\s+)?(\\S[^:#]*?):/); if (m) { path.unshift(m[1].trim()); searchIndent = li; } } return { indent: rawIndent, path }; @@ -422,7 +441,7 @@ def _serve_html(self): function getValueCompletions(state, pos, path) { const line = state.doc.lineAt(pos); const before = line.text.slice(0, pos - line.from); - const m = before.match(/(\w+)\s*:\s*$/); + const m = before.match(/(\\w+)\\s*:\\s*$/); if (!m) return null; const key = m[1]; if (BOOL_KEYS.has(key)) return ['true','false'].map(v => ({ label: v, type: 'keyword' })); @@ -433,7 +452,7 @@ def _serve_html(self): } function xml2dbCompleter(context) { - const word = context.matchBefore(/\w*/); + const word = context.matchBefore(/\\w*/); if (!word || (word.from === word.to && !context.explicit)) return null; const { path } = getContext(context.state, context.pos); const valueOpts = getValueCompletions(context.state, context.pos, path); @@ -454,12 +473,33 @@ def _serve_html(self): let currentTab = 'erd'; let debounceTimer = null; let mermaidCounter = 0; +let erdScale = 1, erdX = 0, erdY = 0; +let erdLastContent = null; +let dragState = null; function erdKey() { return document.querySelector('input[name="erd_names"]:checked').value === 'db' ? 'erd_db' : 'erd'; } +function updateErdTransform() { + const canvas = document.getElementById('erd-canvas'); + if (canvas) canvas.style.transform = `translate(${erdX}px,${erdY}px) scale(${erdScale})`; +} +function updateZoomLabel() { + const el = document.getElementById('zoom-level'); + if (el) el.textContent = Math.round(erdScale * 100) + '%'; +} +function applyZoom(newScale, cx, cy) { + newScale = Math.max(0.1, Math.min(10, newScale)); + const ratio = newScale / erdScale; + erdX = cx - ratio * (cx - erdX); + erdY = cy - ratio * (cy - erdY); + erdScale = newScale; + updateErdTransform(); + updateZoomLabel(); +} + const view = new EditorView({ doc: TMPL_CONFIG_YAML_JSON, extensions: [ @@ -490,16 +530,27 @@ def _serve_html(self): if (currentTab === 'erd') { erdNamesEl.classList.add('visible'); const erd = (outputs && outputs[erdKey()]) || ''; - if (!erd) { contentEl.innerHTML = '

No ERD available.

'; return; } + if (!erd) { + contentEl.classList.remove('erd-mode'); + contentEl.innerHTML = '

No ERD available.

'; + return; + } try { const id = 'g' + (++mermaidCounter); const { svg } = await mermaid.render(id, erd); - contentEl.innerHTML = svg; + if (erd !== erdLastContent) { erdScale = 1; erdX = 0; erdY = 0; erdLastContent = erd; } + contentEl.classList.add('erd-mode'); + contentEl.innerHTML = '
'; + document.getElementById('erd-canvas').innerHTML = svg; + updateErdTransform(); + updateZoomLabel(); } catch(e) { + contentEl.classList.remove('erd-mode'); contentEl.innerHTML = '
' + escapeHtml(String(e)) + '
'; } } else { erdNamesEl.classList.remove('visible'); + contentEl.classList.remove('erd-mode'); contentEl.innerHTML = '
' + escapeHtml((outputs && outputs[currentTab]) || '') + '
'; } } @@ -517,6 +568,43 @@ def _serve_html(self): radio.addEventListener('change', () => { if (currentTab === 'erd') renderTab(); }); }); +document.getElementById('zoom-in').addEventListener('click', () => { + const r = contentEl.getBoundingClientRect(); + applyZoom(erdScale * 1.25, r.width / 2, r.height / 2); +}); +document.getElementById('zoom-out').addEventListener('click', () => { + const r = contentEl.getBoundingClientRect(); + applyZoom(erdScale / 1.25, r.width / 2, r.height / 2); +}); +document.getElementById('zoom-reset').addEventListener('click', () => { + erdScale = 1; erdX = 0; erdY = 0; + updateErdTransform(); + updateZoomLabel(); +}); + +contentEl.addEventListener('wheel', e => { + if (!contentEl.classList.contains('erd-mode')) return; + e.preventDefault(); + const rect = contentEl.getBoundingClientRect(); + applyZoom(erdScale * (e.deltaY < 0 ? 1.15 : 1 / 1.15), e.clientX - rect.left, e.clientY - rect.top); +}, { passive: false }); + +contentEl.addEventListener('mousedown', e => { + if (!contentEl.classList.contains('erd-mode') || e.button !== 0) return; + dragState = { startX: e.clientX - erdX, startY: e.clientY - erdY }; + contentEl.classList.add('dragging'); + e.preventDefault(); +}); +window.addEventListener('mousemove', e => { + if (!dragState) return; + erdX = e.clientX - dragState.startX; + erdY = e.clientY - dragState.startY; + updateErdTransform(); +}); +window.addEventListener('mouseup', () => { + if (dragState) { dragState = null; contentEl.classList.remove('dragging'); } +}); + async function doRebuild() { try { const r = await fetch('/config', { @@ -601,6 +689,12 @@ def cmd_serve(args: argparse.Namespace) -> None: # --------------------------------------------------------------------------- def main() -> None: + # Show WARNING and below on stderr; ERROR+ is handled by the except block below. + _handler = logging.StreamHandler() + _handler.setFormatter(logging.Formatter("%(message)s")) + _handler.addFilter(lambda r: r.levelno < logging.ERROR) + logging.basicConfig(handlers=[_handler], level=logging.WARNING) + parser = argparse.ArgumentParser( prog="xml2db", description="xml2db: explore and configure XSD-to-database mappings", @@ -657,12 +751,18 @@ def main() -> None: help="Database backend for DDL tab (postgresql, mssql, mysql, …)") args = parser.parse_args() - if args.command == "import": - cmd_import(args) - elif args.command == "render": - cmd_render(args) - else: - cmd_serve(args) + try: + if args.command == "import": + cmd_import(args) + elif args.command == "render": + cmd_render(args) + else: + cmd_serve(args) + except KeyboardInterrupt: + pass + except Exception as e: + print(f"error: {e}", file=sys.stderr) + raise SystemExit(1) if __name__ == "__main__": diff --git a/src/xml2db/document.py b/src/xml2db/document.py index eb87da1..0178ff0 100644 --- a/src/xml2db/document.py +++ b/src/xml2db/document.py @@ -25,18 +25,21 @@ class MergeStats: Attributes: inserted: Rows from the document that were written to target tables. - May be 0 on backends that do not report rowcount for ``INSERT … FROM SELECT`` - (e.g. DuckDB). + 0 when ``row_counts_available`` is ``False``. existing: Rows from the document that were not written because an identical record was already present (hash-deduplicated for reused tables, or because a parent record was already existing for duplicated tables). - May be 0 on backends that do not report rowcount for ``INSERT … FROM SELECT``. + 0 when ``row_counts_available`` is ``False``. duration: Seconds spent executing merge statements. + row_counts_available: ``False`` when the backend does not report rowcount for + ``INSERT … FROM SELECT`` (e.g. DuckDB); ``inserted`` and ``existing`` are + then meaningless. """ inserted: int existing: int duration: float + row_counts_available: bool = True @dataclass @@ -45,15 +48,17 @@ class LoadStats: Attributes: inserted: Rows from the document that were written to target tables. - May be 0 on backends that do not report rowcount for ``INSERT … FROM SELECT`` - (e.g. DuckDB). + 0 when ``row_counts_available`` is ``False``. existing: Rows from the document that were not written because an identical record was already present (hash-deduplicated for reused tables, or because a parent record was already existing for duplicated tables). - May be 0 on backends that do not report rowcount for ``INSERT … FROM SELECT``. + 0 when ``row_counts_available`` is ``False``. duration_temp_insert: Seconds spent inserting data into temporary staging tables. duration_merge: Seconds spent merging temporary tables into target tables. duration_cleanup: Seconds spent dropping temporary tables. + row_counts_available: ``False`` when the backend does not report rowcount for + ``INSERT … FROM SELECT`` (e.g. DuckDB); ``inserted`` and ``existing`` are + then meaningless. """ inserted: int @@ -61,6 +66,7 @@ class LoadStats: duration_temp_insert: float duration_merge: float duration_cleanup: float + row_counts_available: bool = True class Document: @@ -476,6 +482,7 @@ def merge_into_target_tables(self, single_transaction: bool = True) -> MergeStat """ inserted = 0 existing = 0 + row_counts_available = False t0 = time.perf_counter() for tables in ( [self.model.fk_ordered_tables] @@ -494,17 +501,24 @@ def merge_into_target_tables(self, single_transaction: bool = True) -> MergeStat # INSERT … FROM SELECT (e.g. DuckDB); skip those tables. if result.rowcount >= 0: table_inserted = result.rowcount + row_counts_available = True if table_inserted is not None: inserted += table_inserted if tb.is_reused and tb.type_name in self.data: existing += ( len(self.data[tb.type_name]["records"]) - table_inserted ) - if inserted == 0: - logger.info("No rows were inserted!") - else: - logger.info(f"Inserted rows: {inserted}, existing rows: {existing}") - return MergeStats(inserted=inserted, existing=existing, duration=time.perf_counter() - t0) + if row_counts_available: + if inserted == 0: + logger.info("No rows were inserted!") + else: + logger.info(f"Inserted rows: {inserted}, existing rows: {existing}") + return MergeStats( + inserted=inserted, + existing=existing, + duration=time.perf_counter() - t0, + row_counts_available=row_counts_available, + ) def insert_into_target_tables( self, @@ -574,6 +588,7 @@ def insert_into_target_tables( duration_temp_insert=duration_temp, duration_merge=merge_stats.duration, duration_cleanup=duration_cleanup, + row_counts_available=merge_stats.row_counts_available, ) def extract_from_database( diff --git a/src/xml2db/model.py b/src/xml2db/model.py index ba7bf0d..049fd6a 100644 --- a/src/xml2db/model.py +++ b/src/xml2db/model.py @@ -32,7 +32,7 @@ class DataModel: It is the main entry point for `xml2db`. This class allows parsing an XSD file to build a representation of the XML schema, simplify it and convert it into - a set of database tables. It also allows [parsing XML documents](./#xml2db.model.DataModel.parse_xml) that fit this + a set of database tables. It also allows [parsing XML documents](data_model.md#xml2db.model.DataModel.parse_xml) that fit this XML schema and importing their content into a database. Args: