Resumable HTTP uploads

config/simdb.cfg

@@ -34,3 +34,4 @@ result_backend = redis://redis:6379/0
 
 [partition]
 data = /data/simdb/partition
+http = /data/simdb/http

docker-compose.yml

@@ -20,6 +20,7 @@ services:
       - ./validation:/app/validation:ro
       - ./config:/app/config:ro
       - ./tmp/partition_data:/data/simdb/partition:ro
+      - ./tmp/http:/data/simdb/http
       - ./upload_folder:/data/simdb/simulations
     depends_on:
       redis:
@@ -38,6 +39,7 @@ services:
     volumes:
       - ./config:/app/config:ro
       - ./tmp/partition_data:/data/simdb/partition:ro
+      - ./tmp/http:/data/simdb/http
       - ./upload_folder:/data/simdb/simulations
     depends_on:
       redis:

docs/cli.md

@@ -14,7 +14,6 @@ Options:
 Commands:
   alias       Query remote and local aliases.
   config      Query/update application configuration.
-  database    Manage local simulation database.
   manifest    Create/check manifest file.
   provenance  Create the PROVENANCE_FILE from the current system.
   remote      Interact with the remote SimDB service.
@@ -142,7 +141,6 @@ Options:
   --help  Show this message and exit.
 ```
 
-
 ## Manifest
 
 
@@ -421,30 +419,38 @@ Usage: simdb remote [NAME] query [OPTIONS] [CONSTRAINTS]...
       NAME=[mod]VALUE
 
   Where `[mod]` is an optional query modifier. Available query modifiers are:
-      eq:  - This checks for equality (this is the same behaviour as not providing any modifier).
+      eq:  - This checks for equality (this is the same behaviour as not providing
+             any modifier).
       in:  - This searches inside the value instead of looking for exact matches.
       gt:  - This checks for values greater than the given quantity.
       agt: - This checks for any array elements are greater than the given quantity.
       ge:  - This checks for values greater than or equal to the given quantity.
-      age: - This checks for any array elements are greater than or equal to the given quantity.
+      age: - This checks for any array elements are greater than or equal to the given
+             quantity.
       lt:  - This checks for values less than the given quantity.
-      alt:  - This checks for any array elements are less than the given quantity.
+      alt: - This checks for any array elements are less than the given quantity.
       le:  - This checks for values less than or equal to the given quantity.
-      ale:  - This checks for any array elements are less than or equal to the given quantity.
+      ale: - This checks for any array elements are less than or equal to the given
+              quantity.
 
   Modifier examples:
       alias=eq:foo                                                performs exact match
-      summary.code.name=in:foo                                    matches all names containing foo
-      summary.heating_current_drive.power_additional.value=agt:0  matches all simulations where any array element
-      of summary.heating_current_drive.power_additional.value is greater than 0
+      summary.code.name=in:foo                                    matches all names
+          containing foo
+      summary.heating_current_drive.power_additional.value=agt:0  matches all
+          simulations where any array element of
+          summary.heating_current_drive.power_additional.value is greater than 0
 
-  Any string comparisons are done in a case-insensitive manner. If multiple constraints are provided then simulations
-  are returned that match all given constraints.
+  Any string comparisons are done in a case-insensitive manner. If multiple
+  constraints are provided then simulations are returned that match all given
+  constraints.
 
   Examples:
-      sim remote query workflow.name=in:test       finds all simulations where workflow.name contains test
-                                                       (case-insensitive)
-      sim remote query pulse=gt:1000 run=0         finds all simulations where pulse is > 1000 and run = 0
+      sim remote query workflow.name=in:test       finds all simulations where
+                                                   workflow.name contains test
+                                                   (case-insensitive)
+      sim remote query pulse=gt:1000 run=0         finds all simulations where pulse
+                                                   is > 1000 and run = 0
 
 Options:
   -m, --meta-data TEXT  Additional meta-data field to print.
@@ -603,20 +609,44 @@ Options:
   --help  Show this message and exit.
 
 Commands:
-  delete    Delete the ingested simulation with given SIM_ID (UUID or...
-  info      Print information on the simulation with given SIM_ID (UUID...
-  ingest    Ingest a MANIFEST_FILE.
-  list      List ingested simulations.
-  modify    Modify the ingested simulation.
-  pull      Pull the simulation with the given SIM_ID (UUID or alias)...
-  push      Push the simulation with the given SIM_ID (UUID or alias) to...
-  query     Perform a metadata query to find matching local simulations.
-  validate  Validate the ingested simulation with given SIM_ID (UUID or...
+  data        Fetch IDS field data for simulation SIM_ID (UUID or alias)...
+  delete      Delete the ingested simulation with given SIM_ID (UUID or...
+  info        Print information on the simulation with given SIM_ID (UUID...
+  ingest      Ingest a MANIFEST_FILE.
+  list        List ingested simulations.
+  modify      Modify the ingested simulation.
+  pull        Pull the simulation with the given SIM_ID (UUID or alias)...
+  push        Push the simulation with the given SIM_ID (UUID or alias)...
+  push_http   Push the simulation with the given SIM_ID to the REMOTE...
+  push_local  Push the simulation with the given SIM_ID (UUID or alias)...
+  query       Perform a metadata query to find matching local simulations.
+  validate    Validate the ingested simulation with given SIM_ID (UUID or...
 ```
 
 
 ```text
-Usage: simdb simulation delete [OPTIONS] SIM_ID
+Usage: simdb simulation data [OPTIONS] [REMOTE] SIM_ID IDS_PATH
+
+  Fetch IDS field data for simulation SIM_ID (UUID or alias) from REMOTE.
+
+  IDS_PATH format:
+      ids_name[:<occurrence>]/path/to/field
+
+  Examples:
+      simdb sim data iter 4dd781b... profiles_1d[0]/grid/rho_tor_norm
+      simdb sim data 4dd781b... equilibrium:0/time_slice[0]/profiles_1d/psi
+
+Options:
+  --username TEXT    Username used to authenticate with the remote.
+  --password TEXT    Password used to authenticate with the remote.
+  --dd-version TEXT  Convert IDS data to the requested Data Dictionary
+                     version, e.g. 4.1.1.
+  --help             Show this message and exit.
+```
+
+
+```text
+Usage: simdb simulation delete [OPTIONS] [SIM_ID]
 
   Delete the ingested simulation with given SIM_ID (UUID or alias).
 
@@ -703,6 +733,41 @@ Options:
 ```
 
 
+```text
+Usage: simdb simulation push_http [OPTIONS] [REMOTE] SIM_ID
+
+  Push the simulation with the given SIM_ID to the REMOTE over resumable HTTP.
+
+  Unlike push_local, this does not require a filesystem shared with the
+  server: the file bytes are uploaded over HTTP using a resumable protocol and
+  staged into the server's 'http' partition. An interrupted push can be
+  resumed by re-running the command.
+
+Options:
+  --username TEXT  Username used to authenticate with the remote.
+  --password TEXT  Password used to authenticate with the remote.
+  --replaces TEXT  SIM_ID of simulation to deprecate and replace.
+  --add-watcher    Add the current user as a watcher of the simulation.
+  --help           Show this message and exit.
+```
+
+
+```text
+Usage: simdb simulation push_local [OPTIONS] [REMOTE] SIM_ID
+
+  Push the simulation with the given SIM_ID (UUID or alias) to the REMOTE.
+
+Options:
+  --username TEXT  Username used to authenticate with the remote.
+  --password TEXT  Password used to authenticate with the remote.
+  --replaces TEXT  SIM_ID of simulation to deprecate and replace.
+  --add-watcher    Add the current user as a watcher of the simulation.
+  --timeout FLOAT  Maximum number of seconds to wait for ingestion to
+                   complete.  [default: 600.0]
+  --help           Show this message and exit.
+```
+
+
 ```text
 Usage: simdb simulation query [OPTIONS] [CONSTRAINTS]...
 
@@ -712,7 +777,8 @@ Usage: simdb simulation query [OPTIONS] [CONSTRAINTS]...
       NAME=[mod]VALUE
 
   Where `[mod]` is an optional query modifier. Available query modifiers are:
-      eq: - This checks for equality (this is the same behaviour as not providing any modifier).
+      eq: - This checks for equality (this is the same behaviour as not providing any
+            modifier).
       ne: - This checks for value that do not equal.
       in: - This searches inside the value instead of looking for exact matches.
       ni: - This searches inside the value for elements that do not match.
@@ -722,22 +788,26 @@ Usage: simdb simulation query [OPTIONS] [CONSTRAINTS]...
       le: - This checks for values less than or equal to the given quantity.
 
   For the following modifiers, VALUE should not be provided.     exist: - This
-  returns simulations where metadata with NAME exists, regardless of the
-  value.
+  returns simulations where metadata with NAME exists, regardless
+  of the value.
 
   Modifier examples:
       responsible_name=foo        performs exact match
       responsible_name=in:foo     matches all names containing foo
       pulse=gt:1000               matches all pulses > 1000
-      sequence=exist:             matches all simulations that have "sequence" metadata values
+      sequence=exist:             matches all simulations that have "sequence"
+                                  metadata values
 
-  Any string comparisons are done in a case-insensitive manner. If multiple constraints are provided then simulations
-  are returned that match all given constraints.
+  Any string comparisons are done in a case-insensitive manner. If multiple
+  constraints are provided then simulations are returned that match all given
+  constraints.
 
   Examples:
-      sim simulation query workflow.name=in:test       finds all simulations where workflow.name contains test
+      sim simulation query workflow.name=in:test       finds all simulations where
+                                                       workflow.name contains test
                                                        (case-insensitive)
-      sim simulation query pulse=gt:1000 run=0         finds all simulations where pulse is > 1000 and run = 0
+      sim simulation query pulse=gt:1000 run=0         finds all simulations where
+                                                       pulse is > 1000 and run = 0
 
 Options:
   -m, --meta-data TEXT  Additional meta-data field to print.

docs/cli.md.in

@@ -10,10 +10,6 @@
 
 {{ config }}
 
-## Database
-
-{{ database }}
-
 ## Manifest
 
 {{ manifest }}

docs/user_guide.md

@@ -297,6 +297,83 @@ simdb simulation push <SIM_ID>
 
 This will upload all the metadata associated with your simulation to the remote server as well as taking copies of all input and output data specified. For non-IMAS data the `file` URIs will be used to locate the files to transfer, whereas for `imas` URIs SimDB will discover which files need to be transferred based on the IMAS backend specified in the URI. The files are copied to the server using an HTTP data transfer.
 
+### Pushing Local Simulations (Optimized for Shared File Systems)
+
+If your local environment and the remote SimDB server share a common local file system (for example, on the ITER network where both you and the server can access same physical file paths directly), uploading large datasets over HTTP can be slow and redundant.
+
+In this scenario, you should use the `push_local` command:
+
+```bash
+simdb simulation push_local <SIM_ID>
+```
+
+Unlike the standard `push` command, `push_local` only sends the simulation metadata and the storage file paths to the server. The remote server then:
+1. Validates the simulation metadata against active schemas.
+2. Queues the file copy operation in the background using an asynchronous Celery task queue.
+3. Automatically completes the ingestion once background file copying finishes.
+
+The CLI command will block and print real-time updates while waiting for the background ingestion to complete:
+```text
+Waiting for ingestion to complete... queued -> copy_files -> completed
+Successfully pushed simulation <UUID>
+```
+
+#### Configuring Partitions (Shared Storage Mappings)
+
+To enable `push_local` to map and resolve files correctly between your local environment and the remote server, you must configure **partitions** in your `simdb.cfg` configuration file.
+
+Partitions define mappings between short, logical names (such as `data` or `work`) and their absolute paths on your local file system.
+
+##### 1. Defining Partitions on the Client
+Add a `[partition]` section to your `~/.config/simdb/simdb.cfg` file, specifying the directory paths for each partition:
+
+```ini
+[partition]
+data = /home/user/my_simdb_data
+work = /work/imas/shared
+sdcc = /
+```
+
+*Note on `sdcc` partition mapping:*
+In environments like the ITER network, files are often located under absolute paths like `/sdcc/projects/...`. Mapping the `sdcc` partition to the system root `/` ensures that any path beginning with `/sdcc` is correctly matched and converted to a partition-relative URI (e.g., `/sdcc/projects/my_run` becomes `sdcc:sdcc/projects/my_run`). When several partitions contain a file, the partition with the most specific (deepest) path wins, so such a catch-all mapping never shadows the other partitions.
+
+##### 2. How Partitions are Resolved
+When you run `push_local`:
+* SimDB scans the manifest's input/output files and checks if any path falls under one of your defined local partitions.
+* If a match is found (e.g., `/home/user/my_simdb_data/scenarios/run1.txt` is inside `/home/user/my_simdb_data`), SimDB converts the file URI into a partition-relative scheme: `data:scenarios/run1.txt`.
+* The remote server receives this logical URI. As long as the server also has the `data` partition configured (even if mounted at a different absolute path like `/mnt/shared/partition`), it resolves the URI to `/mnt/shared/partition/scenarios/run1.txt` and completes ingestion.
+
+This mapping mechanism allows clients and servers to share data over a network or cluster filesystem even if they mount it at different absolute paths.
+
+### Pushing Simulations over Resumable HTTP
+
+When your local environment and the remote server do **not** share a filesystem, but you still want the background ingestion workflow of `push_local` (rather than the single, non-resumable transfer of `push`), use the `push_http` command:
+
+```bash
+simdb simulation push_http <SIM_ID>
+```
+
+`push_http` uploads each file's bytes to the server over HTTP using the IETF "Resumable Uploads for HTTP" protocol (draft-ietf-httpbis-resumable-upload-11). The uploaded files are staged into a server-side partition named `http`, and from then on the flow is identical to `push_local`: SimDB sends the metadata, the server queues a background copy into its upload folder, and the CLI blocks while printing ingestion status updates.
+
+Because the protocol is resumable, an interrupted upload (lost connection, Ctrl-C) does not have to start over. Re-running `push_http` for the same simulation asks the server how many bytes it already received for each file and continues from that offset.
+
+The size of each upload chunk is governed by the server: it advertises a maximum append size via the `Upload-Limit` response header, and the client sizes its chunks to stay within that bound. The limit defaults to 8 MiB and can be tuned on the server with the `server.max_append_size` config option (for example, to fit within a reverse proxy's request body limit).
+
+Each chunk is integrity-checked using the RFC 9530 digest field. The client sends a `Content-Digest` (SHA-256) with every chunk so the server can verify that chunk before appending it. A digest mismatch is rejected with a `400` response and the offending bytes are not stored, so corruption in transit cannot be silently committed.
+
+#### Configuring the `http` Partition on the Server
+
+Unlike `push_local`, `push_http` needs **no client-side partition configuration** - the file paths are taken directly from the local simulation. The server, however, must define where uploaded bytes are staged by configuring a partition named `http` in its `simdb.cfg`:
+
+```ini
+[partition]
+http = /var/lib/simdb/http-staging
+```
+
+When a file is uploaded to `<sim_uuid>/<relative/path>`, the server writes it to `<http partition>/<sim_uuid>/<relative/path>` and references it with an `http://<sim_uuid>/<relative/path>` URI. The background ingestion task resolves that URI against the `http` partition, copies the file into the simulation's upload folder, and finally removes the staged copy from the `http` partition.
+
+Subfolder structure is handled exactly as it is for `push_local`: files keep their partition-relative layout, so multi-file IMAS datasets (HDF5, ASCII, MDSplus backends) stay contained within their own directory and are reconstructed correctly on the server, while standalone files (such as an IMAS netcdf `.nc`) are not given a spurious enclosing folder.
+
 ## Pulling simulations from a remote
 
 The mirror to pushing simulations is the `pull` command. This command will pull the simulation metadata from the SimDB remote to your local SimDB database and download the simulation data into a directory of your choosing. Once you have pulled a simulation it will appear in any local SimDB queries you perform. The command looks as follows:

pyproject.toml

@@ -41,7 +41,7 @@ dependencies = [
     "distro>=1.8.0",
     "email-validator>=1.1",
     "imas-python>=2.0.1",
-    "netCDF4>=1.5",
+    "netCDF4>=1.7.2",
     "numpy>=1.14",
     "pydantic>=2.10.6",
     "python-dateutil>=2.6",

src/simdb/checksum.py

@@ -1,14 +1,42 @@
 import hashlib
 from pathlib import Path
+from typing import Callable, Optional
 
 from simdb.imas.utils import SimDBUrl
 
+#: Algorithm used for all catalog checksums.
+CHECKSUM_ALGORITHM = "sha1"
+#: Buffer size for reading files while hashing. Larger reads mean far fewer
+#: syscalls on big files, which noticeably speeds up checksumming.
+READ_CHUNK_SIZE = 1024 * 1024
 
-def sha1_checksum(uri: SimDBUrl) -> str:
-    """Generate a SHA1 checksum from the given file.
+
+def hash_file(
+    path: Path,
+    algorithm: str = CHECKSUM_ALGORITHM,
+    progress: Optional[Callable[[int], None]] = None,
+) -> str:
+    """Return the hex digest of ``path`` computed with ``algorithm``.
+
+    @param progress: optional callback invoked with the number of bytes read for
+                     each block, suitable for advancing a progress bar.
+    """
+    digest = hashlib.new(algorithm)
+    with path.open("rb") as file:
+        for chunk in iter(lambda: file.read(READ_CHUNK_SIZE), b""):
+            digest.update(chunk)
+            if progress is not None:
+                progress(len(chunk))
+    return digest.hexdigest()
+
+
+def file_checksum(uri: SimDBUrl, algorithm: str = CHECKSUM_ALGORITHM) -> str:
+    """Generate a checksum for the file at ``uri``.
+
+    Checksums use :data:`CHECKSUM_ALGORITHM` (SHA-1).
 
     :param uri: the URI of the file to checksum
-    :return: a string containing the hex representation of the computed SHA1 checksum
+    :return: a string containing the hex representation of the computed checksum
     """
     if uri.scheme != "file":
         raise ValueError(f"invalid scheme for file checksum: {uri.scheme}")
@@ -21,8 +49,4 @@ def sha1_checksum(uri: SimDBUrl) -> str:
     if not path.is_file():
         raise ValueError("File appears to be a directory")
 
-    sha1 = hashlib.sha1()
-    with path.open("rb") as file:
-        for chunk in iter(lambda: file.read(4096), b""):
-            sha1.update(chunk)
-    return sha1.hexdigest()
+    return hash_file(path, algorithm)