[DE-8304] Deprecate synchronous upload capability (#468)

* Deprecate synchronous upload capability

* Disallow mixed file upload types in one call

* Address greptile

* Update tests that still use sync upload

* Remove sync append in deduplication test

* Update add_items_from_dir to be async only

* Fix test_create_update_dataset_from_dir

* Remove deprecated asynchronous=True flag from test files

* Update CLAUDE.md

Author: edwinpav

CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to the [Nucleus Python Client](https://github.com/scaleapi/n
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.19.0](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.19.0) - 2026-07-07
+
+### Changed
+- **Breaking:** `dataset.append()` now always uses the async pipeline and returns an `AsyncJob`. The `asynchronous` and `batch_size` parameters are deprecated and ignored. All uploads (local and remote) go through the async Step Function pipeline, which handles phash computation, image optimization, and NLS search indexing.
+- `dataset.add_items_from_dir()` now returns the `AsyncJob` for the upload (or `None` when no items are found) instead of blocking. Call `job.sleep_until_complete()` to wait until items are queryable and to surface upload errors.
+
+### Removed
+- Synchronous upload paths for images and videos. All uploads now use the async pipeline. Use `job.sleep_until_complete()` to block until processing finishes.
+- `UploadResponse` class — `append()` now returns `AsyncJob`.
+- `construct_append_payload()` and `construct_append_scenes_payload()` functions.
+- `check_all_paths_remote()` function.
+- The already deprecated `dataset.append_scenes()` method — use `dataset.append()` instead.
+- Synchronous branches from `_append_scenes()` and `_append_video_scenes()`.
+
 ## [0.18.8](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.18.8) - 2026-06-17
 
 ### Fixed

CLAUDE.md

@@ -35,6 +35,7 @@ Pure refactors / doc-only PRs (#456) sometimes skip the version bump. When in do
 
 - `nucleus/__init__.py` — `NucleusClient`, top-level operations.
 - `nucleus/dataset.py` — `Dataset` class. Most user-facing methods live here (item upload/fetch, generators, queries, slices, autotags, exports). Generators page through the backend via `nucleus/utils.py:paginate_generator`.
+- **Uploads are async-only.** `Dataset.append()` always returns an `AsyncJob` (the `asynchronous`/`batch_size` params are deprecated and ignored). Items aren't queryable until the job finishes — callers and test fixtures must `job.sleep_until_complete()` before reading `dataset.items`, creating slices, or uploading annotations/predictions keyed by `reference_id`. `add_items_from_dir()` returns the job; `create_dataset_from_dir()` awaits it internally.
 - `nucleus/dataset_item.py` — `DatasetItem` dataclass. **`DatasetItem.from_json` is the single deserialization entry point** for items coming back from the API — every SDK method that returns a `DatasetItem` (generators, queries, `iloc`/`refloc`/`loc`, the `items` property) routes through it. To expose a new server-side field on items, add it to the dataclass + `from_json` and you're done on the SDK side.
 - `nucleus/utils.py` — `convert_export_payload` and `format_dataset_item_response` are the shared shapers used by the export and single-item endpoints. They wrap raw JSON into typed objects via the respective `from_json` classmethods.
 - `nucleus/constants.py` — All API payload keys are constants here. When adding a new field, add a `*_KEY` constant first and reference it from `from_json` / `to_payload` rather than inlining the string.

conftest.py

@@ -46,7 +46,8 @@ def CLIENT():
 @pytest.fixture()
 def dataset(CLIENT: "NucleusClient"):
     test_dataset = CLIENT.create_dataset(TEST_DATASET_NAME, is_scene=False)
-    test_dataset.append(TEST_DATASET_ITEMS)
+    job = test_dataset.append(TEST_DATASET_ITEMS)
+    job.sleep_until_complete()
     yield test_dataset
 
 

nucleus/__init__.py

@@ -171,7 +171,6 @@
 from .model_run import ModelRun
 from .payload_constructor import (
     construct_annotation_payload,
-    construct_append_payload,
     construct_box_predictions_payload,
     construct_model_creation_payload,
     construct_segmentation_payload,
@@ -190,7 +189,6 @@
 from .retry_strategy import RetryStrategy
 from .scene import Frame, LidarScene, VideoScene
 from .slice import Slice
-from .upload_response import UploadResponse
 from .utils import create_items_from_folder_crawl
 from .validate import Validate
 
@@ -605,19 +603,6 @@ def delete_dataset_item(self, dataset_id: str, reference_id) -> dict:
         dataset = self.get_dataset(dataset_id)
         return dataset.delete_item(reference_id)
 
-    @deprecated("Use Dataset.append instead.")
-    def populate_dataset(
-        self,
-        dataset_id: str,
-        dataset_items: List[DatasetItem],
-        batch_size: int = 20,
-        update: bool = False,
-    ):
-        dataset = self.get_dataset(dataset_id)
-        return dataset.append(
-            dataset_items, batch_size=batch_size, update=update
-        )
-
     @deprecated(msg="Use Dataset.ingest_tasks instead")
     def ingest_tasks(self, dataset_id: str, payload: dict):
         dataset = self.get_dataset(dataset_id)
@@ -1407,10 +1392,12 @@ def create_dataset_from_dir(
         dataset = self.create_dataset(
             name=dataset_name, use_privacy_mode=use_privacy_mode
         )
-        dataset.add_items_from_dir(
+        job = dataset.add_items_from_dir(
             existing_dirname=existing_dirname,
             privacy_mode_proxy=privacy_mode_proxy,
             allowed_file_types=allowed_file_types,
             skip_size_warning=skip_size_warning,
         )
+        if job is not None:
+            job.sleep_until_complete()
         return dataset

nucleus/async_job.py

@@ -51,8 +51,8 @@ class AsyncJob:
         client = nucleus.NucleusClient(YOUR_SCALE_API_KEY)
         dataset = client.get_dataset("ds_bwkezj6g5c4g05gqp1eg")
 
-        # When kicking off an asynchronous job, store the return value as a variable
-        job = dataset.append(items=YOUR_DATASET_ITEMS, asynchronous=True)
+        # dataset.append() always returns an AsyncJob
+        job = dataset.append(items=YOUR_DATASET_ITEMS)
 
         # Poll for status or errors
         print(job.status())