fix(core): always omit license details from full-scan diff request (#CE-224 follow-on)

The full-scan diff request (fullscans.stream_diff) now always sets
include_license_details=false, decoupled from the --exclude-license-details
flag. This prevents the CE-224 truncation crash (Unterminated string / JSON
parse failure on large repos, reported by the tremendous org) from recurring
even when the flag is not passed.

Why this is safe (no output changes): the license fields the diff endpoint can
embed are never consumed off the diff. With --generate-license off, the only
consumer (the legal/FOSSA artifact builder) never runs. With --generate-license
on, get_license_text_via_purl re-fetches license data from the dedicated PURL
endpoint and overwrites whatever the diff embedded before anything reads it.
Either way the embedded payload was dead weight that only bloated the response.

--exclude-license-details still works but its scope is now narrower: it controls
only the dashboard report URL, not the internal diff payload. Help text updated.
Core.get_added_and_removed_packages(..., include_license_details=True) remains as
an explicit override seam (exercised in tests).

Minor bump to 2.4.0: outputs are provably unchanged, but this is a deliberate
default-behavior change (2.3.0 made the flag propagate; 2.4.0 makes the lean diff
the default), which warrants a minor bump per the project's semver policy.

Signed-off-by: lelia <2418071+lelia@users.noreply.github.com>

Author: lelia

CHANGELOG.md

@@ -1,5 +1,46 @@
 # Changelog
 
+## 2.4.0
+
+### Changed: license details are no longer requested on the full-scan diff
+
+The internal full-scan diff request (`fullscans.stream_diff`, used to compare
+alerts between two scans) now always sets `include_license_details=false`,
+regardless of the `--exclude-license-details` flag.
+
+**Why this is safe (no output changes):** the license fields the diff endpoint
+can embed were never actually consumed off the diff:
+
+- With `--generate-license` **off**, the only consumer of a package's
+  `licenseDetails`/`licenseAttrib` — the legal/FOSSA artifact builder — is never
+  invoked, so the embedded license data was parsed and immediately discarded.
+- With `--generate-license` **on**, the CLI re-fetches license data from the
+  dedicated PURL endpoint (`get_license_text_via_purl`) and **overwrites**
+  whatever the diff embedded before anything reads it.
+
+So in every code path the diff's license payload was dead weight. On large
+dependency trees it inflated the diff response past ~2.3 MB and truncated it
+mid-string, crashing `response.json()` with
+`Unterminated string starting at: ...` (CE-224, reported by the `tremendous`
+org). Dropping it keeps the diff lean with **zero change to any output
+artifact** (SBOM, legal/FOSSA attribution, report contents).
+
+**Why a minor bump (2.4.0), not a patch:** this is a deliberate default-behavior
+change. 2.3.0 fixed the `--exclude-license-details` flag so it correctly
+propagated to the diff; this release goes further and makes the lean diff the
+default so the crash cannot recur even when the flag is not passed. Per the
+project's semver policy a default-behavior change warrants a minor bump, even
+though outputs are provably unchanged.
+
+**Effect on `--exclude-license-details`:** the flag still works, but its scope is
+now narrower — it controls only the human-facing dashboard report URL
+(`?include_license_details=false`), not the internal diff payload. Its `--help`
+text was updated to reflect this.
+
+Override seam: `Core.get_added_and_removed_packages(..., include_license_details=True)`
+can still request embedded license details explicitly (used in tests); nothing
+in the CLI wires the user flag to it anymore.
+
 ## 2.3.1
 
 ### New: brotli-compressed `.socket.facts.json` upload

pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "socketsecurity"
-version = "2.3.1"
+version = "2.4.0"
 requires-python = ">= 3.11"
 license = {"file" = "LICENSE"}
 dependencies = [

socketsecurity/__init__.py

@@ -1,3 +1,3 @@
 __author__ = 'socket.dev'
-__version__ = '2.3.1'
+__version__ = '2.4.0'
 USER_AGENT = f'SocketPythonCLI/{__version__}'

socketsecurity/config.py

@@ -705,7 +705,12 @@ def create_argument_parser() -> argparse.ArgumentParser:
         "--exclude-license-details",
         dest="exclude_license_details",
         action="store_true",
-        help="Exclude license details from the diff report (boosts performance for large repos)"
+        help=(
+            "Exclude license details from the dashboard report URL. "
+            "As of 2.4.0 the internal diff request always omits license details "
+            "(they were unused there and bloated large-repo responses), so this "
+            "flag now only affects the report link, not diff performance."
+        )
     )
     output_group.add_argument(
         "--max-purl-batch-size",

socketsecurity/core/__init__.py

@@ -1070,14 +1070,35 @@ def get_added_and_removed_packages(
             self,
             head_full_scan_id: str,
             new_full_scan_id: str,
-            include_license_details: bool = True
+            include_license_details: bool = False
     ) -> Tuple[Dict[str, Package], Dict[str, Package], Dict[str, Package]]:
         """
         Get packages that were added and removed between scans.
 
         Args:
             head_full_scan_id: Previous scan (maybe None if first scan)
             new_full_scan_id: New scan just created
+            include_license_details: Whether to ask the diff endpoint to embed
+                per-package license attribution/details in the response.
+
+                Defaults to ``False`` on purpose. The diff endpoint exists to
+                compare alerts between two scans; the license fields it can embed
+                are never consumed off the diff:
+                  * When ``--generate-license`` is OFF, the only consumer of
+                    ``Package.licenseDetails``/``licenseAttrib`` (the legal/FOSSA
+                    artifact builder) is never invoked, so the embedded license
+                    data is parsed and then dropped on the floor.
+                  * When ``--generate-license`` is ON, ``get_license_text_via_purl``
+                    re-fetches license data from the dedicated PURL endpoint and
+                    OVERWRITES whatever the diff embedded, before anything reads it.
+                Either way the embedded license payload is dead weight, and on
+                large dependency trees it inflated the diff response past ~2.3MB
+                and truncated it mid-string, crashing ``response.json()``
+                (CE-224, customer: Tremendous). Defaulting to ``False`` keeps the
+                diff lean with zero change to any output artifact. The parameter
+                is retained as an explicit override seam, not wired to the
+                ``--exclude-license-details`` user flag (which still governs the
+                human-facing dashboard report URL).
 
         Returns:
             Tuple of (added_packages, removed_packages) dictionaries
@@ -1299,15 +1320,23 @@ def create_new_diff(
                 except OSError as e:
                     log.warning(f"Failed to clean up temporary file {temp_file}: {e}")
 
-        # Handle diff generation - now we always have both scans
+        # Handle diff generation - now we always have both scans.
+        #
+        # Note: we intentionally do NOT forward params.include_license_details
+        # (the --exclude-license-details user flag) into the diff request. The
+        # diff path never consumes embedded license data (see
+        # get_added_and_removed_packages docstring), so requesting it only bloats
+        # the response and risks the CE-224 truncation crash on large repos. The
+        # user flag still controls the dashboard report URL below; it just no
+        # longer gates this internal diff payload.
         (
             added_packages,
             removed_packages,
             packages
         ) = self.get_added_and_removed_packages(
             head_full_scan_id,
             new_full_scan.id,
-            include_license_details=getattr(params, "include_license_details", True)
+            include_license_details=False
         )
 
         # Separate unchanged packages from added/removed for --strict-blocking support

tests/core/test_sdk_methods.py

@@ -95,13 +95,17 @@ def test_get_added_and_removed_packages(core):
     # Get two different scans to compare
     added, removed, all_packages = core.get_added_and_removed_packages("head", "new")
 
-    # Verify SDK was called correctly
+    # Verify SDK was called correctly.
+    # include_license_details defaults to "false": the diff path never consumes
+    # embedded license data (license artifacts come from the PURL endpoint), so
+    # requesting it only bloats the response and risks the CE-224 truncation
+    # crash on large repos.
     core.sdk.fullscans.stream_diff.assert_called_once_with(
         core.config.org_slug,
         "head",
         "new",
         use_types=True,
-        include_license_details="true",
+        include_license_details="false",
     )
 
     # Verify the results
@@ -116,6 +120,18 @@ def test_get_added_and_removed_packages(core):
     assert "dp2_t1" in removed  # Verify transitive dependencies are also tracked
     assert "pypi/direct_package_1@1.6.0" in all_packages  # Unchanged package is in full package map
 
+def test_get_added_and_removed_packages_license_override(core):
+    """The include_license_details override seam still works when explicitly requested."""
+    core.get_added_and_removed_packages("head", "new", include_license_details=True)
+
+    core.sdk.fullscans.stream_diff.assert_called_once_with(
+        core.config.org_slug,
+        "head",
+        "new",
+        use_types=True,
+        include_license_details="true",
+    )
+
 def test_empty_alerts_preserved(core):
     """Test that empty alerts arrays stay as empty arrays and don't become None"""
     # Get the scan that contains dp2 (which has empty alerts array)