Enhance service bus handling of invalid JSON in receive_message function

[James Chapman (JC-wk)]

resolves #4976

What is being addressed

Service bus will now skip malformed payloads instead of crashing

How is this addressed

Summary of Changes

1. Bug Fix in runner.py:
   • Caught json.JSONDecodeError when deserializing the Service Bus message payload.
   • Sent the malformed message to the dead-letter queue via await receiver.dead_letter_message(msg, reason="InvalidJSON") to clear it from the queue and aid in troubleshooting.
   • Added a continue statement inside the except block to prevent processing of empty/malformed message structures, avoiding downstream TypeError crashes.
2. Added Unit Tests in test_runner.py:
   • Added a new unit test test_receive_message_bad_json to simulate receiving a malformed/non-JSON payload, asserting that:
   • The message is successfully sent to the dead-letter queue.
   • The message is not processed or marked complete (i.e. skipped).

• Updated CHANGELOG.md
• Increment version

[Copilot]

Pull request overview

Improves the resource processor’s Service Bus session receiver loop to tolerate malformed/non-JSON messages by dead-lettering them and continuing processing, instead of crashing the runner.

Changes:

• Dead-letter malformed Service Bus messages on json.JSONDecodeError and skip further processing of that message.
• Add a unit test covering the malformed JSON path in receive_message.
• Bump resource_processor version and add an Unreleased changelog entry.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
resource_processor/vmss_porter/runner.py	Dead-letters invalid JSON messages and continues the receive loop.
resource_processor/tests_rp/test_runner.py	Adds a unit test to verify invalid JSON messages are dead-lettered and not completed.
resource_processor/_version.py	Patch version bump.
CHANGELOG.md	Adds an Unreleased BUG FIXES entry for the change.

[github-actions]

Unit Test Results

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit 978e793.

♻️ This comment has been updated with latest results.

[Jack Morris (rudolphjacksonm)]

/test 59a309a

[github-actions]

🤖 pr-bot 🤖

🏃 Running tests: https://github.com/microsoft/AzureTRE/actions/runs/27306698867 (with refid a20534b2)

(in response to this comment from Jack Morris (@rudolphjacksonm))

[Jack Morris (rudolphjacksonm)]

/test e7218a4

[github-actions]

🤖 pr-bot 🤖

🏃 Running tests: https://github.com/microsoft/AzureTRE/actions/runs/28122108599 (with refid a20534b2)

(in response to this comment from Jack Morris (@rudolphjacksonm))

[Marcus Robinson (marrobi)]

James Chapman (@JC-wk) Opus 4.8 review, let me kniw if agree:

A couple of things before merge:

2. (Optional) Guard the dead-letter call. If a message is already in a state where dead_letter_message can fail (e.g. lock lost / already settled), the raised exception would bubble to the outer except Exception and we'd re-enter the same situation. Consider wrapping the dead-letter in its own try/except so a settle failure is logged rather than aborting the batch, e.g.:

   except json.JSONDecodeError as e:
       logger.error(f"Received bad service bus resource request message: {e}")
       try:
           await receiver.dead_letter_message(msg, reason="InvalidJSON", error_description=str(e))
       except Exception:
           logger.exception("Failed to dead-letter malformed message")
       continue

3. (Nit) Test brittleness. test_receive_message_bad_json asserts the exact exception string "Expecting value: line 1 column 1 (char 0)". That text comes from the stdlib and could change across Python versions. Asserting on reason="InvalidJSON" (and maybe that error_description is non-empty) would be more robust, but this is minor.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (3)

CHANGELOG.md:20

• The changelog entry references PR #4932, but this PR is described as resolving issue #4976. The link/number should match the actual issue/PR for traceability.

* Fix to enhance service bus handling of invalid JSON in receive_message function ([#4932](https://github.com/microsoft/AzureTRE/pull/4932))

resource_processor/vmss_porter/runner.py:76

• json.loads can succeed for payloads that are still malformed for this code path (e.g., JSON string/array, or missing required keys). In that case message["id"]/etc will raise TypeError/KeyError, so the receiver can still crash despite the new JSONDecodeError handling. Consider validating the decoded payload is an object with required fields and dead-lettering it similarly before starting the trace span.

                        try:
                            message = json.loads(str(msg))
                        except (json.JSONDecodeError) as e:
                            logger.error(f"Received bad service bus resource request message: {e}")
                            await receiver.dead_letter_message(msg, reason="InvalidJSON", error_description=str(e))

resource_processor/tests_rp/test_runner.py:158

• The test asserts an exact JSONDecodeError message string in error_description. That message can vary across Python versions/implementations, making the unit test brittle. Prefer asserting that dead-lettering happened and that error_description contains a stable substring (or is non-empty) rather than matching the full text.

    mock_receiver.dead_letter_message.assert_awaited_once_with(
        "invalid_json_string",
        reason="InvalidJSON",
        error_description="Expecting value: line 1 column 1 (char 0)"
    )

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

resource_processor/vmss_porter/runner.py:77

• This handles only JSON syntax errors. Payloads such as null, [], or "text" deserialize successfully, but the subsequent message["id"] access raises TypeError; the outer catch then reconnects without settling the poison message, so the reported failure remains possible. Validate that the decoded payload is an object with the required resource-request fields and dead-letter invalid structures as well (with a corresponding test).

                            continue

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

resource_processor/vmss_porter/runner.py:82

• This presence-only check still lets malformed requests reach processing. For example, an object containing these four keys but omitting name, version, or parameters passes here, then build_porter_command raises at helpers/commands.py:91 or :132 after an in-progress status has already been sent, so the message remains unsettled and is retried. Invalid field types (such as parameters: null or a list-valued action) have the same problem. Validate the complete resource-request schema, including all consumed fields and their types, before invoking any processing, and dead-letter schema validation failures.

                            missing_fields = REQUIRED_RESOURCE_REQUEST_FIELDS - message.keys()
                            if missing_fields:
                                raise ValueError(f"Resource request message is missing fields: {sorted(missing_fields)}")

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

resource_processor/vmss_porter/runner.py:115

• This handler also catches schema-validation ValueErrors, so valid JSON with missing or mistyped fields is dead-lettered with the misleading reason InvalidJSON. This makes DLQ diagnostics inaccurate. Split parse and schema failures into distinct reasons, or use a broader reason such as InvalidResourceRequest, and update the test assertions accordingly.

                                await receiver.dead_letter_message(msg, reason="InvalidJSON", error_description=str(e))