Skip to content

Make smoke tests capability probes; fix 5 that crash on engine divergence - #718

Open
eerxuan wants to merge 1 commit into
documentdb:mainfrom
eerxuan:smoke-tests-capability-probe
Open

Make smoke tests capability probes; fix 5 that crash on engine divergence#718
eerxuan wants to merge 1 commit into
documentdb:mainfrom
eerxuan:smoke-tests-capability-probe

Conversation

@eerxuan

@eerxuan eerxuan commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Smoke tests are meant to answer "does the engine support this feature?" — one minimal probe per feature. Several were written as full round-trip assertions, so they failed on cosmetic divergences (e.g. ok returned as int32 vs the double the expected value used) even though the feature works, or crashed with unhelpful TypeError/IndexError when a setup step behaved differently.

Framework (framework/assertions.py)

  • assertCommandSupported(result) — passes if the command returned a result rather than erroring; no value/BSON-type comparison. The capability probe.
  • assertChangeStreamEvent(result) — command succeeded and cursor.nextBatch is non-empty (an event was emitted); no comparison of event contents.

Fixes (5 that errored on non-matching behaviour)

  • getMore / killCursors: extract the cursor id defensively so a failed setup find reports cleanly through the single assertion instead of raising 'OperationFailure' object is not subscriptable.
  • changeStream createIndexes / dropIndexes / modify: extract the cursor id defensively and assert an event was emitted, instead of indexing nextBatch[0] into an IndexError.

Reduced to capability probes (25)

index create/properties/types, minN/maxN accumulator+expression+window, $rank, planCacheListFilters, and the collStats/connectionStatus/dataSize/dbStats/hostInfo/logout diagnostics. These now assert only that the command is supported; exact response shape stays the job of each feature's detailed tests.

Each test keeps exactly one assertion (test-format validator compliant).

Not changed here

The assert-on-error smoke tests (commitTransaction, abortTransaction, compactStructuredEncryptionData, setIndexCommitQuorum, authenticate). Asserting a specific error is a poor support signal and can't be special-cased per engine; converting them to happy-case probes is a follow-up.

Testing

Verified against a live engine: 24 non-replica-set fixed smoke tests pass; the 3 $setWindowFields ones pass with the window-fields feature flag on; the 3 changeStream ones are replica_set-gated.

…ence

Smoke tests are meant to answer "does the engine support this feature?" — one
minimal probe per feature. Several were written as full round-trip assertions,
so they failed on cosmetic divergences (e.g. `ok` returned as int32 vs the
double the expected value used) even though the feature works, or crashed with
unhelpful TypeError/IndexError when a setup step behaved differently.

framework/assertions.py:
  - assertCommandSupported(result): passes if the command returned a result
    rather than erroring; no value/BSON-type comparison. The capability probe.
  - assertChangeStreamEvent(result): command succeeded and cursor.nextBatch is
    non-empty (an event was emitted); no comparison of event contents.

Fixes (5 that errored on non-matching behaviour):
  - getMore / killCursors: extract the cursor id defensively so a failed setup
    find reports cleanly through the single assertion instead of raising
    "'OperationFailure' object is not subscriptable".
  - changeStream createIndexes / dropIndexes / modify: extract the cursor id
    defensively and assert an event was emitted, instead of indexing
    nextBatch[0] into an IndexError.

Reduced to capability probes (25): index create/properties/types, minN/maxN
accumulator+expression+window, $rank, planCacheListFilters, and the collStats/
connectionStatus/dataSize/dbStats/hostInfo/logout diagnostics. These now assert
only that the command is supported; exact response shape stays the job of each
feature's detailed tests.

Each test keeps exactly one assertion (test-format validator compliant).

Not changed here: the assert-on-error smoke tests (commitTransaction,
abortTransaction, compactStructuredEncryptionData, setIndexCommitQuorum,
authenticate). Asserting a specific error is a poor support signal and can't be
special-cased per engine; converting them to happy-case probes is a follow-up.

Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
@eerxuan
eerxuan requested a review from a team as a code owner August 12, 2026 21:36
@documentdb-triage-tool documentdb-triage-tool Bot added compatibility test Compatibility test related enhancement New feature or request labels Aug 12, 2026
@documentdb-triage-tool

Copy link
Copy Markdown

🤖 Auto-triaged by documentdb-triage-tool.

Applied: compatibility test, enhancement
Project fields suggested: Component test-coverage · Priority P0 · Effort XL · Status Needs Review
Confidence: 0.82 (mixed)

Reasoning

component from path globs (test-coverage, test-framework); P0 keyword "crash" in title; effort from diff stats (129+175 LOC, 31 files); LLM: Adds new assertion helpers and refactors ~30 smoke tests to be defensive capability probes, touching framework/assertions.py and multiple test files across one component.

If a label is wrong, remove it manually and ping @patty-chow so the rules can be tuned. The bot will not re-label items that already have component labels.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

compatibility test Compatibility test related enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant