Remove $documents dependency from expression test helpers - #715
Open
eerxuan wants to merge 5 commits into
Open
Conversation
The expression compatibility tests use a collectionless
`{aggregate: 1}` + `$documents: [{}]` pipeline purely as a scaffold
to feed a single empty document into $project. They do not test
$documents itself, yet they inherit a hard dependency on $documents
support from the shared helpers.
Swap the scaffold for `collection.insert_one({})` plus an aggregate
over the named collection. This is logically identical: both feed
exactly one empty document to $project, so literal expressions and
field references (which resolve to missing against an empty doc)
behave the same. The `collection` fixture is function-scoped, so each
test still runs against a fresh single-document collection.
Updates the two shared helpers (execute_expression, execute_project)
that ~205 files inherit, plus two inline call sites in
test_expressions_combination_variables.py. The _with_insert sibling
helpers are unchanged.
Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
|
🤖 Auto-triaged by documentdb-triage-tool. Applied: Reasoningcomponent from path globs (test-coverage); effort from diff stats (22+11 LOC, 2 files); LLM: Removes a hard If a label is wrong, remove it manually and ping |
The $documents-removal commit made execute_expression/execute_project
insert a document and aggregate over the whole collection. Two newly
synced system-variable tests relied on the old $documents:[{}] contract
and broke:
- test_root_empty_document: needs a truly field-less input so $$ROOT is
{}, but an inserted doc always carries an auto _id. It now shapes its
own pipeline ($replaceWith:{$literal:{}}) instead of the shared helper.
- test_now_identical_across_getmore_batches: pre-loads 300 docs, so the
whole-collection helper emitted 300 rows. It now uses an inline
pipeline with $limit:1 to collapse to the single expected row.
The shared helpers stay on plain insert_one({}) so the ~359 literal
expression call sites gain no $replaceWith/$limit dependency.
Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
The replica-set test job was hitting GitHub's 6h max execution limit. Root cause: the suite creates and drops tens of thousands of collections per run; WiredTiger keeps an open fd per table it has touched (closing idle ones only very lazily) and, on the replica set, dropped collections become drop-pending idents whose files linger until majority-committed and swept. So the descriptor count tracks total collections created over the run, not the number live at once, and mid-run mongod runs out of descriptors. That surfaces as a WiredTiger WT_PANIC (EMFILE -> fassert) and crashes the server; every remaining test then blocks on connection timeouts and the job crawls for hours. Standalone is unaffected because it reclaims dropped tables immediately and never approaches the ceiling. Two changes: 1. Raise ulimits.nofile to 1048576 (the limit MongoDB's own packaging ships) on both mongod targets. The replica set peaks near 60k open files partway through the suite; 64000 was not enough headroom -- it still crashed at ~31k collections -- so this is set well above the peak. 2. Add a target-death watchdog and a per-test --timeout to the test job so that if a target ever does die mid-run, the job fails in about a minute instead of timing out every remaining test against a dead target for hours. The watchdog sanity-checks its probe first and steps aside if it cannot read a healthy baseline, so it cannot false-abort a healthy run. Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
eerxuan
force-pushed
the
remove-documents-dependency-upstream
branch
from
August 12, 2026 23:22
bec907b to
3681e1b
Compare
These tests write documents to advance the logical clock (the behavior
under test), then assert a variable/literal expression via execute_expression.
The old helper ran collectionless ($documents: [{}]) so those writes never
affected it. The new helper aggregates over collection.name, so every document
the test wrote becomes an output row -- the assertion expected [{result: X}]
but got one identical row per document.
Each affected assertion evaluates only $$CLUSTER_TIME and Python-computed
literals (never a document field), so the value is identical on any row. Add
{$limit: 1} to collapse the aggregation to the single-row evaluation the
$documents scaffold used to provide -- preserving exactly what each test
verifies without reintroducing the $documents dependency this PR removes.
This is the same shaping the PR already applied to
test_now_identical_across_getmore_batches. The 5 tests whose collection stays
empty are unaffected and still use the helper directly.
Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
Watchdog (pr-tests.yml): require two consecutive low container-count
readings before aborting. A single reading of the resource-heavy suite's
`docker compose ps` can transiently error and read as 0, which would
false-abort a healthy run; a real crash stays low across polls, so this
rules out the false abort while adding at most one poll interval to a
genuine fast-fail.
Helper docstrings (utils.py): the insert+aggregate form leaves the input
row with an auto-generated `_id` rather than the field-less row
`$documents: [{}]` produced, so note that an expression reading `$_id`,
`$$ROOT`, or `$$CURRENT` sees that id and diverges (references to any other
missing field are still identical). Documentation only; no behavior change.
Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Remove the dependency on the
$documentsaggregation stage from the sharedexpression test helpers (
execute_expression,execute_project) and twoinline call sites in
test_expressions_combination_variables.py.Why
These tests use a collectionless
{aggregate: 1}pipeline with{$documents: [{}]}purely as a scaffold to feed a single empty documentinto
$project. They do not test$documentsitself — yet through theshared helpers, ~205 expression test files inherit a hard dependency on
$documentssupport. Any engine that does not implement$documentscannotrun these tests at all, even though the behavior under test has nothing to do
with that stage.
Removing this dependency lets the expression compatibility suite run against
engines that don't yet support the
$documentsfeature.How
Swap the scaffold for
collection.insert_one({})plus an aggregate over thenamed collection:
This is logically identical: both feed exactly one empty document into
$project, so literal expressions and field references (which resolve tomissing against an empty document) behave the same. The
collectionfixture isfunction-scoped, so each test still runs against a fresh single-document
collection.
Scope
expressions/utils/utils.py— the two shared helpers (execute_expression,execute_project) inherited by ~205 files.test_expressions_combination_variables.py— two inline$documentssites(
test_let_two_lets_same_projection,test_let_error_cross_let_variable_ref)._with_insertsibling helpers are unchanged (they already insert apopulated document).