Fix pathologically slow assertion diffs for large inputs (#8998)#14543
Open
kirilklein wants to merge 2 commits into
Open
Fix pathologically slow assertion diffs for large inputs (#8998)#14543kirilklein wants to merge 2 commits into
kirilklein wants to merge 2 commits into
Conversation
…8998) Comparing very large strings, lists, or dataclasses in an ``assert`` could hang for a long time (sometimes minutes) while pytest built the failure diff. The cost comes from ``difflib.ndiff``: its character-level "fancy replace" step is quadratic in the size of the differing region, and the underlying ``SequenceMatcher`` is quadratic in the number of lines (a large nested structure can pretty-print to hundreds of thousands of lines). Add a deterministic size heuristic (no wall-clock timeouts, per the maintainer discussion in the issue): when the input is too large for ``ndiff`` to be fast, fall back to a coarser line-level ``unified_diff``, capped to a bounded number of lines so it always completes in milliseconds, and note this in the output. Smaller comparisons keep the existing detailed ``ndiff`` output unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
for more information, see https://pre-commit.ci
Member
|
We have a flying MR to use generator in assert repr that could help with this when we don't have to show the actual output. (#14523) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #8998.
Problem
Comparing very large strings, lists, or dataclasses inside an
assertcan hang for a long time (sometimes minutes) while pytest builds the failure diff.Profiling the reproductions from the issue confirms the root cause is
difflib.ndiff:SequenceMatcheris quadratic in the number of lines — a large nested structure pretty-prints to a huge number of lines (the dataclass example in the issuepformats to ~418,000 lines).Approach
Following the maintainer discussion in the issue, this uses a deterministic size heuristic rather than wall-clock timeouts (which are non-deterministic and can't reliably interrupt
difflib).A new helper module
_pytest/assertion/_diff.pyprovides:ndiff_too_slow(left_lines, right_lines)—Truewhen the combined input exceeds a character budget or a line-count budget, the two dimensions that makendiffslow.fast_unified_diff(...)— a coarse but fast line-leveldifflib.unified_diff, capped to a bounded number of lines so it always completes in milliseconds. It notes in the output that a faster diff is being shown (and how many lines were hidden).Both pathological call sites fall back to it when needed:
compare_text._diff_text(string comparisons)_compare_sequence._compare_eq_iterable(list / dataclass / iterable comparisons)Comparisons below the cutoffs keep the existing detailed
ndiffoutput unchanged.Results
On the reproductions from the issue (dataclass with large lists + two large random strings), with
-v:find_longest_match)Tests
Added regression tests in
testing/test_assertion.py: unit tests for thendiff_too_slowheuristic, and integration tests that large string / many-line / large-iterable comparisons fall back to the fast diff (nondiff?guide lines), still show which lines differ, and emit the line-cap notice. Thresholds were chosen from benchmarking.🤖 Generated with Claude Code