Skip to content

Treat ^ and v as operators even without surrounding whitespace - #639

Merged
fgmacedo merged 4 commits into
fgmacedo:developfrom
chuenchen309:fix/spec-parser-unspaced-classical-operators
Aug 1, 2026
Merged

Treat ^ and v as operators even without surrounding whitespace#639
fgmacedo merged 4 commits into
fgmacedo:developfrom
chuenchen309:fix/spec-parser-unspaced-classical-operators

Conversation

@chuenchen309

@chuenchen309 chuenchen309 commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

parse_boolean_expr has a fast-path that returns the whole expression as a
single variable name when it looks operator-free. It only checked for !, not
the other two documented classical operators ^ (and) / v (or), so an
unspaced guard skipped operator replacement and was resolved as a variable:

cond="a ^ b"   # parsed as (a and b)
cond="a^b"     # resolved as a variable literally named "a^b"

a^b then evaluates wrong (or raises InvalidDefinition: Did not found name 'a^b' at class-definition time). Since !a already worked without spaces, the
inconsistency is surprising — users copy operators from the guards docs, where
^/v are listed with no whitespace requirement.

Gate the fast-path on the module's existing pattern (which already matches
!, ^, and the word-bounded v) instead of the hand-rolled "!" not in expr
check. Genuine single-name variables like avc (no word-boundary v) still
take the fast-path. Added a regression test asserting the unspaced forms match
their spaced equivalents.


This PR was authored by an AI coding agent (Claude Code) running on this account:
the AI found the bug, ran the repro, wrote the test, and wrote this description.
The human account holder reviews every change and is accountable for it. The
verification is real and re-runnable from the diff. If this isn't the kind of
contribution you want, say so and I'll close it.


Update (maintainer follow-up, a0a19a5): the fast-path is now gated on
expr.isidentifier() instead of the module pattern. That keeps the reported
a^b / (a)v(b) fix while preserving unspaced != and a guard literally named
v, and it also fixes the sibling bug where unspaced comparisons (x==1,
x>=1) were swallowed as variable names. Listeners.build now catches
ValueError alongside SyntaxError so expressions like cond="user.age" keep
raising the friendly InvalidDefinition. Regression tests and a release note in
docs/releases/3.2.1.md were added.

parse_boolean_expr has a fast-path that returns the whole string as a single
variable name when it contains no operator. It only checked for "!", not the
other two classical operators "^" (and) and "v" (or), so an unspaced guard like
cond="a^b" skipped operator replacement and was looked up as a variable named
"a^b" -- silently evaluating wrong (or raising InvalidDefinition on a
StateChart). "!a" already worked unspaced, so this was an inconsistency.

Gate the fast-path on the module's existing operator `pattern` (which already
matches !, ^ and the word-bounded v) instead of a hand-rolled "!" check.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>
@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (471fe17) to head (7019e3e).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files
@@            Coverage Diff            @@
##           develop      #639   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           52        52           
  Lines         5505      5507    +2     
  Branches       869       869           
=========================================
+ Hits          5505      5507    +2     
Flag Coverage Δ
unittests 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@fgmacedo fgmacedo left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @chuenchen309, thanks a lot for this contribution, and congratulations on your first PR to the project! 🎉

Great catch: the fast-path really does mishandle unspaced ^/v, and your diagnosis is spot on. While reviewing I found that gating on pattern introduces two behavior regressions that our test suite doesn't cover (which is why CI stayed green). Details and a suggested alternative inline.

Could you also add a short entry about the fix to the open release notes under docs/releases/? We document every user-facing bugfix there.

Comment thread statemachine/spec_parser.py Outdated
Comment thread tests/test_spec_parser.py Outdated
]:
got = parse_boolean_expr(unspaced, variable_hook, operator_mapping)()
want = parse_boolean_expr(spaced, variable_hook, operator_mapping)()
assert got is want, unspaced

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice test, thanks for including it. Since CI stayed green despite the regressions mentioned in spec_parser.py, could you extend the coverage here to pin the behaviors that must keep working? These two cases would have caught them:

def test_unspaced_not_equal_is_a_comparison():
    # "!" is an operator, but "!=" is not a negation: it must keep parsing
    # as a comparison even without surrounding whitespace.
    got = parse_boolean_expr("frodo_age!=51", variable_hook, operator_mapping)()
    want = parse_boolean_expr("frodo_age != 51", variable_hook, operator_mapping)()
    assert got is want is True


def test_bare_v_is_a_variable_name():
    # "v" is only an operator between operands; a lone "v" is a plain variable.
    expr = parse_boolean_expr("v", variable_hook, operator_mapping)
    assert expr.__name__ == "v"

Note the !=51 (not !=50): a misparse resolves the whole string as an unknown variable, which this variable_hook defaults to False, so a !=50 case would pass for the wrong reason.

The fast-path in `parse_boolean_expr` returned the whole expression as a
single variable name whenever it found no `!`, no space and no `In(`. Any
other operator written without surrounding whitespace was swallowed into a
variable name, so `cond="a^b"` or `cond="items>0"` were resolved as variables
literally named `a^b` / `items>0`.

Gating on the module's `pattern` (as originally proposed) fixes `^`/`v` but
regresses two working cases: unspaced `!=` (skipped by the `\!(?!=)`
lookahead) stops parsing as a comparison, and a guard literally named `v`
starts being replaced by ` or `.

The invariant the fast-path wants is "the whole string is a single variable
name", which `str.isidentifier()` checks directly. This keeps `!=` and a bare
`v` working, makes the `" "` / `"In("` checks redundant, and also fixes the
sibling bug where unspaced comparisons like `x==1` were swallowed too.

Expressions with a structure that is invalid in a boolean context (e.g.
`cond="user.age"`) now reach the AST allowlist and raise `ValueError`, so
`Listeners.build` catches it alongside `SyntaxError` to keep reporting the
friendly `InvalidDefinition`.

Signed-off-by: Fernando Macedo <fgmacedo@gmail.com>
@fgmacedo

fgmacedo commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Hi @chuenchen309, thanks again for spotting this one. Since the PR has been idle for a couple of weeks and I'd like to ship it in 3.2.1, I pushed the review follow-ups directly to your branch (your original commit is untouched, so the fix stays credited to you).

What I applied, in a0a19a5:

  • spec_parser.py: replaced the pattern.search(expr) gate with expr.isidentifier(). It expresses the invariant the fast-path actually wants ("the whole string is a single variable name"), keeps your a^b / (a)v(b) fix, and avoids the two regressions from the review (unspaced != and a guard literally named v). It also fixes the sibling bug where unspaced comparisons like x==1 / x>=1 were swallowed as variable names.
  • dispatcher.py: Listeners.build now catches ValueError alongside SyntaxError. Expressions such as cond="user.age" now reach the AST allowlist, and this keeps the friendly InvalidDefinition: Failed to parse boolean expression '...' instead of leaking ValueError: Unsupported expression structure: Attribute.
  • tests/test_spec_parser.py: kept your test_classical_operators_without_spaces and added the three regression tests that pin the behaviors CI was not covering: unspaced !=, unspaced comparison, and a bare v as a variable name. I verified all three fail on your previous commit and pass now.
  • docs/releases/3.2.1.md: added the user-facing entry with a runnable doctest, as we document every bugfix there.

Full suite is green (1947 passed) with 100% branch coverage, plus ruff, mypy and pyright.

Merging once CI goes green. Thanks for the contribution!

Catching a bare `ValueError` around `parse_boolean_expr` was too broad: the
`variable_hook` runs user code while resolving names (`search_name` reads
attributes from the model, which may execute a property), so an unrelated
`ValueError` raised there was reported as
`InvalidDefinition: Failed to parse boolean expression '<name>'`, hiding the
real cause.

The parser now raises `UnsupportedExpression` (a `ValueError` subclass, so
`io.evaluators` and user code catching `ValueError` keep working) when a node
kind is rejected by the allowlist, and `Listeners.build` catches only that.
Errors from name resolution propagate untouched.

Also parametrizes the unspaced-operator tests, adding a falsy case, and drops
an inaccurate claim from the release notes: a name that is not a valid
identifier can never be silently resolved, since `search_name` only matches
names present in `dir(obj)`.

Signed-off-by: Fernando Macedo <fgmacedo@gmail.com>
@fgmacedo

fgmacedo commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Follow-up on my own change: I reviewed a0a19a5 again and the except (SyntaxError, ValueError) I had added was too broad, so e0ee324 narrows it.

The variable_hook runs user code while resolving names: Listeners.search_name reads attributes from the model with getattr, which executes properties. A model whose property raised ValueError("the model is not configured") was reported as InvalidDefinition: Failed to parse boolean expression 'is_ready', hiding the real cause.

The parser now raises UnsupportedExpression (a ValueError subclass, so io.evaluators and any user code catching ValueError keep working) when the AST allowlist rejects a node, and Listeners.build catches only that. Errors coming from name resolution propagate untouched, with a regression test pinning it.

Same commit also parametrizes the unspaced-operator tests (your two cases are the first entries, plus a falsy case and the two comparison cases) and drops an inaccurate sentence from the release note.

I also checked the security side, since the change routes more strings into the parser: __import__('os'), ().__class__, a.__class__, open(...), subscripts, comprehensions, lambdas, walrus and f-strings are all still rejected by the allowlist, with In(...) remaining the only callable. CI is green across 3.10 to 3.14 with 100% branch coverage.

SonarCloud python:S5778: the `with pytest.raises(...)` block held two calls
that could raise (`Model()` and the state machine constructor), so the test
could pass for the wrong reason if the model constructor started raising
`ValueError` on its own. Build the model outside the block.

Signed-off-by: Fernando Macedo <fgmacedo@gmail.com>
@sonarqubecloud

sonarqubecloud Bot commented Aug 1, 2026

Copy link
Copy Markdown

@fgmacedo
fgmacedo merged commit d911f53 into fgmacedo:develop Aug 1, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants