Regression in 8.x: empty heredoc fails to parse (regression of #101)

[livingstaccato]

Describe the bug

An empty heredoc fails to parse. This is a regression of #101 ("Unexpected token exception on empty heredoc"), fixed in 2023 and working in 7.2.1 (which returns '').

Software:

• OS: macOS 15 (arm64)

• Python version: 3.11.16

• python-hcl2 versions tested, each in a clean isolated environment:

  • 7.2.1 — returns {'bar': ''}
  • 7.3.1 — returns {'bar': ''}
  • 8.0.0rc1 — raises
  • 8.1.0 — raises
  • 8.1.2 — raises

  So this arrived with the v8 rewrite rather than in a later 8.1.x change.

Snippet of HCL2 code causing the unexpected behaviour:

bar = <<EOF
EOF

Expected behavior (7.2.1)

{'bar': ''}

Exception traceback:

lark.exceptions.UnexpectedCharacters: No terminal matches '<' in the current parser context, at line 1 col 7

bar = <<EOF
      ^
Expected one of:
	* FLOAT_LITERAL
	* HEREDOC_TEMPLATE
	* MINUS
	...

followed by

lark.exceptions.UnexpectedToken: Unexpected token Token('STRING_CHARS', '<<EOF\nEOF\n\n') at line 1, column 7.

Root cause

In hcl2/hcl2.lark, the v8 HEREDOC_TEMPLATE terminal requires at least one body line: the \n? that used to follow the opening marker in 7.x is gone, so a marker immediately followed by the closing delimiter does not match and the input falls through to STRING_CHARS.

A heredoc whose body is a single empty line does parse, which makes the failing case look arbitrary:

>>> hcl2.loads("bar = <<EOF\n\nEOF\n")   # one blank line - OK
{'bar': '"<<EOF\n\nEOF"'}
>>> hcl2.loads("bar = <<EOF\nEOF\n")      # no body - raises

Impact

Generated configurations legitimately contain empty heredocs (an empty user-data script, an empty policy document), and a whole file fails to parse because of one.

[livingstaccato]

Correction and an upgrade to the severity: this is not only a parse failure. In most files it is silent data loss, and the report above understates it.

Having failed to match at the empty heredoc, the lexer scans on and matches a later delimiter, absorbing everything in between:

import hcl2

src = '''locals {
  a = <<EOF
EOF

  b = <<EOF
real body
EOF
}
'''
hcl2.loads(src)

8.1.2 returns:

{'locals': [{'a': '"<<EOF\nEOF\n\n  b = <<EOF\nreal body\nEOF"', '__is_block__': True}]}

b is gone, folded into a's value. No exception is raised. The UnexpectedToken in the original report only appears when the empty heredoc has no later delimiter to latch onto, which makes the loud case the rarer one.

Why the suite did not catch it

test/helpers/terraform-config/ carried an empty-heredoc fixture, added with #117 when this was fixed the first time. The v8 test reorganisation replaced that layout with test/integration/specialized/heredocs.tf, whose 7 heredoc cases all have bodies. Grepping the current suite, no zero-line heredoc appears anywhere, so nothing guarded the behaviour.

On fixing it

Two obvious patches both look right and are both wrong, which seems worth recording:

• Restoring the bare \n? that 21b7cf8 dropped fixes the empty case but keeps the run-on above, because the body can still stretch to a later delimiter.
• Dropping the \n before \s* fixes both of those but lets \s* match nothing, so a word merely ending in the delimiter terminates the body: <<EOF\nsayEOF\nEOF then truncates at sayEOF.

What holds up on all of them is making the body and its trailing newline a single lazy optional group:

HEREDOC_TEMPLATE : /<<(?P<heredoc>[a-zA-Z][a-zA-Z0-9._-]+)\n(?:(?:.|\n)*?\n)??\s*(?P=heredoc)\n/
HEREDOC_TEMPLATE_TRIM : /<<-(?P<heredoc_trim>[a-zA-Z][a-zA-Z0-9._-]+)\n(?:(?:.|\n)*?\n)??\s*(?P=heredoc_trim)\n/

The laziness is load-bearing: a greedy ? prefers to match a body and reintroduces the run-on. The delimiter still has to begin its own line, so the rule from #194 is preserved.

Checked against the current suite (1391 passing before, 1400 after adding coverage for empty, empty-trimmed and blank-line-only heredocs plus a trailing attribute that proves the lexer stops where it should). 6 of the new tests fail without the grammar change. Happy to open a PR if that is useful, or leave it here if you would rather fix it your own way.

---

This comment, and the investigation behind it, were produced by an AI assistant (Claude) working on behalf of the issue author. Every reproduction and test result above was executed rather than inferred, but please review with that provenance in mind.