Regression in 8.x: Terraform escape sequences are no longer processed (regression of #171)

[livingstaccato]

Describe the bug

Terraform escape sequences in quoted strings are no longer processed. "line1\nline2" loads as the literal characters line1\nline2 (backslash + n) instead of an actual newline. Same for \t, \r, \", and \\.

This is a regression of #171 ("Terraform escape sequences are not processed when parsing strings"), fixed in 2025-01 and working in 7.2.1/7.3.1.

Software:

• OS: macOS 15 (arm64)

• Python version: 3.11.16

• python-hcl2 versions tested, each in a clean isolated environment:

  • 7.2.1 — correct
  • 7.3.1 — correct
  • 8.0.0rc1 — wrong
  • 8.1.0 — wrong
  • 8.1.2 — wrong

  So this arrived with the v8 rewrite rather than in a later 8.1.x change.

Snippet of HCL2 code causing the unexpected behaviour:

a = "quote \"in\" here"
b = "line1\nline2"
c = "tab\there"
d = "back\\slash"

Expected behavior (7.2.1)

{'a': 'quote "in" here', 'b': 'line1\nline2', 'c': 'tab\there', 'd': 'back\\slash'}

Actual behavior (8.1.2, quotes stripped for clarity)

{'a': 'quote \\"in\\" here', 'b': 'line1\\nline2', 'c': 'tab\\there', 'd': 'back\\\\slash'}

strip_string_quotes=True removes the surrounding quotes but leaves every escape unprocessed.

Root cause

7.x ran DictTransformer.process_escape_sequences() from strip_quotes(). The v8 rewrite (#203) serializes StringRule contents verbatim (hcl2/rules/strings.py), and no equivalent step exists.

Impact

A parsed value no longer matches what Terraform sees for the same source, and the difference is invisible until the string is used. Because the escape is still one backslash short of round-tripping, re-emitting through hcl2.dumps then re-parsing accumulates further backslashes.

Note this interacts with #289: making strip_string_quotes=True the default would surface unprocessed escapes to everyone by default.

[livingstaccato]

Narrowing this report after reading the implementation, because as filed it overstates the problem.

Default output keeping escape sequences raw is defensible by design: the dict is source-shaped so that dumps() can reconstruct the original, and '"line1\nline2"' is the source. Judged as a change to default output, this is a deliberate consequence of the v8 bidirectional pipeline rather than a defect, and I should not have framed it that way.

What does look like a defect is narrower. strip_string_quotes is documented as producing "backwards-compatible output", and docs/06_migrating_to_v8.md presents it as the v7 path. v7 stripped the quotes and resolved the escapes, in the same strip_quotes() call. v8 does only the first half, so the option delivers neither the source form nor the value:

import hcl2
from hcl2.utils import SerializationOptions

hcl2.loads(r'a = "line1\nline2"' + "\n", serialization_options=SerializationOptions(strip_string_quotes=True))
# {'a': 'line1\\nline2'}   -- quotes gone, escape still raw

Since that option is already documented as one-way and not round-trippable, resolving escapes there costs nothing in reconstruction fidelity.

Worth noting one detail if this is fixed: v7's process_escape_sequences ran its replacements sequentially, so \\n became a backslash followed by a newline. HCL specifies a backslash followed by the letter "n" — an escaped backslash cannot lend its escape to the character after it. Resolving in a single pass gives the correct result and matches what OpenTofu evaluates the same source to, so a fix here need not be bug-compatible with v7.

This overlaps #310, which reports the same option unquoting string literals inside expressions. Both are the same code path in StringRule.serialize, so they are probably one fix.

---

This comment, and the investigation behind it, were produced by an AI assistant (Claude) working on behalf of the issue author. Every reproduction and test result above was executed rather than inferred, but please review with that provenance in mind.