`preserve_heredocs=False` returns heredoc bodies that differ from what Terraform evaluates

[livingstaccato]

Summary

With SerializationOptions(preserve_heredocs=False), loads() returns the
value of a heredoc rather than its source form. Three things about that value
differ from what Terraform/OpenTofu evaluates the same source to, on inputs as
small as three lines.

Every expected value below was produced by evaluating the source with OpenTofu
v1.12.5 (tofu console, jsonencode of the resulting local), not by reading
the spec.

This is not a regression. All of it behaves the same way in 7.2.1, so none
of it arrived with the v8 rewrite and nothing here is urgent. I am raising it
because the values disagree with the reference implementation, not because
anything recently broke — please triage it accordingly.

Reproduction

import hcl2
from hcl2 import SerializationOptions

OPTS = SerializationOptions(preserve_heredocs=False, strip_string_quotes=True)

CASES = {
    "trailing newline":  ("x = <<EOT\nline\nEOT\n",                       "line\n"),
    "trailing blank":    ("x = <<EOT\nbody  \n\nEOT\n",                   "body  \n\n"),
    "tab indent":        ("x = <<-EOT\n\ta\n\t\tb\n\tEOT\n",              "a\n\tb\n"),
    "whitespace line":   ("x = <<-EOT\n    a\n      \n    b\n    EOT\n",  "a\n      \nb\n"),
}

for name, (src, opentofu) in CASES.items():
    got = hcl2.loads(src, serialization_options=OPTS)["x"]
    print(f"{'ok ' if got == opentofu else 'BAD'} {name:17} hcl2={got!r:24} opentofu={opentofu!r}")

Actual output on 8.1.3:

BAD trailing newline  hcl2='line'                   opentofu='line\n'
BAD trailing blank    hcl2='body  \n'               opentofu='body  \n\n'
BAD tab indent        hcl2='\ta\n\t\tb'             opentofu='a\n\tb\n'
BAD whitespace line   hcl2='a\n  \nb'               opentofu='a\n      \nb\n'

History

case	7.2.1	8.1.3	OpenTofu
trailing newline	'line'	'line'	'line\n'
trailing blank	'body'	'body \n'	'body \n\n'
tab indent	'\ta\n\t\tb'	'\ta\n\t\tb'	'a\n\tb\n'
whitespace line	'a\n \nb'	'a\n \nb'	'a\n \nb\n'

Cases 3 and 4 are byte-identical across both major versions. Case 2 is the only
one that has moved, and it moved in the right direction: #318 replaced a blanket
rstrip("\n\t ") that discarded the whole blank line, so 8.1.3 keeps more of it
than 7.2.1 did — just not the last newline.

So no fix is being asked to restore anything. What follows is a request to
change longstanding behaviour to match the reference implementation, which is a
larger ask than a regression fix and worth weighing as one.

The three causes

1. The newline before the closing marker is dropped (cases 1 and 2)

_strip_closing_marker_line in hcl2/rules/strings.py removes the closing
marker's indentation and the newline separating it from the last content
line:

text = re.sub(r"[ \t]*\Z", "", text)
return re.sub(r"\r?\n\Z", "", text)

The indentation is correctly not part of the value — the spec says the closing
marker "may also have an arbitrary number of spaces preceding it on its line".
The newline is a different matter. The spec ends the template when the
identifier "subsequently appears again on a line of its own", so every content
line, the last one included, is terminated by its own newline, and that newline
is content. <<EOT\nline\nEOT is "line\n".

Case 2 is the same bug one line further in: once the final newline is consumed,
the blank line before it becomes the new last line and loses its newline too.

2. <<- measures indentation in spaces only (case 3)

HeredocTrimTemplateRule.serialize computes
len(line) - len(line.lstrip(" ")). A tab-indented heredoc measures 0 on every
line, so min_spaces is 0 and no dedent happens at all.

The spec's wording is on the current code's side — it says "the minimum number
of leading spaces". But the reference implementation does not read it that
narrowly: OpenTofu dedents <<-EOT\n\ta\n\t\tb\n\tEOT to "a\n\tb\n", removing
one tab from each line. Measuring in characters (line.lstrip()) reproduces
OpenTofu on both tab- and space-indented input, and is identical to the current
behaviour whenever the indentation is spaces.

Reasonable to close as wontfix if matching the spec's letter is the goal — but
then it is worth saying so in the option's docs, since the practical effect is
that a tab-indented heredoc silently keeps its indentation.

3. A whitespace-only line is dedented (case 4)

The blank-line handling added in #318 correctly skips whitespace-only lines when
measuring the margin, but the slice line[min_spaces:] is still applied to
them afterwards. OpenTofu leaves such a line exactly as written: a six-space
line inside a four-space heredoc stays six spaces rather than becoming two.

The same defect on the write side

Fixing the read exposes a matching bug in strings_to_heredocs.
_deserialize_string_as_heredoc builds f"<<EOF\n{content}\nEOF", adding a
newline before the closing marker that the value already carries — so the body
is emitted one line longer than the string it came from.

The two errors cancel inside this library's own round trip, which is why no test
notices. They do not cancel against Terraform. Running
test/integration/specialized/heredocs.tf through
flatten → strings_to_heredocs → restore and evaluating both files with
OpenTofu, 5 of the 11 locals come back with different values:

BAD simple             original='hello world\n'      restored='hello world'
BAD with_quotes        original='say "hello"\n'      restored='say "hello"'
BAD with_backslashes   original='path\\to\\file\n'   restored='path\\to\\file'
BAD json_content       original='{"key": "value"}\n' restored='{"key": "value"}'
BAD blank_line_only    original='\n'                 restored=''

A value that does not end in a newline cannot be written as a heredoc at all
without gaining one, so such a value has to stay a quoted string.

Suggested shape of the fix

Given the text between the opening marker's newline and the closing delimiter,
the value is that text with only the closing marker's horizontal whitespace
removed:

body = re.sub(r"[ \t]*\Z", "", match.group(2))   # closing marker indent only

and then, for <<-, a dedent that leaves whitespace-only lines alone:

lines = body.split("\n")
indents = [len(line) - len(line.lstrip()) for line in lines if line.strip()]
margin = min(indents) if indents else 0
lines = [line[margin:] if line.strip() else line for line in lines]
body = "\n".join(lines)

That reproduces OpenTofu on all four cases above, and on twelve more I checked
(indented closing marker, trailing spaces on a content line, interior blank
line, empty body, blank-line-only body, uneven dedent, single-character and
hyphenated delimiters, and CRLF forms of both heredoc kinds).

Compatibility

This changes the value loads() returns under preserve_heredocs=False for
input that parses cleanly today. The default preserve_heredocs=True source
form is untouched, so round-tripping is unaffected.

Unlike the value changes shipped in 8.1.3, this one is not restoring pre-8.x
behaviour — 7.2.1 returned the same wrong values — so anyone consuming heredoc
bodies since v7 has been seeing these values all along and may have compensated
downstream. That argues for a minor release rather than a patch, or for putting
it behind an option if you would rather not move the default at all.

Happy to open a PR in whichever of those shapes you prefer. The branch I have
updates the 42 existing assertions that encode the old values, and adds a
bin/heredoc_ground_truth script that re-derives every expectation from
whatever tofu/terraform binary is on PATH, so the provenance can be checked
rather than taken on trust. It is deliberately not wired into the test run —
the suite must pass without a Terraform binary.

---

---

The investigation behind this report was produced with AI assistance, working on my behalf. Every reproduction, version comparison and measurement cited was executed rather than inferred; I reviewed it before filing.

---

This issue, and the investigation behind it, were produced by an AI assistant (Claude) working on behalf of the author. Please review with that provenance in mind.

[livingstaccato]

Update: opened as #335, which also closes #330.

_{🤖 Drafted with Claude Code.}