diff --git a/CHANGELOG.md b/CHANGELOG.md index 61dc143a..8cab16a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,18 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. ## \[Unreleased\] -- Nothing yet. +### Fixed + +- A heredoc whose interpolation spans lines is not flattened. The quoted form cannot hold one: the newlines inside `${...}` are expression source, where OpenTofu rejects an escaped newline and a raw one makes the string span lines, which it also rejects. It used to emit the raw version -- output neither Terraform nor this library could read, written with no error -- and now hands the heredoc back in the form `preserve_heredocs=True` produces, which reads back as that heredoc. Declining is the only answer that does not change what the document means. ([#347](https://github.com/amplify-education/python-hcl2/issues/347)) +- `$${` and `%%{` resolve to `${` and `%{` in the value form, in both quoted strings and heredocs. They are HCL's escapes for a literal sigil, exactly as `\"` is for a quote, and OpenTofu evaluates `"$${esc}"` to the six characters `${esc}`; returning them doubled made the value differ from the one Terraform reads, in the one mode that promises the value. ([#336](https://github.com/amplify-education/python-hcl2/issues/336)) +- `strings_to_heredocs` resolves every escape the reader does. It knew `\n`, `\r`, `\"` and `\\`, so `"a\tb\n"` was written into the body as a backslash and a `t` -- two characters where Terraform reads one tab -- and `\uNNNN` fared the same. It now uses `process_escape_sequences`, the package's one implementation of that alphabet. ([#329](https://github.com/amplify-education/python-hcl2/issues/329)) +- Escapes are no longer added or resolved inside `${...}`. The text there is expression source, where a nested `"..."` is a string literal of its own: OpenTofu reads `"${upper("a")}"` as `A` and rejects `"${upper(\"a\")}"` outright, so escaping through an interpolation produced source the reference implementation will not parse, and resolving through one closed a nested literal early. Both directions now work on the spans the text is actually made of, and the scan knows the things inside an expression that can carry a non-structural brace: a string literal, HCL's `#`, `//` and `/* */` comments, and the nested expressions a string literal may itself contain -- OpenTofu evaluates `${1 /* } */ + 2}` to 3 and `"a ${upper("v${ "{" }w")} b"` to `a V{W b`, so counting either brace closed the expression inside itself. ([#339](https://github.com/amplify-education/python-hcl2/issues/339)) +- Flattened heredoc bodies match the values Terraform and OpenTofu evaluate the same source to. Three things differed, all checked against OpenTofu v1.12.5 rather than read off the spec: the newline terminating the last content line was dropped (`<HCL2 deserialization and reconstruction.** | -| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. | +| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb\n"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. Either way the body keeps the newline that terminates its last line, as Terraform's does. | | `force_operation_parentheses` | `bool` | `False` | Force parentheses around all operations | | `preserve_scientific_notation` | `bool` | `True` | Keep scientific notation as-is | | `strip_string_quotes` | `bool` | `False` | Yield string *values* rather than source text: remove surrounding quotes (e.g. `"hello"` instead of `'"hello"'`) and resolve escape sequences (`"a\nb"` becomes a real newline). String literals inside expressions keep their quotes, so `upper("x")` stays `'${upper("x")}'`. **Breaks JSON->HCL2 deserialization and reconstruction.** | @@ -127,7 +127,7 @@ text = dumps(data, deserializer_options=DeserializerOptions( | Field | Type | Default | Description | |---|---|---|---| | `heredocs_to_strings` | `bool` | `False` | Convert heredocs to plain strings | -| `strings_to_heredocs` | `bool` | `False` | Convert strings with `\n` to heredocs | +| `strings_to_heredocs` | `bool` | `False` | Convert newline-terminated strings to heredocs. A value that does not end in a newline is left as a quoted string, because a heredoc body always ends in one and writing it as a heredoc would change the value. | | `object_elements_colon` | `bool` | `False` | Use `:` instead of `=` in object elements | | `object_elements_trailing_comma` | `bool` | `True` | Add trailing commas in object elements | diff --git a/docs/06_migrating_to_v8.md b/docs/06_migrating_to_v8.md index 0c4703bf..467c1d0d 100644 --- a/docs/06_migrating_to_v8.md +++ b/docs/06_migrating_to_v8.md @@ -213,17 +213,18 @@ This restores the v7 dict shape but disables round-trip support and comment pres ```python hcl2.loads('x = <<-EOT\n line1\n line2\n EOT\n', serialization_options=V7_COMPAT) -# {'x': 'line1\nline2'} +# {'x': 'line1\nline2\n'} ``` -Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2"'`), because that output is meant to be reconstructable. +Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2\\n"'`), because that output is meant to be reconstructable. -Two details of heredoc values are easy to trip over, and both match how HCL itself behaves: +Three details of heredoc values are easy to trip over, and all three match how HCL itself behaves: +- **The body ends with a newline.** Every content line is terminated by its own newline, the last one included, so `< bool: + r"""Whether *inner* could possibly resolve to a newline-terminated value. + + Only such a value can be written as a heredoc, and resolving the escapes to + find out costs a pass over the whole string -- for a document where nothing + ends in a newline, every one of those passes is discarded. The last two + characters answer it: a value ends with a newline only if its source ends + with one, escaped or real. + + Conservative on purpose. `"a\\n"` ends with a backslash and an `n` and + passes here, then resolves to those two characters and is rejected by the + check that actually matters. + """ + return inner.endswith("\\n") or inner.endswith("\n") + + +def _unescape_heredoc_body(inner: str) -> str: + r"""Resolve a quoted string's escapes for a body that interprets none. + + A heredoc body is read literally, so anything the quoted form spelled as + an escape has to become the character itself: `\t` a tab, `\u00e9` an + accented e. `process_escape_sequences` is the package's one implementation + of that alphabet, and using it here is what stops this path from resolving + a shorter list than the reader does. + + Only in literal spans. Inside `${...}` the text is expression source, and + an escape there belongs to a string literal written inside the expression: + OpenTofu reads `"${upper("a\"b")}"` as `A"B`, so resolving that `\"` would + close the nested literal early and change what the expression says. + """ + return map_literal_spans(inner, process_escape_sequences) + + +# A line that could end a heredoc: the delimiter word alone, give or take +# surrounding spaces and tabs -- and a carriage return, because the body is +# split on "\n" and a CRLF line hands back its own `\r`. OpenTofu ends a +# heredoc on `EOF\r` exactly as it does on `EOF `, so a CRLF body carrying +# the delimiter has to count. +_CLOSING_MARKER_LINE = re.compile(r"[ \t]*([a-zA-Z][a-zA-Z0-9._-]*)[ \t\r]*") + + +def _heredoc_delimiter(content: str) -> str: + """Return a delimiter the body does not close on its own. + + `EOF` unless the body holds a line that would end the heredoc there, in + which case a numbered variant is used. The word matters: a log excerpt, a + shell script or an embedded config is exactly the sort of value people put + in a heredoc, and `EOF` is exactly the word such a payload tends to + contain. Writing one blindly produced a file that no longer parsed. + """ + occupied = set() + for line in content.split("\n"): + match = _CLOSING_MARKER_LINE.fullmatch(line) + if match is not None: + occupied.add(match.group(1)) + + if "EOF" not in occupied: + return "EOF" + + suffix = 1 + while f"EOF_{suffix}" in occupied: + suffix += 1 + return f"EOF_{suffix}" + + +def _interpolation_spans(text: str) -> List[str]: + """The `${...}` and `%{...}` spans of *text*, in order.""" + return [chunk for kind, chunk in split_template(text) if kind == INTERPOLATION] + + +def _expressible_as_heredoc(content: str, source: str) -> bool: + """Whether *content* can be a heredoc body without changing. + + A heredoc body is read literally, so it can hold a carriage return only + where one ends a line. A lone `\r` makes the file unreadable rather than + merely different: OpenTofu rejects `< LarkRule: if match: return self._deserialize_heredoc(value[1:-1], False) - if self.options.strings_to_heredocs: - inner = value[1:-1] - if "\\n" in inner: - return self._deserialize_string_as_heredoc(inner) + if self.options.strings_to_heredocs and _may_end_a_heredoc(value[1:-1]): + content = _unescape_heredoc_body(value[1:-1]) + # A heredoc's closing marker sits on a line of its own, so + # any body with content in it ends with a newline. A value + # that does not cannot be written as one without gaining + # that character, so it stays a quoted string. + # + # The empty string is the one value this excludes that a + # heredoc could in fact express -- `< HeredocTemplateRule: - """Convert a quoted string with escaped newlines back into a heredoc.""" - # Single-pass unescape: \\n → \n, \\" → ", \\\\ → \ - content = re.sub( - r'\\(n|"|\\)', - lambda m: "\n" if m.group(1) == "n" else m.group(1), - inner, - ) - heredoc = f"< HeredocTemplateRule: + """Wrap an unescaped body, already newline-terminated, in heredoc syntax.""" + delimiter = _heredoc_delimiter(content) + heredoc = f"<<{delimiter}\n{content}{delimiter}" return HeredocTemplateRule([HEREDOC_TEMPLATE(heredoc)]) def _deserialize_expression(self, value: str) -> ExprTermRule: diff --git a/hcl2/rules/strings.py b/hcl2/rules/strings.py index 76ce0660..e9a991a6 100644 --- a/hcl2/rules/strings.py +++ b/hcl2/rules/strings.py @@ -17,6 +17,7 @@ STRING_CHARS, TEMPLATE_STRING, ) +from hcl2.template import has_multi_line_span, map_literal_spans, resolve_escaped_markers from hcl2.utils import ( HEREDOC_PATTERN, HEREDOC_TRIM_PATTERN, @@ -27,23 +28,40 @@ ) -def _strip_closing_marker_line(text: str) -> str: - r"""Drop the closing marker line's indentation and the one newline before it. +def _strip_closing_marker_indent(text: str) -> str: + r"""Drop the whitespace indenting the closing marker on its own line. A heredoc body always ends ``...\n``, where ```` is the - whitespace preceding the closing marker on its own line. The spec allows - "an arbitrary number of spaces preceding it", and neither that indentation - nor the newline separating it from the last content line is part of the - value. The newline may be ``\r\n``, since heredocs parse in CRLF files. - - Everything else is: additional blank lines, and trailing spaces on a - content line. The latter are safe because a content line always ends with - its own newline, so the indentation match never reaches them. This replaces - a blanket ``rstrip("\n\t ")``, which could not tell the two apart and - discarded both. + whitespace preceding the closing marker. The spec allows "an arbitrary + number of spaces preceding it", and that indentation is not part of the + value. + + The newline before it *is*. The spec ends the template where the delimiter + "subsequently appears again on a line of its own", so every content line, + the last one included, is terminated by its own newline: ``< str: + r"""Escape literal text so it can sit inside a quoted string. + + A carriage return is escaped alongside the newline: raw, it would break the + quoted string it is being written into. OpenTofu rejects `"ab"` with + "No closing marker was found for the string". + """ + return text.replace("\\", "\\\\").replace('"', '\\"').replace("\r", "\\r").replace("\n", "\\n") class InterpolationRule(LarkRule): @@ -131,16 +149,29 @@ def serialize(self, options=SerializationOptions(), context=SerializationContext @staticmethod def _serialize_part_as_value(part, options, context) -> str: - """Serialize one part, resolving escapes in literal text only. + """Serialize one part into what the reader sees, by its terminal. + + Literal text has its escapes resolved. An interpolation is passed + through untouched: its text is expression source, not literal content, + so an escape inside it is not this string's to resolve. - Interpolations and escaped interpolation/directive markers are passed - through untouched: their text is expression source, not literal - content, so an escape inside them is not this string's to resolve. + `$${` and `%%{` are neither. They are escapes for a literal `${` and + `%{`, so the value carries the single sigil -- `"$${esc}"` is the six + characters `${esc}` to Terraform, not seven. """ serialized = part.serialize(options, context) - if part.content.lark_name() == "STRING_CHARS": + terminal = part.content.lark_name() + if terminal == "STRING_CHARS": return process_escape_sequences(serialized) - return serialized + if terminal in ("ESCAPED_INTERPOLATION", "ESCAPED_DIRECTIVE"): + return serialized[1:] + # Anything else is a nested template rule -- a directive and everything + # it encloses arrive as one part, so a marker written between `%{ if }` + # and `%{ endif }` never reaches the branch above and stayed doubled, + # while the same content in a heredoc resolved. The helper is + # span-aware: it resolves the markers in the literal stretches and + # leaves the directives themselves, which are expression source, alone. + return resolve_escaped_markers(serialized) class HeredocTemplateRule(LarkRule): @@ -164,19 +195,28 @@ def heredoc(self): def serialize(self, options=SerializationOptions(), context=SerializationContext()) -> Any: """Serialize the heredoc, optionally stripping to a plain string.""" heredoc = self.heredoc.serialize(options, context) + raw = heredoc if not options.preserve_heredocs: match = HEREDOC_PATTERN.match(heredoc) if not match: raise RuntimeError(f"Invalid Heredoc token: {heredoc}") - heredoc = _strip_closing_marker_line(match.group(2)) + heredoc = _strip_closing_marker_indent(match.group(2)) if options.strip_string_quotes: - # The caller asked for the value, so hand back the body as-is: - # real newlines, no escaping. The escaping below exists only to - # build the quoted-string *source* form returned otherwise. - return heredoc - heredoc = heredoc.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") - return f'"{heredoc}"' + # The caller asked for the value: real newlines, no escaping. + # `$${` and `%%{` are resolved, being escapes for a literal + # `${` and `%{` rather than characters of the value. + return resolve_escaped_markers(heredoc) + # Only the literal spans are escaped. Inside `${...}` the text is + # expression source, and escaping a quote there rewrites someone + # else's code: `${upper("a")}` would become `${upper(\\"a\\")}`, + # which OpenTofu rejects outright. + if has_multi_line_span(heredoc): + # No quoted spelling exists, so the heredoc is handed back as + # it was written -- the same form `preserve_heredocs=True` + # produces, which reads back as this heredoc. + return f'"{raw.rstrip(self._trim_chars)}"' + return '"' + map_literal_spans(heredoc, _escape_for_quoted_source) + '"' result = heredoc.rstrip(self._trim_chars) if options.strip_string_quotes: @@ -197,49 +237,56 @@ def lark_name() -> str: def serialize(self, options=SerializationOptions(), context=SerializationContext()) -> Any: """Serialize the trim heredoc, stripping common leading whitespace.""" # See https://github.com/hashicorp/hcl2/blob/master/hcl/hclsyntax/spec.md#template-expressions - # This is a special version of heredocs that are declared with "<<-" - # This will calculate the minimum number of leading spaces in each line of a heredoc - # and then remove that number of spaces from each line - + # This is a special version of heredocs that are declared with "<<-", + # whose body is dedented by the smallest indent any of its lines carries. heredoc = self.heredoc.serialize(options, context) + raw = heredoc if not options.preserve_heredocs: match = HEREDOC_TRIM_PATTERN.match(heredoc) if not match: raise RuntimeError(f"Invalid Heredoc token: {heredoc}") - heredoc = match.group(2) + body = "\n".join(self._dedent(_strip_closing_marker_indent(match.group(2)))) + if options.strip_string_quotes: + # The caller asked for the value: real newlines, no escaping. + return resolve_escaped_markers(body) + if has_multi_line_span(body): + return f'"{raw.rstrip(self._trim_chars)}"' + return '"' + map_literal_spans(body, _escape_for_quoted_source) + '"' + + result = heredoc.rstrip(self._trim_chars) + if options.strip_string_quotes: + return result + return f'"{result}"' - heredoc = _strip_closing_marker_line(heredoc) - lines = heredoc.split("\n") + @staticmethod + def _dedent(body: str) -> List[str]: + """Split *body* into lines and remove the common leading whitespace.""" + lines = body.split("\n") - # calculate the min number of leading spaces in each line + # The margin is the smallest indent any content line carries. + # # The spec measures "any literal string at the start of each line", so a # blank line offers no measurement. Counting it as zero would drag the - # minimum down and cancel the dedent for every other line -- which only - # became reachable once blank lines stopped being stripped above. - min_spaces = sys.maxsize + # margin down and cancel the dedent for every other line. + # + # It also says "spaces", but the reference implementation does not read + # that as narrowly: OpenTofu dedents a tab-indented `<<-` heredoc by one + # tab per level. Measuring whitespace characters rather than spaces + # alone matches it, and is identical to counting spaces on the + # space-indented input that reading the letter of the spec would cover. + margin = sys.maxsize for line in lines: if not line.strip(): continue - leading_spaces = len(line) - len(line.lstrip(" ")) - min_spaces = min(min_spaces, leading_spaces) - if min_spaces == sys.maxsize: - min_spaces = 0 - - # trim off that number of leading spaces from each line - lines = [line[min_spaces:] for line in lines] - - if not options.preserve_heredocs: - lines = [line.replace("\\", "\\\\").replace('"', '\\"') for line in lines] - - if options.strip_string_quotes: - # Value, not source: join with real newlines regardless of - # preserve_heredocs, and skip the escaping done for the quoted form. - return "\n".join(lines) - - sep = "\\n" if not options.preserve_heredocs else "\n" - inner = sep.join(lines) - return '"' + inner + '"' + margin = min(margin, len(line) - len(line.lstrip())) + if margin == sys.maxsize: + margin = 0 + + # A line that offered no measurement is left exactly as written -- + # OpenTofu keeps a six-space line inside a four-space heredoc at six + # spaces rather than two. + return [line[margin:] if line.strip() else line for line in lines] class TemplateStringRule(LarkRule): diff --git a/hcl2/template.py b/hcl2/template.py new file mode 100644 index 00000000..80d00a01 --- /dev/null +++ b/hcl2/template.py @@ -0,0 +1,209 @@ +"""Splitting template text into the parts that mean different things. + +A quoted string or a heredoc body is not one run of literal characters. HCL +reads `${...}` and `%{...}` as expression source, and `$${`/`%%{` as escapes +for a literal `${`/`%{`. Anything that rewrites such text -- resolving escapes, +adding them, or turning one form into another -- has to know which span it is +looking at, or it corrupts the expression inside. + +The grammar already separates these for a quoted string, which is why +`StringRule._serialize_part_as_value` can do the right thing by asking each +part for its terminal. A heredoc body arrives as one opaque token, so the same +distinction has to be recovered from the text. That is what this does. + +The scan is a single left-to-right pass, because the two questions cannot be +answered separately: a splitter run before escapes are resolved would see the +`${` inside `$${` and open an interpolation that is not there. +""" + +from typing import Callable, Iterator, Tuple + +LITERAL = "literal" +INTERPOLATION = "interpolation" + +_OPENERS = {"$": "${", "%": "%{"} +_ESCAPES = {"$": "$${", "%": "%%{"} + + +def _skip_string(text: str, index: int) -> int: + """Return the index just past the string literal opening at *index*. + + A string literal inside an expression is itself a template, so a `${` or + `%{` in it opens a nested expression whose own literals may hold quotes + and braces. OpenTofu evaluates `"a ${upper("v${ "{" }w")} b"` to + `a V{W b`, so taking the next quote as the terminator ended this literal + at the one that *opens* the innermost one, and the brace after it was + then counted as structural. + + The escapes stay escapes here: `"a ${upper("v$${x}w")} b"` is `a V${X}W b`, + so `$${` does not open anything. + """ + length = len(text) + index += 1 + while index < length: + char = text[index] + if char == "\\": + index += 2 + continue + if char in _OPENERS: + if text.startswith(_ESCAPES[char], index): + index += len(_ESCAPES[char]) + continue + if text.startswith(_OPENERS[char], index): + end = _scan_expression(text, index + 1) + if end == -1: + # Unbalanced, so there is no literal to close either. + return length + index = end + continue + if char == '"': + return index + 1 + index += 1 + return length + + +def _skip_comment(text: str, index: int) -> int: + """Return the index just past the comment opening at *index*, or *index*. + + HCL writes them three ways, and all three may hold a brace: `#` and `//` + run to the end of the line, `/* */` to its terminator. OpenTofu evaluates + `${1 /* } */ + 2}` to 3, so a scan that counts that brace closes the + expression in the middle of itself. + """ + if text.startswith("/*", index): + end = text.find("*/", index + 2) + return len(text) if end == -1 else end + 2 + if text.startswith("//", index) or text[index] == "#": + end = text.find("\n", index) + return len(text) if end == -1 else end + return index + + +def _scan_expression(text: str, start: int) -> int: + """Return the index just past the `}` closing the span opened at *start*. + + Braces nest, and three things inside an expression may carry one that is + not structural: a string literal, a comment, and the body of a heredoc + written inline. The first two are skipped here. A brace in a heredoc body + is not, which is a known gap rather than an oversight -- recognising one + means matching its delimiter, and the case has not been seen in the wild. + """ + depth = 0 + index = start + length = len(text) + while index < length: + char = text[index] + if char == "\\": + # A backslash pair is one unit. Without this the scan enters string + # mode at the quote of a `\\"`, then reads the real closing quote as + # another escape and runs to the end of the text, swallowing + # everything after the expression into one span. + index += 2 + continue + if char == '"': + index = _skip_string(text, index) + continue + if char == "#" or text.startswith("//", index) or text.startswith("/*", index): + skipped = _skip_comment(text, index) + if skipped != index: + index = skipped + continue + if char == "{": + depth += 1 + elif char == "}": + depth -= 1 + if depth == 0: + return index + 1 + index += 1 + # Unbalanced. Reported as such rather than swallowed: handing the rest + # back as an expression means nothing escapes it, and the heredoc paths + # then emit raw newlines and unescaped quotes into what is supposed to be + # quoted-string source. Treated as literal it is at least well-formed. + return -1 + + +def split_template(text: str) -> Iterator[Tuple[str, str]]: + """Yield `(kind, chunk)` pairs covering *text* exactly once. + + `kind` is `LITERAL` for text the reader treats as characters, including + the `$${` and `%%{` escapes themselves, and `INTERPOLATION` for a `${...}` + or `%{...}` span, whose content is expression source. + """ + if "$" not in text and "%" not in text: + # The overwhelmingly common case, and the one this used to make + # expensive: a per-character loop where a scan for two characters + # answers the question. `process_escape_sequences` has the same guard. + if text: + yield LITERAL, text + return + + index = 0 + literal_start = 0 + length = len(text) + + while index < length: + char = text[index] + if char == "\\": + index += 2 + continue + if char in _OPENERS: + escape = _ESCAPES[char] + if text.startswith(escape, index): + # An escape is literal text, and consuming it here is what + # stops the `${` inside it from opening a span. + index += len(escape) + continue + if text.startswith(_OPENERS[char], index): + end = _scan_expression(text, index + 1) + if end == -1: + # Nothing closes it, so the rest is literal text -- and the + # leading run has to stay unyielded, or the tail below + # repeats it. + break + if literal_start != index: + yield LITERAL, text[literal_start:index] + yield INTERPOLATION, text[index:end] + index = end + literal_start = index + continue + index += 1 + + if literal_start != length: + yield LITERAL, text[literal_start:length] + + +def resolve_escaped_markers(text: str) -> str: + """Resolve `$${` and `%%{` into the single sigil they stand for. + + Only in literal spans: inside `${...}` the same characters are expression + source, where `$${` does not mean a literal `${`. + """ + return "".join( + chunk.replace("$${", "${").replace("%%{", "%{") if kind == LITERAL else chunk + for kind, chunk in split_template(text) + ) + + +def map_literal_spans(text: str, transform: Callable[[str], str]) -> str: + """Apply *transform* to the literal spans of *text*, leaving the rest alone. + + An escape belongs to the string that carries it, not to an expression + written inside it: escaping or unescaping through an interpolation rewrites + someone else's source and changes what it means. + """ + return "".join(transform(chunk) if kind == LITERAL else chunk for kind, chunk in split_template(text)) + + +def has_multi_line_span(text: str) -> bool: + """Whether any `${...}` or `%{...}` in *text* runs across a line. + + Such a body has no quoted spelling. The newlines inside the span are + expression source, where OpenTofu rejects an escaped one -- "This character + is not used within the language" -- and a raw one makes the quoted string + span lines, which it rejects as well. So the flattened form cannot express + it, and the only answer that does not change what the document means is to + decline. + """ + return any( + kind == INTERPOLATION and ("\n" in chunk or "\r" in chunk) for kind, chunk in split_template(text) + ) diff --git a/test/integration/specialized/heredocs_flattened.json b/test/integration/specialized/heredocs_flattened.json index 95fb4e55..43c6fd2c 100644 --- a/test/integration/specialized/heredocs_flattened.json +++ b/test/integration/specialized/heredocs_flattened.json @@ -1,16 +1,16 @@ { "locals": [ { - "simple": "\"hello world\"", - "multiline": "\"line1\\nline2\\nline3\"", - "with_quotes": "\"say \\\"hello\\\"\"", - "with_backslashes": "\"path\\\\to\\\\file\"", - "trimmed": "\"indented1\\nindented2\"", - "trimmed_mixed": "\"line1\\n line2\\nline3\"", - "json_content": "\"{\\\"key\\\": \\\"value\\\"}\"", + "simple": "\"hello world\\n\"", + "multiline": "\"line1\\nline2\\nline3\\n\"", + "with_quotes": "\"say \\\"hello\\\"\\n\"", + "with_backslashes": "\"path\\\\to\\\\file\\n\"", + "trimmed": "\"indented1\\nindented2\\n\"", + "trimmed_mixed": "\"line1\\n line2\\nline3\\n\"", + "json_content": "\"{\\\"key\\\": \\\"value\\\"}\\n\"", "empty": "\"\"", "empty_trimmed": "\"\"", - "blank_line_only": "\"\"", + "blank_line_only": "\"\\n\"", "after_empty": "\"still parsed\"", "__is_block__": true } diff --git a/test/integration/specialized/heredocs_restored.tf b/test/integration/specialized/heredocs_restored.tf index 05832d52..a7bad307 100644 --- a/test/integration/specialized/heredocs_restored.tf +++ b/test/integration/specialized/heredocs_restored.tf @@ -1,12 +1,18 @@ locals { - simple = "hello world" + simple = <b"` with "No + closing marker was found for the string", while `"a\rb"` evaluates to a + carriage return, which is what the heredoc body actually held. + + The value form is unaffected: it hands back the body, so its newlines and + carriage returns stay real characters. + """ + + FLAT = SerializationOptions(preserve_heredocs=False) + VALUE = SerializationOptions(preserve_heredocs=False, strip_string_quotes=True) + + def test_heredoc_source_form_escapes_carriage_returns(self): + source = loads("a = < str: + flattened = loads(source, serialization_options=self.FLAT) + return dumps(flattened, deserializer_options=self.HEREDOCS) + + def _round_trip(self, source: str) -> str: + restored = self._restore(source) + return loads(restored, serialization_options=self.VALUE)["a"] + + def test_the_value_is_unchanged(self): + self.assertEqual(self._round_trip("a = < str: + return dumps({"x": value}, deserializer_options=self.HEREDOCS) + + def _round_trip(self, value: str) -> str: + return loads(self._write(value), serialization_options=self.VALUE)["x"] + + def test_an_ordinary_body_still_uses_eof(self): + self.assertEqual(self._write(r'"plain\n"'), "x = < str: + return dumps({"a": value}, deserializer_options=HEREDOCS) + + def test_a_tab_becomes_a_tab(self): + self.assertEqual(self._body(r'"a\tb\n"'), "a = < str: + return dumps({"x": value}, deserializer_options=HEREDOCS) + + def test_a_synthesised_sigil_keeps_the_value_quoted(self): + # The value is `\u0024\u007bfoo\u007d\n`, spelled with chr() so this + # test's own source cannot be confused with the characters it means. + esc = chr(92) + "u" + source = '"' + esc + "0024" + esc + "007bfoo" + esc + "007d" + chr(92) + "n" + '"' + self.assertEqual(self._written(source), "x = " + source + "\n") + + def test_a_demoted_interpolation_keeps_the_value_quoted(self): + source = '"' + chr(92) + "u0024" + "${b}" + chr(92) + "n" + '"' + self.assertEqual(self._written(source), "x = " + source + "\n") + + def test_a_real_interpolation_still_converts(self): + self.assertEqual(self._written(r'"${b}\n"'), "x = < str: + return loads(f'a = "{body}"\n', serialization_options=QUOTED_VALUE)["a"] + + def _heredoc(self, body: str) -> str: + return loads(f"a = < list: + return [kind for kind, _ in split_template(body)] + + def _spans(self, body: str) -> list: + return list(split_template(body)) + + def test_a_brace_in_a_nested_expressions_string(self): + # tofu: "a ${upper("v${ "{" }w")} b" -> a V{W b + body = 'a ${upper("v${ "{" }w")} b' + self.assertEqual(self._kinds(body), [LITERAL, INTERPOLATION, LITERAL]) + + def test_that_span_covers_the_whole_expression(self): + body = 'a ${upper("v${ "{" }w")} b' + self.assertEqual(self._spans(body)[1], (INTERPOLATION, '${upper("v${ "{" }w")}')) + + def test_three_levels_of_nesting(self): + # tofu: "a ${upper("p${ lower("Q${ "{" }R") }s")} b" -> a PQ{RS b + body = 'a ${upper("p${ lower("Q${ "{" }R") }s")} b' + self.assertEqual(self._kinds(body), [LITERAL, INTERPOLATION, LITERAL]) + + def test_an_escape_in_a_nested_string_opens_nothing(self): + # tofu: "a ${upper("v$${x}w")} b" -> a V${X}W b + body = 'a ${upper("v$${x}w")} b' + self.assertEqual(self._spans(body)[1], (INTERPOLATION, '${upper("v$${x}w")}')) + + def test_a_directive_nested_in_a_string(self): + # tofu: "a ${upper("v%{ if true }y%{ endif }w")} b" -> a VYW b + body = 'a ${upper("v%{ if true }y%{ endif }w")} b' + self.assertEqual(self._kinds(body), [LITERAL, INTERPOLATION, LITERAL]) + + def test_an_unterminated_nested_expression_is_still_literal(self): + # No closing brace anywhere, so nothing is an interpolation. + self.assertEqual(self._kinds('a ${upper("v${ x") b'), [LITERAL])