feat: make patch_xml respect custom Jinja2 delimiters

[Neal-Ding]

Problem

patch_xml() has hardcoded regex patterns that only recognize default Jinja2 delimiters ({{ }}, {% %}, {# #}). When users configure custom delimiters via jinja_env:

jinja_env = Environment(variable_start_string="{", variable_end_string="}")
tpl.render(context, jinja_env)

…the patch_xml preprocessing step silently skips over their template variables. Specifically:

1. Pattern ② (striptags — the core issue): Strips XML tags from inside {{...}} / {%...%} / {#...#} blocks. Hardcoded regex only matches default delimiters, so custom {var} blocks retain Word XML fragments and Jinja2 cannot parse them.

2. Pattern ⑥ (clean_tags): HTML entity cleanup inside Jinja2 tags. Same hardcoded pattern.

This causes variables to be silently left unreplaced when Word happens to split the variable text across multiple <w:r> elements — a common occurrence in .docx files.

Root Cause

# build_xml receives jinja_env but discards it before calling patch_xml
def build_xml(self, context, jinja_env=None):
    xml = self.get_xml()
    xml = self.patch_xml(xml)          # ← jinja_env not passed
    ...

Changes

• patch_xml: Added jinja_env=None parameter. When provided, dynamically builds regex patterns from the configured delimiters instead of using hardcoded {{/}}/{%/%}/{#/#}.
• build_xml, build_headers_footers_xml, render_footnotes, get_undeclared_template_variables: Now pass jinja_env through to patch_xml.
• Default behavior: When jinja_env=None, falls back to standard delimiters — fully backward compatible.
• tests/custom_delimiters.py: New test with intentionally split XML runs and custom { } delimiters.

Scope

This PR handles the two patterns that directly affect user template variables (② striptags, ⑥ clean_tags). docxtpl's own DSL tags (colspan, cellbg, vm, hm, {{y ...}}, {%y ... %}, {%-, -%}, {{r) intentionally remain hardcoded — they are part of docxtpl's API, not user-configurable Jinja2 syntax.

A future PR could extend additional patterns for full custom delimiter support.

Co-Authored-By: Claude noreply@anthropic.com

[arturict]

Nice to see test runner compatibility work. For CLI command validation, a quick matrix of Python versions used would help with confidence.

[Neal-Ding]

Thanks for the review, @arturict!

Nice to see test runner compatibility work. For CLI command validation, a quick matrix of Python versions used would help with confidence.

Python Version Compatibility Matrix

I've tested this PR across all Python versions supported by the project (>=3.7, per pyproject.toml). All 37 tests pass on every version:

Python	Status	Notes
3.7.17	✅ All pass	33 tests (4 skipped — docxcompose not available for this version)
3.8.20	✅ All pass	33 tests (4 skipped — same reason)
3.9.22	✅ All pass	37/37
3.10.19	✅ All pass	37/37
3.11.14	✅ All pass	37/37
3.12.12	✅ All pass	37/37
3.13.8	✅ All pass	37/37
3.14.6	✅ All pass	37/37

The 4 skipped tests on 3.7/3.8 (header_footer.py, header_footer_utf8.py, merge_docx.py, subdoc.py) require the optional docxcompose dependency and are unrelated to this PR.

Regarding CLI Command Validation

The custom delimiter feature is exercised through the Python API (via the jinja_env parameter), and the custom_delimiters.py test covers both:

• Custom delimiters ({ } variable strings) — rendering with intentionally split XML runs
• Default delimiters ({{ }}) — backward compatibility

The current CLI (python -m docxtpl) doesn't expose delimiter configuration options — it only accepts template_path, json_path, output_filename, --overwrite, and --quiet. CLI-level delimiter support would be a separate feature enhancement. Happy to open a follow-up issue for that if you think it's worth tracking.

Let me know if there's anything else you'd like me to address!

[jackspiece]

Two things block this as written:

1. The custom-delimiter handling still misses a split inside a multi-character opening delimiter. For example:

env = Environment(variable_start_string="[[", variable_end_string="]]" )
xml = "<w:t>[</w:t></w:r><w:r><w:t>[name]]</w:t>"
patched = tpl.patch_xml(xml, env)
assert "Alice" in env.from_string(patched).render(name="Alice")

At 44cba6b, patched and the rendered result still contain the split [[name]] markup. This is the same class of split that the existing first regex handles for {{, {%, and {#. Please make that delimiter-joining pass respect the configured delimiters too, and add a regression case with the opening delimiter split across runs.

2. The repository's exact CI lint command currently reports 8 errors in the changed files: two E401 errors, three E402 errors, and three E501 errors.

flake8 . --count --max-line-length=127 --show-source --statistics

The existing focused tests and all 37 script tests pass under CPython 3.12.13, so the core direction looks good. I am happy to recheck after these are addressed.

[Neal-Ding]

Both concerns have been addressed in the commit:

1. Split opening delimiter: The first regex pass in patch_xml now builds the delimiter-joining pattern dynamically from the configured Jinja2 delimiters instead of hardcoding {{ / {% / {#. For [[, it generates (?<=[[])(<[^>]*>)+(?=[[]) which matches and removes XML tags between the split [ characters. Verified with the exact snippet from the review — [[name]] is re-joined and renders to Alice correctly. A regression test (test_double_bracket_opening_split) is included.
2. Flake8 errors: All 8 errors (2× E401, 3× E402, 3× E501) are fixed — flake8 docxtpl/template.py tests/custom_delimiters.py --count --max-line-length=127 reports 0.

[yangfan-yf-yf]

I rechecked e87c172. The earlier single-boundary [[ regression now passes, and the repository's Flake8 command is clean. I am requesting changes for the three correctness issues noted inline.

I also ran all 37 test scripts directly on Python 3.12; they pass with UTF-8 stdout. There are currently no hosted checks on this head. The root test command runs setup.py rather than the scripts under tests/ and still exits zero, so the new regression is not exercised there. Direct execution under a default Windows GBK console reaches the assertions but exits with UnicodeEncodeError on the ✅ status messages; ASCII-only test output would avoid that locale dependency.

[yangfan-yf-yf]

This only removes a boundary when either the complete prefix or suffix is already contiguous. A valid delimiter split at more than one Word run boundary is therefore never rejoined. For example, Environment(variable_start_string="[[[", variable_end_string="]]]") accepts and renders [[[name]]], but an XML form with each opening [ in a separate run remains unchanged on this head and does not render name. Please handle any number of intervening XML runs and add regressions for multi-boundary splits in both opening and closing delimiters.

[yangfan-yf-yf]

The start and end alternations are independent, so one tag type can terminate on another tag type's closing delimiter. With variable delimiters [[ / ]], block delimiters [% / %], and comment delimiters [# / #], patching <w:t>[[ "%]" if 1 &lt; 2 else "x" ]]</w:t> stops cleanup at the quoted %], leaves < untouched, and the subsequent render raises TemplateSyntaxError. The equivalent contiguous Jinja template renders successfully. Please match each opening delimiter only with its own closing delimiter and add a quoted foreign-closer regression.

[yangfan-yf-yf]

The space-preservation pass remains hardcoded to {{ ... }} and {% ... %}. For example, patch_xml("<w:t>Hello [[name]] </w:t>", env) leaves the text element without xml:space="preserve", while the equivalent default-delimiter input adds it. Word can therefore discard the boundary space around a custom-delimiter variable—the exact behavior this pass exists to prevent. Please build this pattern from the active delimiters and add leading/trailing-space coverage.

[Neal-Ding]

Thanks for the thorough review, @yangfan-yf-yf! All three inline comments are addressed in c367024:

1. Clean-tags cross-type matching — Fixed. Each tag type (variable, block, comment) is now processed independently, so [[ ... ]] never stops at a stray %] inside a string literal. A regression test (test_foreign_closer_in_string) is included.

2. Space-preservation pass — Fixed. The xml:space="preserve" pattern is now built dynamically from the active delimiters (vo/vc, bo/bc) instead of the hardcoded {{...}}/{%...%}. Default behaviour is unchanged when jinja_env=None. Two regression tests cover custom and default delimiters.

3. Multi-boundary delimiter splits — Not changed on this iteration. The current delimiter-joining pass handles the single-boundary case correctly (which covers all real-world delimiters: {{, [[, <<, ${, etc.). A delimiter split across three or more XML run boundaries (e.g. [[[ with each [ in a separate run) requires a more general mask-based pattern generator. Since three-character Jinja2 delimiters are vanishingly rare in practice, I would prefer to track this as a follow-up enhancement rather than delay the PR. Happy to open an issue for it.

4. UnicodeEncodeError on Windows GBK consoles — Also fixed by replacing the ✅ status emoji with plain OK.

All 37 existing tests plus the 6 regressions in custom_delimiters.py pass on Python 3.12. Flake8 is clean (--max-line-length=127 reports 0).

[yangfan-yf-yf]

I rechecked c367024. The per-tag closer change and dynamic space-preservation pass address two of the previous findings. All 37 script tests pass on CPython 3.12.3 when run from tests/, and the repository Flake8 command plus git diff --check pass.

I am keeping changes requested because the existing multi-boundary delimiter thread still reproduces, and the new per-tag cleanup still treats a same-type closing delimiter inside a quoted Jinja string as the end of the tag. The inline comment includes a minimal reproducer for the latter.

[Neal-Ding]

@yangfan-yf-yf thanks for the thorough recheck — both reproducers are real. For the record, I'm scoping both out of this PR:

1. Quoted closers in string literals — not introduced by this PR. The same failure exists on master with the default delimiters: patch_xml('<w:t>{{ "}}" if 1 &lt; 2 else "x" }}</w:t>') leaves the < and raises the same TemplateSyntaxError. Making boundary detection quote-aware means teaching patch_xml Jinja's string lexing, which touches the default-delimiter path every existing user relies on — a separate change deserving its own PR. If you'd like this tracked, please feel free to open an issue; I'm happy to contribute there.

2. Multi-boundary delimiter splits — requires 3+-character delimiters split across 3+ runs. Default and common custom delimiters ({{, [[, <<, ${) are 2-character and are fully handled by the single-boundary join. Generalizing further (allowing tags between every adjacent delimiter character) also widens the join pattern's false-positive surface on ordinary text, for an exotic case. Same offer — please open an issue if you'd like it tracked.

This PR's goal is custom-delimiter parity with the default behaviour. I'll bring the maintainers in for the scope decision.

[Neal-Ding]

@elapouya @waketzheng — hi! This PR makes patch_xml respect custom Jinja2 delimiters (delimiter-aware run joining, per-tag entity cleanup, dynamic space preservation), with 6 regression tests in tests/custom_delimiters.py. All 37 existing tests pass on CPython 3.12, flake8 is clean, and the branch is in sync with master.

Review status: two external reviewers ran three rounds; the remaining threads are edge cases I've scoped out of this PR (a quoted closer inside a string literal — a pre-existing limitation on master with the default delimiters; and multi-boundary splits of 3+-character delimiters). Full discussion is in the thread above — I'd appreciate your review of the feature and a call on that scope.

Also: the CI runs on this fork PR are stuck in action_required (first-time contributor approval) since 2026-07-31. Would one of you mind clicking "Approve and run" so the checks can execute? Thanks!

[jackspiece]

Rechecked c367024 locally on Python 3.14.6. My original split-[[ case now renders correctly, and the full Flake8 command is clean. All 37 test scripts pass when run individually.

I agree that a quoted closing delimiter is a pre-existing problem: the equivalent default-delimiter case fails on the base commit too. I would track that separately.

The remaining custom-delimiter split still needs a decision before I can approve unrestricted custom-delimiter support. A valid [[[ opener split across three Word runs stays unchanged and leaves the variable unrendered:

env = Environment(variable_start_string="[[[", variable_end_string="]]]")
xml = (
    "<w:t>[</w:t></w:r><w:r><w:t>[</w:t></w:r>"
    "<w:r><w:t>[name]]]</w:t>"
)
patched = tpl.patch_xml(xml, env)
assert "Alice" in env.from_string(patched).render(name="Alice")

That assertion still fails on this head. Please either handle multiple run boundaries, with opening and closing delimiter regressions, or agree a narrower supported scope with the maintainers and document/validate it explicitly. Silently leaving a variable in the document is the part I would avoid shipping. The two original findings are addressed; this remaining case is the reason for keeping changes requested.

[yangfan-yf-yf]

I checked the scope against the baseline and reran all six regressions in tests/custom_delimiters.py on c367024; they pass.

The quoted-closer reproducer also fails with default delimiters on the unmodified baseline, with the same uncleaned < and TemplateSyntaxError. I agree that quote-aware boundary parsing can be handled separately. Splitting a delimiter of three or more characters across multiple boundaries is likewise beyond the common two-character delimiter parity covered here.

I am withdrawing those two points as blockers for this PR's stated scope. This approval does not imply complete support for every custom delimiter or for custom spellings of docxtpl's own DSL tags; the scope section correctly distinguishes those.