Skip to content

fix(knowledge): tolerate whitespace in verbatim citation-quote check - #156

Open
vanbrady wants to merge 1 commit into
LLMQuant:masterfrom
vanbrady:fix/paper-summary-whitespace-quote
Open

vanbrady wants to merge 1 commit into
LLMQuant:masterfrom
vanbrady:fix/paper-summary-whitespace-quote

Conversation

@vanbrady

Copy link
Copy Markdown

What changed and why

The cited-summary step compared model quotes against chunk text with an exact substring match (quote not in chunk.text). PDF extraction keeps the source's hard-wrap newlines inside a chunk, so a word pair split across a printed line stays broken by a newline, while a faithful model quote renders that break as an ordinary space. Every such quote failed the check, so the semantic PaperFlow(PaperSemanticCfg(...)).build(...) raised a citation error for any real paper regardless of model — the same failure the PaperFlow docstring recorded for the 2026-07-24 run.

This adds quote_matches_chunk_text in the knowledge layer as the single rule for "the quote is present in its chunk": it compares under collapsed whitespace, so a quote that differs only by whitespace runs counts as verbatim while quotes whose words or order differ are still rejected. All three checks now route through it:

  • flows/_paper_summary._validate_research_draft (research-draft validation)
  • PaperGlobalSummary.from_draft (citation resolution)
  • PaperSemanticResult reload validator

so build-time and reload-time agree.

Verified end to end against arXiv 1706.03762v7: buildLocalKnowledgeLibrary.put_paper → summary search now completes and returns a cited global summary.

Verification performed

  • bash scripts/verify.sh — ruff format/check, basedpyright, lint-imports, pytest --cov: 491 passed, 86% coverage.
  • New tests: whitespace-variant quote accepted and non-matching quote rejected, at both the knowledge layer (tests/knowledge/test_paper.py) and the flows research validation (tests/flows/test_paper.py).
  • Manual end-to-end run of the build → persist → search path on arXiv 1706.03762v7.

🤖 Generated with Claude Code

The cited-summary step compared model quotes against chunk text with an exact substring match (`quote not in chunk.text`). PDF extraction keeps the source's hard-wrap newlines inside a chunk, so a word pair split across a printed line stays broken by a newline, while a faithful model quote renders that break as an ordinary space. Every such quote failed the check, so the semantic build raised a citation error for any real paper regardless of model — the same failure the `PaperFlow` docstring recorded for the 2026-07-24 run.

Add `quote_matches_chunk_text` in the knowledge layer as the single rule for "the quote is present in its chunk": it compares under collapsed whitespace, so a quote that only differs by whitespace runs counts as verbatim while quotes whose words or order differ are still rejected. Route all three checks through it — the research-draft validation in `flows/_paper_summary`, `PaperGlobalSummary.from_draft`, and the `PaperSemanticResult` reload validator — so build-time and reload-time agree.

Verified end to end against arXiv 1706.03762v7: build -> put_paper -> summary search now completes and returns a cited global summary.

Verification: bash scripts/verify.sh (ruff format/check, basedpyright, lint-imports, pytest --cov) — 491 passed, 86% coverage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant