Skip to content

Fix CSV sniffing samples truncated in the middle of a line - #140

Open
Voyagerroc-Lab wants to merge 1 commit into
Textualize:mainfrom
Voyagerroc-Lab:fix/csv-sniffer-complete-lines
Open

Fix CSV sniffing samples truncated in the middle of a line#140
Voyagerroc-Lab wants to merge 1 commit into
Textualize:mainfrom
Voyagerroc-Lab:fix/csv-sniffer-complete-lines

Conversation

@Voyagerroc-Lab

Copy link
Copy Markdown

Piping a valid TSV with long rows into rich - --csv can exit with Could not determine delimiter: the 1,024-character sniffing sample ends partway through a row. This also affects comma, semicolon, and pipe delimiters.

Sample the first 100 complete physical lines for both delimiter and header detection. This bounds the number of sampled lines while allowing fields longer than 1,024 characters. Rendering still consumes the full input, including rows beyond the sample and the existing head/tail selection.

Related to #54. This addresses truncated sampling; csv.Sniffer remains heuristic. #89 provides a separate manual dialect override and retains the truncated sample in its automatic path. The extension fallback from #39 remains covered by tests.

Validation on Windows, Python 3.11.9, in a fresh virtual environment (pip install -e . pytest click==8.1.8 black==22.3.0; Click 8.1 retains the FC alias used by the existing test):

  • Before the fix: new tests had 13 failures, 2 passes, with delimiter-detection errors; the existing test passed separately.
  • After the fix: python -m pytest -q16 passed.
  • Covers all four supported delimiters, 300- and 1,500-character fields, stdin with LF/CRLF, output beyond the sample, missing final newline, head/tail, and CSV/TSV extension fallback.
  • Black 22.3.0 checks on both changed files, pip check, and git diff --check pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant