Skip to content

gh-148286: Fix undefined behaviour in io.StringIO.read() after an overseek - #154913

Open
matthiasgoergens wants to merge 4 commits into
python:mainfrom
matthiasgoergens:stringio-overseek-ub
Open

gh-148286: Fix undefined behaviour in io.StringIO.read() after an overseek#154913
matthiasgoergens wants to merge 4 commits into
python:mainfrom
matthiasgoergens:stringio-overseek-ub

Conversation

@matthiasgoergens

@matthiasgoergens matthiasgoergens commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

_io_StringIO_read_impl() clamps the read size to zero when the position is past the end of the buffer, but still forms the pointer self->buf + self->pos before returning. Nothing is read through it, but forming it is undefined behaviour on its own, and since self->buf is a Py_UCS4 * the byte offset is self->pos * 4 — so for self->pos == 2**62 - 1 the multiplication wraps and yields a pointer four bytes below self->buf.

io.StringIO("abc").seek(2**62 - 1)   # then .read()
stringio.c:352: addition of unsigned offset to 0x... overflowed to 0x...

_stringio_readline() already guards exactly this condition a few lines below, with the comment "In case of overseek, return the empty string", so this applies the same guard on the read path. self->pos += size is a no-op here because size is zero, and ENSURE_REALIZED() is deliberately left ahead of the guard so the state transition and its error path are unchanged.

The existing test_io suite already covers this path — that is how UBSan found it — so no new test is needed. With the Modules/_io/stringio.c entry removed from Tools/ubsan/suppressions.txt, test_io passes under UBSAN_OPTIONS=halt_on_error=1, where before it aborts at stringio.c:352.

…an overseek

_io_StringIO_read_impl() clamps the read size to zero when the position is
past the end of the buffer, but still formed the pointer

    output = self->buf + self->pos;

before returning the empty string. Nothing is read through that pointer,
but forming it is undefined behaviour on its own, and because self->buf is
a Py_UCS4 *, the byte offset is self->pos * 4: for self->pos == 2**62 - 1
the multiplication wraps and yields a pointer four bytes BELOW self->buf.
That is what UBSan reported:

    io.StringIO("abc").seek(2**62 - 1) followed by .read()
    stringio.c:352: addition of unsigned offset to 0x... overflowed to 0x...

Return the empty string early, exactly as _stringio_readline() already
does for the same overseek condition. self->pos += size is a no-op here
since size is zero, and ENSURE_REALIZED() is kept ahead of the guard so
the state transition and its error path are unchanged.

The existing test_io suite already covers this path, which is how UBSan
found it. The Modules/_io/stringio.c entry in Tools/ubsan/suppressions.txt
is no longer needed; with it removed, test_io passes under
UBSAN_OPTIONS=halt_on_error=1, where before it aborts at stringio.c:352.
@@ -0,0 +1,7 @@
Fix undefined behaviour in :meth:`!io.StringIO.read` after an overseek.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same idea as with the ctypes PR, avoid implementation/infra details.

@picnixz picnixz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a regression test.

@bedevere-app

bedevere-app Bot commented Jul 31, 2026

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

Review feedback: picnixz asked for a regression test, StanFromIreland
for the NEWS entry to drop implementation and infrastructure detail.

The existing test_overseek only seeks one past the end, which is why it
never exercised this: the pointer arithmetic only wraps for positions
above PY_SSIZE_T_MAX // 4. The new test covers 2**31, 2**62 and
sys.maxsize, and sits in MemoryTestMixin so it runs for BytesIO as well
as StringIO, in both the C and pure-Python implementations. Verified
that all four already satisfy it, which is the point -- it pins the
documented behaviour rather than detecting the UB.

NEWS entry cut to the user-visible statement.
@matthiasgoergens

Copy link
Copy Markdown
Contributor Author

Thanks both.

Regression test — added in 0d659fa, as test_overseek_far_past_end.

One thing worth being explicit about, because it affects how much the
test is worth: it passes on unfixed builds too. I checked before
writing it — io.StringIO and _pyio.StringIO, and BytesIO for both,
already return '' with the correct tell() and getvalue() at every
offset it tries. The bug is undefined behaviour, and UB is not generally
observable; a normal build gives no signal here.

So the test pins the documented behaviour rather than detecting the
defect. It is still worth having: the existing test_overseek only
seeks one past the end, so nothing covered the large-offset case at all,
and that is precisely where the arithmetic wraps — the pointer overflow
only occurs above PY_SSIZE_T_MAX // 4. It is in MemoryTestMixin, so
it runs for BytesIO as well as StringIO, C and pure-Python.

What actually guards the fix is the other half of the diff: removing

pointer-overflow:Modules/_io/stringio.c

from Tools/ubsan/suppressions.txt, which .github/workflows/reusable-san.yml
consumes. With the suppression gone, the UBSan job fails if the UB comes
back. That is the regression guard; the unit test is the behavioural
one. Happy to add more of the latter if you would like, but I did not
want to imply it catches what it does not.

NEWS entry — trimmed to the user-visible statement, no macro names,
no description of the fix, no suppressions note.

I have made the requested changes; please review again

@bedevere-app

bedevere-app Bot commented Aug 1, 2026

Copy link
Copy Markdown

Thanks for making the requested changes!

@picnixz: please review the changes made to this pull request.

@bedevere-app
bedevere-app Bot requested a review from picnixz August 1, 2026 09:10
Verified under UBSan with CI's own configuration
(--with-undefined-behavior-sanitizer --with-strict-overflow, clang):
CStringIOTest.testSeek already triggers the report, because it does
seek(sys.maxsize) followed by read(4), which is exactly the case where
the pointer arithmetic wraps. That is what produced the diagnostic
recorded in the suppressions file.

My added test was therefore redundant for detection: small overseeks
(11, 100, 2**31) produce no report at all, and the large offsets it used
were already covered. Removed rather than kept as noise.

Left a comment on testSeek instead, since its name gives no hint that
those two lines guard a UB fix, and trimming them would silently remove
the coverage.
@matthiasgoergens

Copy link
Copy Markdown
Contributor Author

Correction to my last comment, and a retraction: the test I added was
redundant and I have removed it
(9d3ba83).

I said the suppression removal was the regression guard and the unit
test the behavioural one. The first half is right; the second oversold
it. CStringIOTest.testSeek already covers this — it has all along:

self.assertEqual(sys.maxsize, bytesIo.seek(sys.maxsize))
self.assertEqual(self.EOF, bytesIo.read(4))

That is exactly the case where the arithmetic wraps, and it is what
produced the diagnostic recorded in Tools/ubsan/suppressions.txt.

Verified rather than argued, building with CI's own configuration
(--with-undefined-behavior-sanitizer --with-strict-overflow, clang 22):

UBSan report
unfixed, testSeek firesaddition of unsigned offset to 0x…a60 overflowed to 0x…a5c
unfixed, overseek by 11 / 100 / 2³¹ none
unfixed, offsets ≥ 2⁶²−1 fires
fixed, full test_io none; all 9 tests OK

So small overseeks produce no report at all, which is why the existing
test_overseek never surfaced this, and why my large-offset test added
no coverage that testSeek did not already provide. I have left a
comment on those two lines instead — the test's name gives no hint that
they guard a UB fix, and trimming them would silently remove the
coverage.

@picnixz — if you would still prefer an explicitly-named regression
test even though it duplicates existing coverage, say so and I will add
one back; I did not want to leave a test in that could not fail.


Unrelated but worth flagging, since it cost me a while and may cost
others the same.
-fno-strict-overflow silently disables clang's
pointer-overflow sanitizer, and re-adding -fsanitize=pointer-overflow
afterwards does not bring it back:

-fsanitize=undefined                                          -> reports
-fsanitize=undefined -fno-strict-overflow                     -> silent
-fsanitize=undefined -fno-strict-overflow
                     -fsanitize=pointer-overflow              -> silent

--with-strict-overflow defaults to no, so an ordinary
./configure --with-undefined-behavior-sanitizer build cannot see any
pointer-overflow: finding. The CI job is fine — reusable-san.yml
passes --with-strict-overflow explicitly — but anyone reproducing a
suppressions-file entry locally with the obvious configure line will get
silence and reasonably conclude the entry is stale. I did, for several
builds, before spotting it.

@StanFromIreland this may be worth a line in Tools/ubsan/README or
next to the pointer-overflow entries, given you own that file. Happy
to send that as a separate PR if useful.

Dropping the '!' from :meth:`!io.StringIO.read` while trimming the entry
broke the docs build: there is no doc target for io.StringIO.read, so
nitpicky mode reports 'py:meth reference target not found'. The original
had the bang for exactly that reason.

Every build check passed; Docs was the only real failure of the three
(the other two are the unresolved-review gate and the aggregate that
depends on it).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants