gh-90533: Implement BytesIO.peek() by marcelm · Pull Request #30808 · python/cpython

marcelm · 2022-01-22T22:08:24Z

bedevere-bot · 2022-08-05T22:44:49Z

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

marcelm · 2022-11-09T13:16:15Z

@AlexWaygood You’ve been the only human to interact with this PR so far, do you possibly have any advice on how to move this forward?

AlexWaygood · 2022-11-09T13:36:44Z

Hi @marcelm — sorry for the delay in anybody looking at this. I haven't studied your PR in detail (or thought about whether the proposal is a good idea), but it looks well put together at first glance.

I'll try to take a look soon. I won't be able to review the C code, but I can comment on whether the proposal seems like a good idea, and I can review the Python implementation and the tests.

Note that if this proposal is accepted, it will also need:

Updates to the docs for BytesIO (Doc/library/io.rst)
An entry in "What's new in Python 3.12" (Doc/whatsnew/3.12.rst)

You can also add yourself to Misc/ACKS as part of this PR, and give yourself credit in the NEWS entry, if you like. (Neither is compulsory.)

marcelm · 2022-11-09T21:29:39Z

Thank you! I pushed a documentation update and will add an entry to the What’s new document in case the PR is reviewed favorably.

AlexWaygood · 2022-11-28T12:25:57Z

I'll defer to @benjaminp's judgement on this one -- I'm really not the right reviewer for this, unfortunately :(

Please ping me again if you still haven't had a review in a few weeks.

marcelm · 2022-11-28T12:43:35Z

Thank you! I appreciate that you took the time.

awalgarg · 2023-07-08T09:38:15Z

Please ping me again if you still haven't had a review in a few weeks.

@AlexWaygood Hi! Been 8 months, no review so far :) How can we take this forward? I'm not the original author of the PR but I also have a use-case for this API and would like to see this change land, happy to help progress this.

marcelm · 2023-07-08T13:14:48Z

Thanks for your interest! I have updated the PR to fix the merge conflict and to reflect that it now needs to target 3.13.

awalgarg · 2023-07-08T13:23:04Z

@marcelm That was quick, thank you so much!

AA-Turner · 2023-09-21T03:52:35Z

cc: @erlend-aasland @vstinner @benjaminp for the io C code.

A

vstinner · 2023-09-21T11:45:16Z

+      .. versionadded:: 3.13
+
+      Return bytes from the current position onwards but without advancing the
+      position. The number of bytes returned may be less or more than requested.


Wait. Why can it more than requested?

Apparently size=0 means "read all". Please document it.

This was just copied from the documentation for BufferedReader.peek(), but I agree that it could be written better. Changed now.

Wait. Why can it more than requested?

BufferedReader.peek() ignores the size argument and just returns whatever it has in its internal buffer.

I would prefer that BytesIO.peek() documents its behavior, rather than BufferedReader.peek() behavior.

Absolutely, it’s already changed.

(BTW, is there a policy on who clicks the "Resolve conversation" button?)

vstinner · 2023-09-21T11:47:32Z

+        pos = self.tell()
+        if size == 0:
+            size = -1
+        b = self.read(size)


Would it be possible to implement it without touching the position? This code is not thread safe. I don't know if it's supposed to be thread-safe. Maybe add a private read method which has an argument to decide to move the position or not.

Same remark for C code.

Good point; I implemented this now so that the position is not changed. I factored out a peek_bytes function in the C version that does not advance the position.

It seems though that neither the C nor the Python version of BytesIO are supposed to be thread safe. (They don’t use locks as e.g. BufferedReader does.) So I would suggest making them so would be a task for a different PR.

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

vstinner

I asked BytesIO.peek(0) to return an empty string, but I didn't notice that BufferedReader.peek(0) returns non-empty string. Now I'm unhappy that the two classes have an inconsistent behavior on peek(0), sorry.

>>> f=open("/etc/issue", "rb")
>>> f.peek(0)
b'\\S\nKernel \\r on \\m (\\l)\n\n'

On a short file, BufferedReader.peek(0) even returned the whole file contents!

Since BufferedReader.peek(0) behavior already exists, I would suggest to use the same behavior on BytesIO.peek(): return non-empty string.

cc @cmaloney

Co-authored-by: Victor Stinner <vstinner@python.org>

marcelm · 2026-04-13T12:55:05Z

Since BufferedReader.peek(0) behavior already exists, I would suggest to use the same behavior on BytesIO.peek(): return non-empty string.

Sure! I have changed it so that the rest of the buffer is returned for size=0.

The signature is still such that the default is size=1. Maybe that is actually a good compromise?

If desired, an explicit peek(0) returns a full (possibly expensive) copy - but the default is cheap.
peek() without arguments returns a single byte, which is the minimum necessary for compatibility with BufferedReader.peek(). Note that the latter also can return a single byte when that is how much is left in the buffer:

$ python3 -c 'f=open("README.rst", "rb");f.read(8191);print(f.peek())'
b'r'

If more than a single byte but not the full buffer is needed, an explicit size can be set.

Alternatively, the default could become size=0, but then the question is how huch of the buffer to return (which is where this PR got stalled last time).

marcelm · 2026-04-15T10:23:18Z

I have looked into the test failures on free-threaded Python but don’t know what is going on.

The _Py_CRITICAL_SECTION_ASSERT_OBJECT_LOCKED(self) assertion in peek_bytes_lock_held fails. That function was factored out from read_bytes_lock_held.

vstinner · 2026-04-15T13:28:30Z

I have looked into the test failures on free-threaded Python but don’t know what is going on.

You must decorate _io.BytesIO.peek with @critical_section:

diff --git a/Modules/_io/bytesio.c b/Modules/_io/bytesio.c
index f43cf1f7e41..98f31f3d1a8 100644
--- a/Modules/_io/bytesio.c
+++ b/Modules/_io/bytesio.c
@@ -511,6 +511,7 @@ _io_BytesIO_read1_impl(bytesio *self, Py_ssize_t size)
 
 
 /*[clinic input]
+@critical_section
 _io.BytesIO.peek
     size: Py_ssize_t = 1
     /

marcelm · 2026-04-15T20:28:44Z

You must decorate _io.BytesIO.peek with @critical_section:

Thanks, that worked!

marcelm · 2026-04-21T07:07:07Z

Is anything else needed? As far as I understand, this PR would need to be merged before beta 1 in two weeks in order to get this into 3.15.

cmaloney · 2026-04-21T07:24:24Z

+      Return bytes from the current position onwards without advancing the position.
+      At least one byte of data is returned if not at EOF.
+      Return an empty :class:`bytes` object at EOF.
+      If the size argument is less than one or larger than the number of available bytes,


nitpick: I don't think the comma at the end of this end of this line is right / needed?

I’m quite convinced it is needed because the first part of the sentence is an "introductory clause or phrase". (But if you swap the order and write "A copy of the buffer ... is returned if ...", no comma is needed before the "if".)

cmaloney · 2026-04-21T07:26:59Z

+    def peek(self, size=1):
+        if self.closed:
+            raise ValueError("peek on closed file")
+        if size < 1:


I don't think behavior here is going to be correct if a negative size is passed in

I have the same instinct whenever I read this line, but note that self._buffer[self._pos:] doesn’t use the size. Or did you mean something else?

cmaloney · 2026-04-21T07:29:18Z

+            self.assertEqual(memio.peek(-1), abc)
+            self.assertEqual(memio.peek(len(abc) + 100), abc)
+            self.assertEqual(memio.tell(), len(buf))
+


could you test that seek to end of file then peek() returns self.EOF?

(Think it's right in implementation, just "no data was read because whole buffer skipped" can catch different cases from "no data was read but buffer has been touched")

Added within a new with self.ioclass(buf) as memio:, not sure if that is what you meant.

cmaloney · 2026-04-21T07:32:56Z


      Return :class:`bytes` containing the entire contents of the buffer.

+   .. method:: peek(size=1, /)


Why not default to size=0 like BufferedReader?

Read through the comments. In general anything returning a bytes rather than a bytes-like memoryview is going to require allocating and copying potentially many bytes... If the copy is really concerning I'd lean returning a memoryview rather than a bytes which is a mandatory copy.

memoryview came up in this comment and the three following it: #30808 (comment)

I don’t know what the conclusion is given your comment. Should a memoryview be returned instead? Most important to me is compatibility with what BufferedReader.peek() returns.

I am not too concerned about the extra memory for a bytes object as long as the default is size=1.

cmaloney · 2026-04-21T07:36:45Z

Also: Please extend the free threading test in Lib/test/test_free_threading/test_io.py to cover the new method

cmaloney · 2026-04-21T07:39:55Z

Overall looking good, definitely think this is in good shape. Just some weird corners (I've spent too much time reading I/O code...)

marcelm

Thanks for the comments! I tried to address them.

Also: Please extend the free threading test in Lib/test/test_free_threading/test_io.py to cover the new method

I have added something there, but wasn’t quite sure what I was doing; can you please have a look?

marcelm · 2026-04-21T09:18:38Z


      Return :class:`bytes` containing the entire contents of the buffer.

+   .. method:: peek(size=1, /)


memoryview came up in this comment and the three following it: #30808 (comment)

I don’t know what the conclusion is given your comment. Should a memoryview be returned instead? Most important to me is compatibility with what BufferedReader.peek() returns.

I am not too concerned about the extra memory for a bytes object as long as the default is size=1.

marcelm · 2026-04-21T09:18:42Z

+      Return bytes from the current position onwards without advancing the position.
+      At least one byte of data is returned if not at EOF.
+      Return an empty :class:`bytes` object at EOF.
+      If the size argument is less than one or larger than the number of available bytes,


I’m quite convinced it is needed because the first part of the sentence is an "introductory clause or phrase". (But if you swap the order and write "A copy of the buffer ... is returned if ...", no comma is needed before the "if".)

marcelm · 2026-04-21T09:19:24Z

+            self.assertEqual(memio.peek(-1), abc)
+            self.assertEqual(memio.peek(len(abc) + 100), abc)
+            self.assertEqual(memio.tell(), len(buf))
+


Added within a new with self.ioclass(buf) as memio:, not sure if that is what you meant.

marcelm · 2026-04-21T09:19:41Z

+    def peek(self, size=1):
+        if self.closed:
+            raise ValueError("peek on closed file")
+        if size < 1:


I have the same instinct whenever I read this line, but note that self._buffer[self._pos:] doesn’t use the size. Or did you mean something else?

the-knights-who-say-ni added the CLA not signed label Jan 22, 2022

This comment was marked as outdated.

Sign in to view

bedevere-bot added the awaiting review label Jan 22, 2022

the-knights-who-say-ni added CLA signed and removed CLA not signed labels Feb 1, 2022

marcelm force-pushed the fix-issue-46375 branch from 3111318 to 675718d Compare April 11, 2022 20:06

marcelm changed the title ~~bpo-46375: Implement BytesIO.peek()~~ gh-90533: Implement BytesIO.peek() Apr 11, 2022

This comment was marked as outdated.

Sign in to view

AlexWaygood added the topic-IO label Apr 14, 2022

ezio-melotti removed the CLA signed label Jul 13, 2022

marcelm force-pushed the fix-issue-46375 branch from 675718d to d4e849f Compare August 5, 2022 22:44

marcelm force-pushed the fix-issue-46375 branch from ef5eb69 to b82fa85 Compare August 5, 2022 23:28

marcelm force-pushed the fix-issue-46375 branch 2 times, most recently from 6998be1 to be39ff2 Compare November 6, 2022 15:11

AlexWaygood added the type-feature A feature request or enhancement label Nov 9, 2022

AlexWaygood self-requested a review November 9, 2022 13:37

kumaraditya303 requested a review from benjaminp November 27, 2022 07:18

AlexWaygood removed their request for review November 28, 2022 12:26

marcelm force-pushed the fix-issue-46375 branch from 5cb3143 to 79d5032 Compare September 19, 2023 08:04

vstinner reviewed Sep 21, 2023

View reviewed changes

marcelm and others added 12 commits April 10, 2026 23:22

Document why size < 0 can happen

3661b65

Update Modules/_io/bytesio.c

cd40d77

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

Do not update pos if peek_bytes failed

04372bd

Size can be negative after truncate or seek

6b9ae8c

Test with size<0 and size>len(buf)

f7406f6

Test peek() after write()

d9528e2

Document BufferedReader.peek and BytesIO.peek similarly

bc8134b

Comment

b6ffca8

Make it more explicit that size is ignored

5fe5645

Return an empty bytes object for size=0

4126a64

Simplify

1ea40c2

Test peek(3) and peek(5)

77e04d6

marcelm force-pushed the fix-issue-46375 branch from 611bdaa to 77e04d6 Compare April 10, 2026 21:37

marcelm requested a review from AA-Turner as a code owner April 10, 2026 21:37

Run clinic.py

4d2f2dd

vstinner reviewed Apr 11, 2026

View reviewed changes

Comment thread Modules/_io/bytesio.c

Comment thread Modules/_io/bytesio.c Outdated

Comment thread Lib/test/test_io/test_memoryio.py Outdated

marcelm and others added 2 commits April 13, 2026 13:46

Apply suggestions from code review

c16bebf

Co-authored-by: Victor Stinner <vstinner@python.org>

Do not return an empty bytes object for size=0

08bd7da

Decorate BytesIO.peek with @critical_section

6174fca

cmaloney reviewed Apr 21, 2026

View reviewed changes

marcelm added 2 commits April 21, 2026 11:09

Test peek returns EOF after seeking to EOF

b8b8cf4

Free-threading test for peek

7ac914e

marcelm commented Apr 21, 2026

View reviewed changes


		Return :class:`bytes` containing the entire contents of the buffer.

		.. method:: peek(size=1, /)

Uh oh!

Conversation

marcelm commented Jan 22, 2022 • edited by AlexWaygood Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

bedevere-bot commented Aug 5, 2022

Uh oh!

marcelm commented Nov 9, 2022

Uh oh!

AlexWaygood commented Nov 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcelm commented Nov 9, 2022

Uh oh!

AlexWaygood commented Nov 28, 2022

Uh oh!

marcelm commented Nov 28, 2022

Uh oh!

awalgarg commented Jul 8, 2023

Uh oh!

marcelm commented Jul 8, 2023

Uh oh!

awalgarg commented Jul 8, 2023

Uh oh!

AA-Turner commented Sep 21, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marcelm commented Apr 13, 2026

Uh oh!

marcelm commented Apr 15, 2026

Uh oh!

vstinner commented Apr 15, 2026

Uh oh!

marcelm commented Apr 15, 2026

Uh oh!

marcelm commented Apr 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmaloney commented Apr 21, 2026

Uh oh!

cmaloney commented Apr 21, 2026

Uh oh!

marcelm left a comment

marcelm commented Jan 22, 2022 •

edited by AlexWaygood

Loading

AlexWaygood commented Nov 9, 2022 •

edited

Loading