Repository navigation
Conversation
When the buffer ends inside an object, the C extension decodes the part it has and keeps it for the next feed. tell() then included those bytes, so it pointed into the middle of the next object instead of at its start. Code that saves tell() to resume a stream later restarted mid-object and decoded garbage. The pure Python implementation already returns the start.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unpacker.tell()is documented as "the starting position of the next object". With the C extension that stops being true once the buffer ends inside an object. The C unpacker decodes the bytes it has, keeps the partial object for the nextfeed(), and counts those bytes instream_offset. Sotell()points into the middle of the next object.This breaks the usual way to resume a stream: read what is there, iterate, save
tell(), and continue from that offset later. A minimal case ispackb([1, 2, 3]) + packb(["x" * 40, "y" * 40]), a 4-byte object followed by an 85-byte one, with only the first 49 bytes available:Starting a new
Unpackerat offset 49 decodes forty121integers from the middle of"y" * 40, with no error. The same happens withfeed(). The pure Python implementation rolls back to the start of the incomplete object, so it returns 4.The C unpacker now records
stream_offsetwhen it starts a new object, andtell()returns that value whilectx.executeshows an incomplete object. Nothing changes once the object is complete or after an error, andread_bytes()already refuses to run while an object is incomplete. The pure Python implementation is unchanged.test_unpack_tell_incomplete_objectintest/test_sequnpack.pycuts the stream at three points inside the second object, for bothfeed()and a file-like object. It checks thattell()is 4 after the first object and 89 once the rest has arrived. With only thetell()change reverted, it fails on the C extension withassert 5 == 4, and it passes on both implementations with the change. The full suite passes with the C extension (179 passed, 1 skipped) and withMSGPACK_PUREPYTHON=1(176 passed, 4 skipped), andruff checkandruff formatreport no changes. I ran this on CPython 3.13 on Linux only. I did not run the free-threaded build or the parallel-thread tests.