bpo-42369: Fix thread safety of zipfile._SharedFile.tell #26974
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `_SharedFile` tracks its own virtual position into the file as `self._pos` and updates it after reading or seeking. `tell()` should return this position instead of calling into the underlying file object, since if multiple `_SharedFile` instances are being used concurrently on the same file, another one may have moved the real file position. Additionally, calling into the underlying `tell` may expose thread safety issues in the underlying file object because it was called without taking the lock. Prior to this fix, the test case in https://bugs.python.org/issue42369#msg381212 reliably caused a `zipfile.BadZipFile: Bad CRC-32 for file 'file1'` after a few dozen reads; with this fix I have not seen this error.
|
Nice! This could certainly help me solve the issue in machine learning when I want to load data from a zip file. |
|
This PR is stale because it has been open for 30 days with no activity. |
serhiy-storchaka
approved these changes
Mar 19, 2022
serhiy-storchaka
requested changes
Mar 19, 2022
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase And if you don't make the requested changes, you will be poked with soft cushions! |
|
I have made the requested changes; please review again (hopefully got the rst formatting right) |
|
Thanks for making the requested changes! @serhiy-storchaka: please review the changes made to this pull request. |
serhiy-storchaka
approved these changes
Mar 20, 2022
serhiy-storchaka
added
type-bug
|
Thanks @kevinmehall for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9. |
|
Thanks @kevinmehall for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10. |
|
GH-32008 is a backport of this pull request to the 3.9 branch. |
|
GH-32009 is a backport of this pull request to the 3.10 branch. |
|
Thank you @kevinmehall. |
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Mar 20, 2022
) The `_SharedFile` tracks its own virtual position into the file as `self._pos` and updates it after reading or seeking. `tell()` should return this position instead of calling into the underlying file object, since if multiple `_SharedFile` instances are being used concurrently on the same file, another one may have moved the real file position. Additionally, calling into the underlying `tell` may expose thread safety issues in the underlying file object because it was called without taking the lock. (cherry picked from commit e730ae7) Co-authored-by: Kevin Mehall <[email protected]>
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
Mar 20, 2022
) The `_SharedFile` tracks its own virtual position into the file as `self._pos` and updates it after reading or seeking. `tell()` should return this position instead of calling into the underlying file object, since if multiple `_SharedFile` instances are being used concurrently on the same file, another one may have moved the real file position. Additionally, calling into the underlying `tell` may expose thread safety issues in the underlying file object because it was called without taking the lock. (cherry picked from commit e730ae7) Co-authored-by: Kevin Mehall <[email protected]>
miss-islington
added a commit
that referenced
this pull request
Mar 20, 2022
The `_SharedFile` tracks its own virtual position into the file as `self._pos` and updates it after reading or seeking. `tell()` should return this position instead of calling into the underlying file object, since if multiple `_SharedFile` instances are being used concurrently on the same file, another one may have moved the real file position. Additionally, calling into the underlying `tell` may expose thread safety issues in the underlying file object because it was called without taking the lock. (cherry picked from commit e730ae7) Co-authored-by: Kevin Mehall <[email protected]>
miss-islington
added a commit
that referenced
this pull request
Mar 20, 2022
The `_SharedFile` tracks its own virtual position into the file as `self._pos` and updates it after reading or seeking. `tell()` should return this position instead of calling into the underlying file object, since if multiple `_SharedFile` instances are being used concurrently on the same file, another one may have moved the real file position. Additionally, calling into the underlying `tell` may expose thread safety issues in the underlying file object because it was called without taking the lock. (cherry picked from commit e730ae7) Co-authored-by: Kevin Mehall <[email protected]>
hello-adam
pushed a commit
to hello-adam/cpython
that referenced
this pull request
Jun 2, 2022
) The `_SharedFile` tracks its own virtual position into the file as `self._pos` and updates it after reading or seeking. `tell()` should return this position instead of calling into the underlying file object, since if multiple `_SharedFile` instances are being used concurrently on the same file, another one may have moved the real file position. Additionally, calling into the underlying `tell` may expose thread safety issues in the underlying file object because it was called without taking the lock. (cherry picked from commit e730ae7) Co-authored-by: Kevin Mehall <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
_SharedFiletracks its own virtual position into the file asself._posand updates it after reading or seeking.tell()should return this position instead of calling into the underlying file object, since if multiple_SharedFileinstances are being used concurrently on the same file, another one may have moved the real file position. Additionally, calling into the underlyingtellmay expose thread safety issues in the underlying file object because it was called without taking the lock.Prior to this fix, the test case in https://bugs.python.org/issue42369#msg381212 reliably caused a
zipfile.BadZipFile: Bad CRC-32 for file 'file1'in less than a second; with this fix I have not seen this error.https://bugs.python.org/issue42369