Skip to content

gh-89739: gh-77140: Support zip64 in zipimport #94146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Mar 28, 2024
Merged

gh-89739: gh-77140: Support zip64 in zipimport #94146

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a836182
Support zip64 in zipimport
thatch Jun 22, 2022
8309b45
Merge branch 'main' into zip64-zipimport
itamaro Jul 8, 2023
f719c63
Use helper method to get files for zip64
itamaro Jul 8, 2023
19d8bfa
Update versionchanged to 3.13
itamaro Jul 8, 2023
ea8c9c7
gps review feedback
itamaro Jul 8, 2023
e82d9da
Merge branch 'main' into zip64-zipimport
itamaro Mar 6, 2024
dc74fc0
Add ``_unpack_uint64`` to ``_bootstrap_external``
itamaro Mar 6, 2024
aa2c416
Merge branch 'main' into zip64-zipimport
gpshead Mar 28, 2024
9089cb1
Add a what's new entry.
gpshead Mar 28, 2024
ed47eb8
--trailing space
gpshead Mar 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
gps review feedback
  • Loading branch information
@itamaro
itamaro committed Jul 8, 2023
commit ea8c9c7487cf7314617cc53e89d862a6010e83bd
46 changes: 22 additions & 24 deletions Lib/zipimport.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -368,25 +368,28 @@ def _read_directory(archive):
file_size = fp.tell()
except OSError:
raise ZipImportError(f"can't read Zip file: {archive!r}",
path=archive)
max_comment_start = max(file_size - MAX_COMMENT_LEN -
END_CENTRAL_DIR_SIZE - END_CENTRAL_DIR_SIZE_64 -
END_CENTRAL_DIR_LOCATOR_SIZE_64, 0)
path=archive)
max_comment_plus_dirs_size = (
MAX_COMMENT_LEN + END_CENTRAL_DIR_SIZE +
END_CENTRAL_DIR_SIZE_64 + END_CENTRAL_DIR_LOCATOR_SIZE_64)
max_comment_start = max(file_size - max_comment_plus_dirs_size, 0)
try:
fp.seek(max_comment_start)
data = fp.read()
data = fp.read(max_comment_plus_dirs_size)
except OSError:
raise ZipImportError(f"can't read Zip file: {archive!r}",
path=archive)
path=archive)
pos = data.rfind(STRING_END_ARCHIVE)
pos64 = data.rfind(STRING_END_ZIP_64)

if (pos64 >= 0 and pos64+END_CENTRAL_DIR_SIZE_64+END_CENTRAL_DIR_LOCATOR_SIZE_64==pos):
# Zip64 at "correct" offset from standard EOCD
buffer = data[pos64:pos64 + END_CENTRAL_DIR_SIZE_64]
if len(buffer) != END_CENTRAL_DIR_SIZE_64:
raise ZipImportError(f"corrupt Zip64 file: {archive!r}",
path=archive)
raise ZipImportError(
f"corrupt Zip64 file: Expected {END_CENTRAL_DIR_SIZE_64} byte "
f"zip64 central directory, but read {len(buffer)} bytes.",
path=archive)
header_position = file_size - len(data) + pos64

central_directory_size = int.from_bytes(buffer[40:48], 'little')
Expand All @@ -396,7 +399,7 @@ def _read_directory(archive):
buffer = data[pos:pos+END_CENTRAL_DIR_SIZE]
if len(buffer) != END_CENTRAL_DIR_SIZE:
raise ZipImportError(f"corrupt Zip file: {archive!r}",
path=archive)
path=archive)

header_position = file_size - len(data) + pos

Expand All @@ -410,7 +413,7 @@ def _read_directory(archive):
# you need to adjust position by 76 for arc to be 0.
else:
raise ZipImportError(f'not a Zip file: {archive!r}',
path=archive)
path=archive)

# Buffer now contains a valid EOCD, and header_position gives the
# starting position of it.
Expand Down Expand Up @@ -473,7 +476,7 @@ def _read_directory(archive):
# internal buffers. See issue #8745.
try:
extra_data_len = header_size - name_size
extra_data = fp.read(extra_data_len)
extra_data = memoryview(fp.read(extra_data_len))

if len(extra_data) != extra_data_len:
raise ZipImportError(f"can't read Zip file: {archive!r}", path=archive)
Expand Down Expand Up @@ -511,27 +514,22 @@ def _read_directory(archive):
if tag == ZIP64_EXTRA_TAG:
if (len(extra_data) - 4) % 8 != 0:
raise ZipImportError(f"can't read header extra: {archive!r}", path=archive)
values = [
int.from_bytes(extra_data[i:i+8], 'little')
for i in range(4, len(extra_data), 8)
]
num_extra_values = (len(extra_data) - 4) // 8
if num_extra_values > 3:
raise ZipImportError(f"can't read header extra: {archive!r}", path=archive)
values = struct.unpack_from(f"<{min(num_extra_values, 3)}Q",
Copy link
Contributor

@jsirois jsirois Apr 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does this symbol come from? I see no struct module import in this file. My Pex too-big-zip tests also find the following under 3.13.0a6:

Failed checking if argv[0] is an import path entry
Traceback (most recent call last):
  File "<frozen zipimport>", line 98, in __init__
  File "<frozen zipimport>", line 520, in _read_directory
NameError: name 'struct' is not defined. Did you forget to import 'struct'?
  File "/tmp/pytest-of-jsirois/pytest-10/test_check0/too-big.pyz", line 1
    PK-
SyntaxError: source code cannot contain null bytes

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A quick experiment seems to indicate just adding the import gets things working. I'll try my hand at sending up a patch.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch! too bad the tests didn't catch it. let me know if you need help with getting that PR going.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'm good, thank you. Filed #118107 to reference in the forthcoming PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix here hopefully: #118108

extra_data, offset=4)

# N.b. Here be dragons: the ordering of these is different than
# the header fields, and it's really easy to get it wrong since
# naturally-occuring zips that use all 3 are >4GB and not
# something that would be checked-in.
# The tests include a binary-edited zip that uses zip64
# (unnecessarily) for all three.
# naturally-occuring zips that use all 3 are >4GB
if file_size == MAX_UINT32:
file_size = values.pop(0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These were problematic as well: values is a tuple; not a list. The new test caught this issue.

if data_size == MAX_UINT32:
data_size = values.pop(0)
if file_offset == MAX_UINT32:
file_offset = values.pop(0)

if values:
raise ZipImportError(f"can't read header extra: {archive!r}", path=archive)

break

# For a typical zip, this bytes-slicing only happens 2-3 times, on
Expand All @@ -542,8 +540,8 @@ def _read_directory(archive):
"zipimport: suspected zip64 but no zip64 extra for {!r}",
path,
)
# XXX These two statements seem swapped because `header_offset` is a
# position within the actual file, but `file_offset` (when compared) is
# XXX These two statements seem swapped because `central_directory_position`
# is a position within the actual file, but `file_offset` (when compared) is
# as encoded in the entry, not adjusted for this file.
# N.b. this must be after we've potentially read the zip64 extra which can
# change `file_offset`.
Expand Down
2 changes: 1 addition & 1 deletion Misc/NEWS.d/next/Library/2022-06-22-14-45-32.gh-issue-89739.CqZcRL.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
The ``zipimport`` module can now read ZIP64 files.
The :mod:`zipimport` module can now read ZIP64 files.
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy