gh-115988: Add missing ARM64 and RISCV filter in lzma module #115989

ivq · 2024-02-27T13:17:46Z

Example:

import lzma

filter = [
    { "id": lzma.FILTER_RISCV },
    { "id": lzma.FILTER_LZMA2 },
]

in_fn = "in.bin"
out_fn = "out.bin"

with open(in_fn, mode="rb") as f:
    in_data = f.read()

with lzma.open(out_fn, mode="wb", filters=filter) as f:
    f.write(in_data)

📚 Documentation preview 📚: https://cpython-previews--115989.org.readthedocs.build/

ghost · 2024-02-27T13:17:49Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2024-02-27T13:17:52Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

terryjreedy

Does test_lzma test the existing constants in any way? Such as for validity of the spelling? If so, the new ones should be added.

Someone more familiar with lzma should do a commit review.

Doc/library/lzma.rst

Modules/_lzmamodule.c

bedevere-app · 2024-02-27T14:19:24Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

bedevere-app · 2024-02-28T04:49:19Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

ivq · 2024-02-28T04:50:01Z

I have made the requested changes; please review again.

terryjreedy · 2024-03-12T16:28:47Z

I cannot do a commit review of this as it is out of my expertise. I requested reviews from two coredevs who have edited the file most recently.

serhiy-storchaka

They are pretty new features (https://xz.tukaani.org/format/xz-file-format-1.2.0.txt):

        1.2.0     2024-01-19    Added RISC-V filter and updated URLs in
                                Sections 0.2 and 7. The URL of this
                                specification was changed.

        1.1.0     2022-12-11    Added ARM64 filter and clarified 32-bit
                                ARM endianness in Section 5.3.2,
                                language improvements in Section 5.4

Adding support for them is a new feature in Python, so you need to add versionadded or versionchanged directives in the module documentation and add the corresponding entry in the What's New file.

You should also specify that the new filters (work? are supported? are available?) only with the specified XZ (library? file format?) version. Please add tests for new filters. They should produce expected result or raise an expected exception depending on the xz version.

I am not sure whether these constants should be defined if the underlying library does not support them. The new tests should show what is better.

bedevere-app · 2024-03-17T14:06:25Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

ivq · 2024-03-17T14:20:56Z

They are pretty new features (xz.tukaani.org/format/xz-file-format-1.2.0.txt):
        1.2.0     2024-01-19    Added RISC-V filter and updated URLs in
                                Sections 0.2 and 7. The URL of this
                                specification was changed.

        1.1.0     2022-12-11    Added ARM64 filter and clarified 32-bit
                                ARM endianness in Section 5.3.2,
                                language improvements in Section 5.4
Adding support for them is a new feature in Python, so you need to add versionadded or versionchanged directives in the module documentation and add the corresponding entry in the What's New file.

You should also specify that the new filters (work? are supported? are available?) only with the specified XZ (library? file format?) version. Please add tests for new filters. They should produce expected result or raise an expected exception depending on the xz version.

I am not sure whether these constants should be defined if the underlying library does not support them. The new tests should show what is better.

OK. Done. Correct my if anything is missing.

serhiy-storchaka

LGTM. You only need to add an entry in the What's New document.

Doc/library/lzma.rst

Modules/_lzmamodule.c

Lib/test/test_lzma.py

gpshead · 2024-03-30T18:00:36Z

I think we should lock the PR right now until things get clear and maybe after the xz code base is audited.

this PR itself seems fine and is the kind of thing we want, though the fate of the upstream code is certainly unknown at the moment it isn't related to this specific issue. (I removed the added binary test data)

@serhiy-storchaka - I'm glad we agree on what needs doing in this PR! I hope I'm not stepping on your toes by going ahead and doing some of that today. =)

We make lzma_version be the runtime version as that is the one most interestin to users. The compile time version remains exposed as the header version.

gpshead · 2024-03-30T19:14:41Z

Alright, I believe this is ready. I'll let @serhiy-storchaka do the final review and merge.

gpshead · 2024-03-31T19:12:05Z

exposing the decimal number from the C library probably isn't right, I like Serhiy's larger issue. These can be structured (tuple). The code to do so won't be much different thanks to Py_BuildValue APIs.

gpshead · 2024-03-31T19:12:33Z

and we can elide the LZMA_ prefix on the names and not include them in __all__. They're in the lzma module after all.

StanFromIreland

This has conflicts, and the version has to be changed to next as this is not making it to 3.13 :-(

Signed-off-by: Chien Wong <m@xv97.com>

ivq · 2025-07-13T09:33:15Z

Merged latest main branch and adjusted available version number. Please review.

Signed-off-by: Chien Wong <m@xv97.com>

furkanonder · 2025-07-13T17:48:55Z

!buildbot onder-riscv64

bedevere-bot · 2025-07-13T17:48:58Z

The regex 'onder-riscv64' did not match any buildbot builder. Is the requested builder in the list of stable builders?

furkanonder · 2025-07-13T17:49:20Z

!buildbot riscv64

bedevere-bot · 2025-07-13T17:49:22Z

🤖 New build scheduled with the buildbot fleet by @furkanonder for commit 1a17fd5 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F115989%2Fmerge

The command will test the builders whose names match following regular expression: riscv64

The builders matched are:

riscv64 Ubuntu23 PR

Add missing ARM64 and RISCV filter in lzma module

68979bc

bedevere-app bot added the awaiting review label Feb 27, 2024

bedevere-app bot mentioned this pull request Feb 27, 2024

ARM64 and RISCV filter is missing in lzma module #115988

Open

terryjreedy requested changes Feb 27, 2024

View reviewed changes

Doc/library/lzma.rst Outdated Show resolved Hide resolved

Modules/_lzmamodule.c Show resolved Hide resolved

Modules/_lzmamodule.c Show resolved Hide resolved

bedevere-app bot added awaiting changes and removed awaiting review labels Feb 27, 2024

Suppress doc warning "py:const reference target not found"

3fa0c17

pitrou mentioned this pull request Feb 29, 2024

[Dev] Remove implicit workflow transitions apache/arrow#40296

Open

ivq requested a review from terryjreedy March 12, 2024 13:44

terryjreedy requested review from methane and serhiy-storchaka March 12, 2024 16:27

serhiy-storchaka reviewed Mar 13, 2024

View reviewed changes

ivq added 3 commits March 17, 2024 17:27

Add lzma version

c1e166a

Add ARM64, RISC-V filter test case

ea51476

Update lzma doc

6189ceb

Add NEWS entry

0f6fd63

Optimise NEWS

9731225

ivq requested a review from serhiy-storchaka March 18, 2024 12:57

serhiy-storchaka approved these changes Mar 18, 2024

View reviewed changes

Doc/library/lzma.rst Outdated Show resolved Hide resolved

bedevere-app bot added awaiting merge and removed awaiting changes labels Mar 18, 2024

Move version doc section

40e9d34

Clarifying comment, docs accuracy.

c62ca63

gpshead reviewed Mar 30, 2024

View reviewed changes

Modules/_lzmamodule.c Outdated Show resolved Hide resolved

gpshead reviewed Mar 30, 2024

View reviewed changes

Lib/test/test_lzma.py Outdated Show resolved Hide resolved

gpshead added 2 commits March 30, 2024 19:00

Expose the version as a number.

045aa40

We make lzma_version be the runtime version as that is the one most interestin to users. The compile time version remains exposed as the header version.

Merge branch 'main' into xz_bcj_filters

8f49f69

gpshead assigned gpshead and serhiy-storchaka Mar 30, 2024

gpshead added the type-feature A feature request or enhancement label Mar 30, 2024

gpshead added 4 commits March 30, 2024 19:05

minor version explanation tweak.

832335d

Adjust the whats new text.

728f286

NEWS wording and formatting tweak.

61c4adf

line length

98adeee

serhiy-storchaka mentioned this pull request Mar 31, 2024

Add structured version info for compression modules #117404

Open

StanFromIreland reviewed Jul 9, 2025

View reviewed changes

ivq added 2 commits July 13, 2025 16:41

Merge remote-tracking branch 'origin/main' into xz_bcj_filters

449b722

Signed-off-by: Chien Wong <m@xv97.com>

We've missed the 3.13 window

f381ca5

Signed-off-by: Chien Wong <m@xv97.com>

ivq force-pushed the xz_bcj_filters branch from d609e87 to f381ca5 Compare July 13, 2025 09:31

ivq requested review from StanFromIreland and gpshead July 13, 2025 09:33

ivq added 2 commits July 13, 2025 17:35

Fix accidentally removed newline when merging remote main

9f0d4d4

Signed-off-by: Chien Wong <m@xv97.com>

Remove unused import

1a17fd5

Signed-off-by: Chien Wong <m@xv97.com>

Uh oh!

gh-115988: Add missing ARM64 and RISCV filter in lzma module #115989

Are you sure you want to change the base?

gh-115988: Add missing ARM64 and RISCV filter in lzma module #115989

Conversation

ivq commented Feb 27, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Feb 27, 2024 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app bot commented Feb 27, 2024

Uh oh!

terryjreedy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-app bot commented Feb 27, 2024

Uh oh!

bedevere-app bot commented Feb 28, 2024

Uh oh!

ivq commented Feb 28, 2024

Uh oh!

terryjreedy commented Mar 12, 2024

Uh oh!

serhiy-storchaka left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-app bot commented Mar 17, 2024

Uh oh!

ivq commented Mar 17, 2024

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gpshead commented Mar 30, 2024

Uh oh!

gpshead commented Mar 30, 2024

Uh oh!

gpshead commented Mar 31, 2024

Uh oh!

gpshead commented Mar 31, 2024

Uh oh!

StanFromIreland left a comment

Choose a reason for hiding this comment

Uh oh!

ivq commented Jul 13, 2025

Uh oh!

furkanonder commented Jul 13, 2025

Uh oh!

bedevere-bot commented Jul 13, 2025

Uh oh!

furkanonder commented Jul 13, 2025

Uh oh!

bedevere-bot commented Jul 13, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

ivq commented Feb 27, 2024 •

edited by github-actions bot

Loading

ghost commented Feb 27, 2024 •

edited by ghost

Loading

serhiy-storchaka left a comment •

edited

Loading