Skip to content

gh-82088: Improve performance of PyLong_As*() for multi-digit ints #135585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 16 commits into from
Jul 10, 2025
Merged

gh-82088: Improve performance of PyLong_As*() for multi-digit ints #135585

merged 16 commits into from
Jul 10, 2025
+80 −35

Conversation

eendebakpt
Copy link
Contributor

@eendebakpt eendebakpt commented Jun 16, 2025 •
edited
Loading

This is a continuation of the work in #15363 by @sir-sigurd.

Performance is improved by

  • Unrolling the part of the loop where no overflow can occur
  • Improving the overflow check so no temporary variable is used

Performance: main

pack n dt 460.21 [ms]
pack N dt 476.32 [ms]
pack l dt 479.76 [ms]
pack L dt 451.84 [ms]

PR

pack n dt 401.02 [ms]
pack N dt 391.06 [ms]
pack l dt 408.83 [ms]
pack L dt 402.37 [ms]

Script:

import time
from struct import Struct

if __name__ == "__main__":
    for key in 'nNlL':
        N = 1000
        pack = Struct(key*N).pack
        values = tuple(range(2**30, 2**30 + N))
            
        t0=time.perf_counter()
        for _ in range(50_000):
            pack(*values);
        dt=time.perf_counter()-t0
        
        print(f'pack {key} dt {dt*1e3:.2f} [ms]')

(performance gain is similar to earlier reported improvements)

@bedevere-app bedevere-app bot mentioned this pull request Jun 16, 2025
Open
eendebakpt
eendebakpt commented Jun 16, 2025
View reviewed changes
Objects/longobject.c Outdated
assert(ULONG_MAX >= ((1UL << PyLong_SHIFT) - 1));
x = digits[i];

#if ((ULONG_MAX >> PyLong_SHIFT)) >= ((1UL << PyLong_SHIFT) - 1)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can also unroll the second digit for the other methods. It gains a bit of performance at the cost of a bit of extra code. The unrolling can be factored out to a helper function (the loop itself is not so easy because there are small differences in how to handle the overflow)

@skirpichev skirpichev mentioned this pull request Jun 17, 2025
Closed
vstinner
vstinner reviewed Jun 17, 2025
View reviewed changes
eendebakpt and others added 2 commits June 17, 2025 16:46
@eendebakpt @vstinner
Update Objects/longobject.c
401d860 
Co-authored-by: Victor Stinner <vstinner@python.org>
@eendebakpt
test
750c676
@eendebakpt eendebakpt changed the title Draft: gh-82088: Improve performance of PyLong_As*() for multi-digit ints gh-82088: Improve performance of PyLong_As*() for multi-digit ints Jun 17, 2025
@eendebakpt eendebakpt requested a review from vstinner June 24, 2025 11:26
vstinner
vstinner reviewed Jun 24, 2025
View reviewed changes
@vstinner
Copy link
Member

I rewrote the benchmark using pyperf:

import pyperf
from struct import Struct

N = 1000

runner = pyperf.Runner()
values = tuple(range(2**30, 2**30 + N))
for key in 'nNlL':
    pack = Struct(key*N).pack
    runner.bench_func(f'pack {key}', pack, *values)

Result:

Benchmark ref change
pack n 7.86 us 5.80 us: 1.35x faster
pack N 7.06 us 5.77 us: 1.22x faster
pack l 9.00 us 6.31 us: 1.43x faster
pack L 7.66 us 6.00 us: 1.28x faster
Geometric mean (ref) 1.32x faster

vstinner
vstinner reviewed Jun 24, 2025
View reviewed changes
eendebakpt
eendebakpt commented Jun 24, 2025
View reviewed changes
vstinner
vstinner approved these changes Jun 24, 2025
View reviewed changes
Copy link
Member

@vstinner vstinner left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I have two last minor suggestions.

@serhiy-storchaka: Would you mind to double check that the integer overflow checks are correct?

Replace:

            prev = x;
            x = (x << PyLong_SHIFT) | v->long_value.ob_digit[i];
            if ((x >> PyLong_SHIFT) != prev) ...

with:

            if (x > (ULONG_MAX >> PyLong_SHIFT)) ...
            x = (x << PyLong_SHIFT) | v->long_value.ob_digit[i];

@eendebakpt @vstinner
Apply suggestions from code review
2d9d098 
Co-authored-by: Victor Stinner <vstinner@python.org>
skirpichev
skirpichev reviewed Jun 25, 2025
View reviewed changes
@vstinner vstinner merged commit d754f75 into python:main Jul 10, 2025
40 checks passed
@vstinner
Copy link
Member

Thanks @eendebakpt, I merged your PR. Let's see how it goes :-)

AndPuQing pushed a commit to AndPuQing/cpython that referenced this pull request Jul 11, 2025
@eendebakpt @blurb-it
@vstinner @AndPuQing
pythongh-82088: Improve performance of PyLong_As*() for multi-digit i…
77512cd 
…nts (python#135585)

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
@eendebakpt @blurb-it
@vstinner @Pranjal095
pythongh-82088: Improve performance of PyLong_As*() for multi-digit i…
0b4d342 
…nts (python#135585)

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vstinner vstinner vstinner approved these changes

@skirpichev skirpichev skirpichev left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@eendebakpt @vstinner @skirpichev
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy