Fast conversion from numpy ndarray to vector #38834

user202729 · 2024-10-21T12:52:31Z

As explained, this allows fast conversion from numpy ndarray to some types of vector.

Side note: is cimport numpy as np safe? what happens if Sage is built without numpy?

Extra context: I originally tried to use memory view, but for some reason it doesn't work: cython/cython#6442

Traceback (most recent call last):
  File "/sage/src/bin/sage-ipython", line 10, in <module>
    banner()
  File "/sage/src/sage/misc/banner.py", line 114, in banner
    print(banner_text(full=True))
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/sage/src/sage/misc/banner.py", line 69, in banner_text
    import sage.all
  File "/sage/src/sage/all.py", line 91, in <module>
    from sage.algebras.all   import *
  File "/sage/src/sage/algebras/all.py", line 25, in <module>
    from sage.algebras.quatalg.all import *
  File "/sage/src/sage/algebras/quatalg/all.py", line 1, in <module>
    from sage.algebras.quatalg.quaternion_algebra import QuaternionAlgebra
  File "/sage/src/sage/algebras/quatalg/quaternion_algebra.py", line 76, in <module>
    from . import quaternion_algebra_cython
  File "sage/algebras/quatalg/quaternion_algebra_cython.pyx", line 1, in init sage.algebras.quatalg.quaternion_algebra_cython
  File "sage/matrix/matrix_integer_dense.pyx", line 1, in init sage.matrix.matrix_integer_dense
  File "sage/modules/vector_integer_dense.pyx", line 1, in init sage.modules.vector_integer_dense
AttributeError: type object 'sage.modules.vector_integer_dense.array' has no attribute '__getattr__'. Did you mean: '__setattr__'?

Using the old buffer syntax is a bit slower.

📝 Checklist

The title is concise and informative.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation and checked the documentation preview.

user202729 · 2024-10-22T09:31:01Z

Now use old buffer syntax, it should work.

Similar changes could be applied to vector_modn_dense.pyx and vector_integer_dense.pyx as well…

github-actions · 2024-10-22T10:16:52Z

Documentation preview for this PR (built with commit 7c54e21; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

user202729 · 2024-10-24T00:02:53Z

What am I supposed to do with the test? Move all numpy-related functions to a separate file and lazily import it? (but then won't the construction of any vector import numpy anyway?)

tscrim

I believe this Cython module gets imported at startup, which since you now have a global cimport of numpy causes the test to fail. Indeed, if I add a cimport numpy to integer.pyx (which is 100% imported at startup), then that test fails. So either you need to make everything local to that method or make sure this file is not loaded at startup.

user202729 · 2024-11-08T05:05:58Z

Lazy import appears to be difficult. I end up with a hacky solution, where I make the function in Python, and cast the pointer to uintptr_t.

I tried to use cdef extern but it doesn't seem to work (I don't know how to specify the correct library to load).

Annoyingly enough there doesn't seem to be mzd_write_bits to write multiple bits in a row, but mzd_set_ui and mzd_xor_bits should work.

It may be interesting to test if precompute the word and use the combination to set the whole word in at once would be faster. (Maybe it will, if for nothing other than loop unrolling.)

tscrim

Thank you. This is okay for me.

For your optimizations, that is beyond my technical abilities and knowledge. Although adding such functions would need to be done upstream I believe.

user202729 · 2024-11-08T12:05:13Z

In https://github.com/sagemath/sage/actions/runs/11736415975/job/32695629971, it gives ModuleNotFoundError: No module named 'sage.modules.numpy_util'

Maybe the test failure is caused by the pxd file not exist… not sure, it works locally for me. (After all we're importing from Python instead of from C, why is the pxd needed?)

I'll just push the commit to test it out, don't merge yet (until tests pass).

… not" This reverts commit 5b2e5e3.

user202729 · 2024-11-08T15:45:23Z

Should be done now.

Test failures are:

2024-11-08T14:13:41.9618970Z sage -t --warn-long 5.0 --random-seed=103363666288970081125621785486426094449 src/sage/categories/simplicial_sets.py  # 1 doctest failed
2024-11-08T14:13:41.9623870Z sage -t --warn-long 5.0 --random-seed=103363666288970081125621785486426094449 src/sage/rings/number_field/galois_group.py  # 1 doctest failed [failed in baseline: unreported failure on macOS (sort order) in GaloisGroup_v2.artin_symbol, https://github.com/sagemath/sage/actions/runs/9525536510/job/26259809272?pr=37998]
2024-11-08T14:13:41.9628610Z sage -t --warn-long 5.0 --random-seed=103363666288970081125621785486426094449 src/sage/rings/number_field/number_field.py  # 2 doctests failed [failed in baseline: unreported failure on macOS (sort order) in NumberField_absolute.[optimized_]subfields, https://github.com/sagemath/sage/actions/runs/9525536510/job/26259809272?pr=37998]
2024-11-08T14:13:41.9635330Z sage -t --warn-long 5.0 --random-seed=103363666288970081125621785486426094449 src/sage/rings/qqbar.py  # 1 doctest failed [failed in baseline: unreported failure on macOS seen in https://github.com/sagemath/sage/actions/runs/9525536510/job/26259809272?pr=37998]

First one #38940 , the rest are clearly written as unrelated.

tscrim

I feel like there should also be a .pxd file, right? Well, I guess that is not strictly necessary given the setup. I forget exactly when they are needed.

user202729 · 2024-11-14T09:30:19Z

If I understood correctly you need it when you want cimport from it to work… in this case we don't actually need that (only need Python import to work).

I can add for consistency though. (though it won't be tested so is also dangerous) The alternative is to leave documentation in the pyx explaining what's going on with the pxd being missing and to add it when useful.

tscrim · 2024-11-15T04:13:17Z

Let's add it for consistency and in case someone down the road has a use case for it. Thank you.

tscrim

Thank you. LGTM.

sagemathgh-38834: Fast conversion from numpy ndarray to vector As explained, this allows fast conversion from numpy ndarray to some types of vector. Side note: is `cimport numpy as np` safe? what happens if Sage is built without numpy? ---------- Extra context: I originally tried to use memory view, but for some reason it doesn't work: cython/cython#6442 ```python Traceback (most recent call last): File "/sage/src/bin/sage-ipython", line 10, in <module> banner() File "/sage/src/sage/misc/banner.py", line 114, in banner print(banner_text(full=True)) ^^^^^^^^^^^^^^^^^^^^^^ File "/sage/src/sage/misc/banner.py", line 69, in banner_text import sage.all File "/sage/src/sage/all.py", line 91, in <module> from sage.algebras.all import * File "/sage/src/sage/algebras/all.py", line 25, in <module> from sage.algebras.quatalg.all import * File "/sage/src/sage/algebras/quatalg/all.py", line 1, in <module> from sage.algebras.quatalg.quaternion_algebra import QuaternionAlgebra File "/sage/src/sage/algebras/quatalg/quaternion_algebra.py", line 76, in <module> from . import quaternion_algebra_cython File "sage/algebras/quatalg/quaternion_algebra_cython.pyx", line 1, in init sage.algebras.quatalg.quaternion_algebra_cython File "sage/matrix/matrix_integer_dense.pyx", line 1, in init sage.matrix.matrix_integer_dense File "sage/modules/vector_integer_dense.pyx", line 1, in init sage.modules.vector_integer_dense AttributeError: type object 'sage.modules.vector_integer_dense.array' has no attribute '__getattr__'. Did you mean: '__setattr__'? ``` Using the [old buffer syntax](https://cython.readthedocs.io/en/0.29.x/sr c/tutorial/numpy.html#working-numpy) is a bit slower. ### 📝 Checklist  - [x] The title is concise and informative. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [x] I have created tests covering the changes. - [x] I have updated the documentation and checked the documentation preview. URL: sagemath#38834 Reported by: user202729 Reviewer(s): Travis Scrimshaw

sagemathgh-39219: Faster conversion from numpy array to matrix mod 2 As in the title. Estimated 50× speedup. ### 📝 Checklist  - [x] The title is concise and informative. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [x] I have created tests covering the changes. - [x] I have updated the documentation and checked the documentation preview. (Reuses the `numpy_util` module from sagemath#38834 for the utility function) Reverse direction: sagemath#39152 URL: sagemath#39219 Reported by: user202729 Reviewer(s): Travis Scrimshaw

github-actions bot added the s: needs review label Oct 21, 2024

user202729 mentioned this pull request Oct 22, 2024

[BUG] embed_pos_in_docstring and fast_getattr gives error on memoryview cython/cython#6442

Open

user202729 force-pushed the fast-numpy-vector branch from bd5f9f8 to 52d7c0c Compare October 22, 2024 09:30

user202729 force-pushed the fast-numpy-vector branch from 52d7c0c to dff5068 Compare October 22, 2024 09:31

tscrim requested changes Nov 7, 2024

View reviewed changes

user202729 added 2 commits November 8, 2024 07:28

Fast conversion from numpy ndarray to vector

3afc011

Move all numpy-related functions to numpy_util and lazy import

90053d4

user202729 force-pushed the fast-numpy-vector branch from dff5068 to 90053d4 Compare November 8, 2024 05:03

user202729 added 2 commits November 8, 2024 15:06

Fix warning on test

0c3b5dd

Fix test situation where numpy is available but numpy_util is not

5b2e5e3

tscrim approved these changes Nov 8, 2024

View reviewed changes

tscrim added s: positive review c: linear algebra t: performance and removed s: needs review labels Nov 8, 2024

Try to add pxd file, see if it works

1ca7846

github-actions bot added s: needs review and removed s: positive review labels Nov 8, 2024

user202729 added 5 commits November 8, 2024 19:49

Revert "Fix test situation where numpy is available but numpy_util is…

daeceba

… not" This reverts commit 5b2e5e3.

Try to add the function signature to pxd file

2e8ad27

Fix build

f9448ca

Rollback to 0c3b5dd

49e7c14

Actually fix meson build (hopefully)

915ace9

Merge branch 'develop' into fast-numpy-vector

bbac616

user202729 requested a review from tscrim November 12, 2024 06:07

tscrim reviewed Nov 14, 2024

View reviewed changes

Add numpy_util.pxd file for consistency

7c54e21

tscrim approved these changes Nov 15, 2024

View reviewed changes

tscrim added s: positive review and removed s: needs review labels Nov 15, 2024

vbraun merged commit 57bfb9d into sagemath:develop Dec 8, 2024
22 of 23 checks passed

github-actions bot removed the s: positive review label Dec 9, 2024

user202729 mentioned this pull request Dec 18, 2024

Make matrix mod 2 conversion to numpy faster & some semantic fixes #39152

Open

5 tasks

user202729 mentioned this pull request Dec 29, 2024

Faster conversion from numpy array to matrix mod 2 #39219

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fast conversion from numpy ndarray to vector #38834

Fast conversion from numpy ndarray to vector #38834

Uh oh!

user202729 commented Oct 21, 2024 •

edited

Loading

Uh oh!

user202729 commented Oct 22, 2024

Uh oh!

github-actions bot commented Oct 22, 2024 •

edited

Loading

Uh oh!

user202729 commented Oct 24, 2024

Uh oh!

tscrim left a comment

Uh oh!

user202729 commented Nov 8, 2024 •

edited

Loading

Uh oh!

tscrim left a comment

Uh oh!

user202729 commented Nov 8, 2024 •

edited

Loading

Uh oh!

user202729 commented Nov 8, 2024

Uh oh!

tscrim left a comment

Uh oh!

user202729 commented Nov 14, 2024

Uh oh!

tscrim commented Nov 15, 2024

Uh oh!

tscrim left a comment

Uh oh!

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Uh oh!

Fast conversion from numpy ndarray to vector #38834

Fast conversion from numpy ndarray to vector #38834

Uh oh!

Conversation

user202729 commented Oct 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Checklist

Uh oh!

user202729 commented Oct 22, 2024

Uh oh!

github-actions bot commented Oct 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

user202729 commented Oct 24, 2024

Uh oh!

tscrim left a comment

Choose a reason for hiding this comment

Uh oh!

user202729 commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tscrim left a comment

Choose a reason for hiding this comment

Uh oh!

user202729 commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

user202729 commented Nov 8, 2024

Uh oh!

tscrim left a comment

Choose a reason for hiding this comment

Uh oh!

user202729 commented Nov 14, 2024

Uh oh!

tscrim commented Nov 15, 2024

Uh oh!

tscrim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

user202729 commented Oct 21, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024 •

edited

Loading

user202729 commented Nov 8, 2024 •

edited

Loading

user202729 commented Nov 8, 2024 •

edited

Loading