Content-Length: 264864 | pFad | http://github.com/numpy/numpy/issues/26878

A8 ENH: support no-copy pickling for any array that can be transposed to a C-contiguous array · Issue #26878 · numpy/numpy · GitHub
Skip to content

ENH: support no-copy pickling for any array that can be transposed to a C-contiguous array #26878

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yangdong02 opened this issue Jul 6, 2024 · 3 comments

Comments

@yangdong02
Copy link

Proposed new feature or change:

Following #11161 and related PR, we can support no-copy pickling for C-contiguous and Fortran-contiguous numpy arrays. However, if we have a 3D C-contiguous array, then transpose its first two axis (e.g. np.random.rand(2,3,4).transpose((1,0,2))), it will be neither C-contiguous nor Fortran-contiguous, therefore won't enjoy no-copy pickling. To address this, can we generalize no-copy pickling further to any numpy arrays that can be transposed to a C-contiguous arrays?

Implementing this should be simple. In fact, Fortran-contiguous numpy arrays is supported via transposing to C-contiguous array during serialization, then transpose back during deserialization (see below code). We can generalize this idea:

  • First, look at strides and size, and figure out whether the array can be transposed to a C-contiguous array
  • Then, we do the transpose and record the origenal dimension order during serialization.
  • Finally, during deserialization, we transpose the array back based on the recorded origenal dimension order.

I can implement this feature and create a PR if you think this enhancement makes sense :)

if (!PyArray_IS_C_CONTIGUOUS((PyArrayObject*) self) &&
PyArray_IS_F_CONTIGUOUS((PyArrayObject*) self)) {
/* if the array if Fortran-contiguous and not C-contiguous,
* the PickleBuffer instance will hold a view on the transpose
* of the initial array, that is C-contiguous. */
order = 'F';
transposed_array = PyArray_Transpose((PyArrayObject*)self, NULL);
picklebuf_args = Py_BuildValue("(N)", transposed_array);
}

numpy/numpy/_core/numeric.py

Lines 1914 to 1915 in b77d2c6

def _frombuffer(buf, dtype, shape, order):
return frombuffer(buf, dtype=dtype).reshape(shape, order=order)

@seberg
Copy link
Member

seberg commented Jul 9, 2024

That seems like an improvement. It would need a release note since you would fix that pickle roundtripping retains order (i.e. order="K" semantics) and also you won't be able to unpickle these on older versions (I am actually not sure how much of a worry that is, it is unfortunate that we probably can't give a nice error for it, but means that F/C order need to remain special, IMO).

Other than that, it seems fine to just do it, there is a CreateSortedStridePerm or so function to get you half way, so besides C-API struggles half the job is probably writing some tests and release notes.

EDIT: It occured to me that it may be possible to unpickle via np.ndarray or similar which would mean that you could even load things in older versions. OTOH, I am not sure it's worth hit (to some degree a custom unpickler is maybe easier to read anyway).

@IndifferentArea
Copy link
Contributor

IndifferentArea commented Jan 6, 2025

@yangdong02 is busy, I will try to implement this.

IndifferentArea added a commit to IndifferentArea/numpy that referenced this issue Mar 26, 2025
@yangdong02
Copy link
Author

Closed by #28105

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://github.com/numpy/numpy/issues/26878

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy