Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster implementation of maps #160

Open
itamarst opened this issue Apr 7, 2019 · 7 comments
Open

Faster implementation of maps #160

itamarst opened this issue Apr 7, 2019 · 7 comments

Comments

@itamarst
Copy link
Contributor

itamarst commented Apr 7, 2019

I recently had to rip out some pyrsistent code because using it was 10× slower than regular Python data structures. It's probably possible to do better than current implementation.

Some possibilities:

@tobgu
Copy link
Owner

tobgu commented Apr 15, 2019

Yeah, it's a well known fact that the pmap is a lot slower than the corresponding built in map. I would not use it for any type of performance critical part of a system.
Even though it is based on a pvector (which is a relatively quick C-implementation) there is a lot of python code executing for each operation.

I'd love to see a C-implementation or an implementation with a very thin wrapper on top of an existing C/C++ implementation like one of the linked ones included in pyrsistent. Writing an implementation from scratch in C requires a fair effort though. Adopting one of the existing implementations may be an option, but making it compatible with the current implementation and the different inherited types would require some thinking.

Happy to take PRs on this! It's unlikely that I will engage in such an effort at this time since I don't have a personal need for it and have very limited time for hobby coding (which this would be) at the moment.

@itamarst
Copy link
Contributor Author

Definitely understand you don't have time for this, just thought it'd be good to have somewhere to track this.

FWIW, Immer seems to support the fancier stuff Pyrsistent does (temporarily-mutable objects), so I suspect it could be made to work with less trouble.

@mattiasw2
Copy link

Erlang added some new implementations of some of their data structures a couple of years ago, the same is used for dictionary and sets.

http://www1.erlang.org/doc/man/gb_trees.html

Prof. Arne Andersson's General Balanced Trees

http://user.it.uu.se/~arnea/abs/gbimpl.html

You find more on http://user.it.uu.se/~arnea/publications.html

@tobgu
Copy link
Owner

tobgu commented Apr 25, 2019

Thanks to both of you for the pointers above!

@arximboldi
Copy link

Hi! Immer author here :) Transients for maps are being implemented right now. I do believe that Immer is the fastest implementation of HAMTs and RRBTS out there (I tested the later claim at a ICFP paper, where I also benchmarked it against the C-backed Pyrsistent vectors). I would be very happy to have Immer become the backend for Pyrsistent, and would love to help with reviewing the code!

@samuelsinayoko
Copy link

Also keen on this. Is https://github.com/MagicStack/immutables an option for this as a backend @tobgu? It's already a Python package so would be easier to manage.
Otherwise would be happy to try and integrate immer but my C++ is very rusty so would need some pointers @arximboldi .

@arximboldi
Copy link

arximboldi commented Jan 17, 2024

@samuelsinayoko Immer does include in it source tree some experiments binding immer::vector to Python (and some benchmarks with Pyrsistent I think) you can check it out here: https://github.com/arximboldi/immer/tree/master/extra/python

For Maps there may be some other subtleties, but I think that could serve as good inspiration.

Immer maps now support transients (very efficiently!) and also support structural-sharing-aware diffing, which has lots of super cool applications. Also Immer vectors support logarithmic concatenatio and slicing, features that Pyrsistent currently does not support. So I think using Immer as a backend for Pyrsistent is actually a fantastic idea!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy