Skip to content

[Feature] Support AVX2 on linux #192

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

lshAlgorithm
Copy link

@lshAlgorithm lshAlgorithm commented May 13, 2025

Support the AVX2 on Linux.
Main change: In rwkv_operators_wkv_v7.inc, it is straightforward, we just need to add things up in vector and reduce then to sum out of loop using horizontal_sum.
Reproduce: I've added the compile option, so just compile it using ./script.sh, and run generate_completion.py following README.
Remaining issues: Because the reduction of AVX vector is instrinsics-depedent, and my pc doesn't support avx512, let alone arm, the code support AVX2 ONLY; and it seems that ggml doesn't fully support AVX2, and doesn't include <immintrin.h>. (Maybe I don't get it right, I just add the header in the code.)
Outcome: Test on Fedora40 with Intel(R) Core(TM) Ultra 9 185H (No GPU). Slightly speedup using rwkv-7-1.5B, fp16, so as rwkv-7-2.9B, fp16. (To be honest, it is about ms level faster, and my PC is not that stable, so I cannot show an accurate speedup...)
Statement: Sorry for the stupid commit explanation, it was meant for private use days before. No much experience in PR, but really wanna contribute to the community. Hope for suggestion.

lshAlgorithm added 3 commits May 14, 2025 02:01
Signed-off-by: lshAlgorithm <lishuhuai_brain@163.com>
Signed-off-by: lshAlgorithm <lishuhuai_brain@163.com>
Signed-off-by: lshAlgorithm <lishuhuai_brain@163.com>
Signed-off-by: lshAlgorithm <lishuhuai_brain@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy