Skip to content

Better 1.5 bit quantization #5971

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Mar 11, 2024
Merged

Better 1.5 bit quantization #5971

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
c9e9acf
Trying blocvks of 16 for IQ1_S - seems slightly better
Mar 8, 2024
cd83a7d
iq1s_blocks16: Adjust scale fudge factor to 1.125
Mar 8, 2024
4c4404a
iq1s_blocks16: going to blocks of 32
Mar 8, 2024
c55e66f
iq1s_blocks16: Use 2*<x^2> as sigma2 in weight adjustment
Mar 8, 2024
864a5c2
iq1s_blocks16: scalar and AVX2 dot products
Mar 8, 2024
f092d04
iq1s_blocks16: CUDA dot product
Mar 8, 2024
fbb001e
iq1s_blocks16: Metal works, Neon does not
Mar 8, 2024
15acc79
iq1s_blocks16: fixed Neon
Mar 8, 2024
8561139
iq1s_blocks16: very slightly faster TG on Metal
Mar 8, 2024
d3da9d1
iq1s_blocks16: speedup Metal by packing codebook into uint32_t's
Mar 9, 2024
7545d69
Formatting
Mar 9, 2024
156220f
iq1s_blocks16: uint32_t codebook is also better in CUDA
Mar 9, 2024
101b18d
iq1s_blocks16: slightly faster Neon dot product
Mar 9, 2024
34bc21f
iq1s_blocks16: faster AVX2 dot product
Mar 9, 2024
9d83171
iq1s_blocks16: adjust to ggml-common.h
Mar 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
iq1s_blocks16: fixed Neon
  • Loading branch information
Iwan Kawrakow committed Mar 11, 2024
commit 15acc7923bb6b77c2d1e6200dce9e25ba28acf46
4 changes: 2 additions & 2 deletions ggml-quants.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9598,8 +9598,8 @@ void ggml_vec_dot_iq1_s_q8_K (int n, float * restrict s, size_t bs, const void

q8b = ggml_vld1q_s8_x4(q8); q8 += 64;

const int32x4_t p1 = ggml_vdotq_s32(q1b.val[0], q8b.val[0], ggml_vdotq_s32(q1b.val[1], q8b.val[1], vdupq_n_s32(0)));
const int32x4_t p2 = ggml_vdotq_s32(q1b.val[2], q8b.val[2], ggml_vdotq_s32(q1b.val[3], q8b.val[3], vdupq_n_s32(0)));
const int32x4_t p1 = ggml_vdotq_s32(ggml_vdotq_s32(vdupq_n_s32(0), q1b.val[0], q8b.val[0]), q1b.val[1], q8b.val[1]);
const int32x4_t p2 = ggml_vdotq_s32(ggml_vdotq_s32(vdupq_n_s32(0), q1b.val[2], q8b.val[2]), q1b.val[3], q8b.val[3]);

sumi1 += vaddvq_s32(p1) * (2*(qh[ib+0] >> 12) + 1);
sumi2 += vaddvq_s32(p2) * (2*(qh[ib+1] >> 12) + 1);
Expand Down
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy