Skip to content

[WIP] [GSOC] KV Caching for LLM inference #27205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 11 commits into
base: 5.x
Choose a base branch
from

Conversation

nklskyoy
Copy link

@nklskyoy nklskyoy commented Apr 7, 2025

This PR Introduces new dnn::ArgKind: DNN_ARG_CACHED. This can be used to store cache of dynamic size, which persists between runs of a net.

The cache management (allocation, growing, writing) is intended to be performed directly from the layers.
The memory is organized in pages, where each page is a Mat.

Caching is particularly useful for LLM inference, where Key and Value tokens are reused for generating subsequent tokens. The KV Caching will be added to attention layer.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

Reference: #27176

@asmorkalov asmorkalov added the pr: needs test New functionality requires minimal tests set label Apr 22, 2025
@fengyuentau fengyuentau self-requested a review April 23, 2025 07:39
@asmorkalov asmorkalov added this to the 5.0-release milestone Apr 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: dnn feature pr: needs test New functionality requires minimal tests set
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy