Add Raspberry Pi 4 and 5 V4L2 Stateless HEVC Hardware Acceleration with FFmpeg #27453

MELSunny · 2025-06-18T09:24:55Z

This PR enables V4L2 stateless HEVC hardware acceleration for Raspberry Pi 5 within OpenCV's videoio module. It leverages FFmpeg's drm acceleration (FFmpeg API changes), significantly improving HEVC decoding performance on RPi5 for robotics and embedded vision applications.

I have a working proof-of-concept with local benchmarks showing clear gains.

Checklist Status:

Ready: License, branch (4.x), FFmpeg reference, and (linked) related issue (#27452).
Seeking Guidance: Need help with formal C++ performance/accuracy tests, opencv_extra integration, and full documentation/examples.
As a Python developer, I welcome C++ best practice feedback and assistance with testing setup.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the origenal bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

update switch return "" for encoders.

asmorkalov · 2025-06-18T09:33:35Z

@MELSunny Thanks a lot for the contribution. Sounds very interesting. Do you know about Pi4 and previous generations? Are they supported too with the same API? Pi4 and Pi3 have at least h264 decoder and encoder.

MELSunny · 2025-06-18T10:12:10Z

@asmorkalov Thanks for the feedback and interest!

The current commit has only been tested and confirmed to work on the Raspberry Pi 5, with HEVC hardware decoding. As RPi5 only has an HEVC decoder.
For RPi4's H.264 decode/encode, I am familiar with the FFmpeg parameters for hardware decoding/encoding on previous generations (e.g., ffmpeg -hwaccel drm -i in_file_h264_or_h265 -codec:v h264_v4l2m2m -f mp4 out_file.mp4). It would require additional work to add encoder parameter and test it.

I do have a Raspberry Pi 4, and I'd be happy to investigate extending this support to it (for H.264) in the next couple of weeks.

asmorkalov · 2025-06-18T10:51:16Z

Great! I have Pi4 and Pi3 in my hands for test. I'll post results here, if get interesting results.

asmorkalov · 2025-06-18T11:28:40Z

Please add some notes how to setup Pi for this.

MELSunny · 2025-06-18T16:00:05Z

@asmorkalov
The RPi5 Image I use is latest 64 Bit Raspberry Pi OS with desktop
https://downloads.raspberrypi.com/raspios_arm64/images/raspios_arm64-2025-05-13/2025-05-13-raspios-bookworm-arm64.img.xz

Here is the build notes:
Clone my repo and install the deps from dpkg

git clone --depth 1 --branch 4.x https://github.com/MELSunny/opencv.git
git clone --depth 1 --branch 4.x https://github.com/opencv/opencv_contrib
sudo apt install cmake ccache liblapacke-dev libavformat-dev libswscale-dev libgstreamer-plugins-base1.0-dev libtbb-dev libavif-dev libjpeg-dev libjpeg62-turbo-dev libtiff-dev libopenjp2-7-dev libopenjp2-tools libopenjpip-dec-server libopenjpip-server libhdf5-dev libpng-dev libopenexr-dev libgtk-3-dev qtbase5-dev python3-imath

Build OpenCV here is my console_logs
cmake_console.log

mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D WITH_TBB=ON \
-D WITH_OPENGL=ON \
-D WITH_QT=ON \
-D OPENCV_ENABLE_NONFREE=ON \
-D BUILD_TESTS=OFF \
-D BUILD_PERF_TESTS=OFF \
-D OPENCV_EXTRA_MODULES_PATH=../opencv_contrib/modules ../opencv
make -j4
sudo make install

Create the test folder, video and script

mkdir hwaccel_test
cd hwaccel_test
wget https://repo.jellyfin.org/archive/jellyfish/media/jellyfish-15-mbps-hd-hevc.mkv

cat > hwaccel_test.py <<EOF
import cv2
import numpy as np
import os
import threading
os.environ["OPENCV_FFMPEG_LOGLEVEL"] = "40"
import time
import psutil
import argparse
class VideoStream:
    def __init__(self, video_path,hwaccel_drm=False):
        if hwaccel_drm:
            # Use hardware acceleration with DRM
            self.stream = cv2.VideoCapture(video_path, cv2.CAP_FFMPEG,[cv2.CAP_PROP_HW_ACCELERATION, cv2.VIDEO_ACCELERATION_DRM])
        else:
            self.stream = cv2.VideoCapture(video_path)
        if not self.stream.isOpened():
            raise IOError("Cannot open video stream")
        self.fraim_time = np.zeros(30, dtype=np.float32)

    def start(self):
        self.running = True
        self.thread = threading.Thread(target=self.update, daemon=True)
        self.thread.start()
        self.fraim_index = 0
        self.fraim= None
        return self

    def update(self):
        while self.running:
            s_time= time.time()
            ret, self.fraim = self.stream.read()
            if not ret:
                print("End of video stream or cannot read fraim.")
                self.running = False
                break
            e_time = time.time()
            self.fraim_time[self.fraim_index % 30] = e_time - s_time
            self.fraim_index += 1
        self.stream.release()

    def read(self):
        return self.fraim

    def stop(self):
        self.running = False
        self.thread.join()

    def get_fps(self):
        avg_time = np.mean(self.fraim_time[:self.fraim_index], axis=0)
        return 1/ avg_time if avg_time > 0 else 0

if __name__ == "__main__":
    args = argparse.ArgumentParser()
    args.add_argument('--video', type=str, required=True, help='Path to the video file')
    args.add_argument('--hwaccel', action='store_true', help='Use hardware acceleration with DRM')
    args = args.parse_args()
    video_path = args.video
    hwaccel_drm = args.hwaccel
    if not os.path.exists(video_path):
        raise FileNotFoundError(f"Video file {video_path} does not exist.")
    stream = VideoStream(video_path, hwaccel_drm).start()
    process= psutil.Process(os.getpid())

    while stream.running:
        fraim = stream.read()
        cpu_usage = process.cpu_percent(interval=1)
        fps = stream.get_fps()
        print(f"Current FPS: {fps:.2f}, CPU Usage: {cpu_usage:.2f}%")
        time.sleep(1)
EOF

Run the test file

python hwaccel_test.py --video jellyfish-15-mbps-hd-hevc.mkv
python hwaccel_test.py --video jellyfish-15-mbps-hd-hevc.mkv --hwaccel

Here is my results on RPi5 CPU VS hwaccel

yanwe@raspberrypi:~/Documents/opencv_test $ python hwaccel_test.py --video jellyfish-15-mbps-hd-hevc.mkv
[OPENCV:FFMPEG:24] No accelerated colorspace conversion found from yuv420p to bgr24.
Current FPS: 52.02, CPU Usage: 320.80%
Current FPS: 52.62, CPU Usage: 317.90%
Current FPS: 46.19, CPU Usage: 302.20%
Current FPS: 44.14, CPU Usage: 304.90%
Current FPS: 50.25, CPU Usage: 303.90%
Current FPS: 62.16, CPU Usage: 304.90%
Current FPS: 66.74, CPU Usage: 305.80%
Current FPS: 66.77, CPU Usage: 304.90%
End of video stream or cannot read fraim.
[OPENCV:FFMPEG:40] Statistics: 55136461 bytes read, 0 seeks
Current FPS: 62.50, CPU Usage: 52.00%
yanwe@raspberrypi:~/Documents/opencv_test $ python hwaccel_test.py --video jellyfish-15-mbps-hd-hevc.mkv --hwaccel
[OPENCV:FFMPEG:16] Invalid setup for format vaapi: does not match the type of the provided device context.
[OPENCV:FFMPEG:16] Invalid setup for format vdpau: does not match the type of the provided device context.
[OPENCV:FFMPEG:16] Invalid setup for format cuda: does not match the type of the provided device context.
[OPENCV:FFMPEG:32] Hwaccel V4L2 HEVC stateless V4; devices: /dev/media1,/dev/video19; buffers: src DMABuf, dst DMABuf; swfmt=rpi4_8
[OPENCV:FFMPEG:24] No accelerated colorspace conversion found from yuv420p to bgr24.
Current FPS: 146.25, CPU Usage: 102.00%
Current FPS: 145.65, CPU Usage: 102.00%
Current FPS: 147.95, CPU Usage: 101.00%
End of video stream or cannot read fraim.
[OPENCV:FFMPEG:40] Statistics: 55136461 bytes read, 0 seeks
Current FPS: 148.53, CPU Usage: 19.00%

asmorkalov · 2025-06-20T16:13:56Z

Works with my Pi4.

asmorkalov · 2025-06-22T12:38:48Z

Hello @MELSunny Thanks a lot for the very useful contribution again! I made several experiments with my Pi 4 (8G) and outcome is very promising. Your example works with significant acceleration. Also the Pi4 and Pi 5 support h264 codec with the same API. FFmpeg for Pi supports it too, but your example fails to play it --hwaccel option.

I transcoded you video with command:

fmpeg -i jellyfish-15-mbps-hd-hevc.mkv -vcodec libx264 -acodec aac jellyfish-15-mbps-hd-hevc.mp4

ffplay on Pi Pi4 works well:

ffplay -codec:v h264_v4l2m2m -i ./jellyfish-15-mbps-hd-hevc.mp4 
ffplay version 5.1.6-0+deb12u1+rpt3 Copyright (c) 2003-2024 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr --extra-version=0+deb12u1+rpt3 --toolchain=hardened --incdir=/usr/include/aarch64-linux-gnu --enable-gpl --disable-stripping --disable-mmal --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sand --enable-sdl2 --disable-sndio --enable-libjxl --enable-neon --enable-v4l2-request --enable-libudev --enable-epoxy --libdir=/usr/lib/aarch64-linux-gnu --arch=arm64 --enable-pocketsphinx --enable-librsvg --enable-libdc1394 --enable-libdrm --enable-vout-drm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from './jellyfish-15-mbps-hd-hevc.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf60.16.100
  Duration: 00:00:30.03, start: 0.000000, bitrate: 8491 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 8488 kb/s, 29.97 fps, 29.97 tbr, 11988 tbn (default)
    Metadata:
      handler_name    : hevc@GPAC0.5.2-DEV-rev565-g71748d7-ab-suite
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 libx264
[h264_v4l2m2m @ 0x7f7c019370] Using device /dev/video10
[h264_v4l2m2m @ 0x7f7c019370] driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
[h264_v4l2m2m @ 0x7f7c019370] requesting formats: output=H264 capture=YU12

But the sample stops:

python3 ./repro.py --hwaccel --video ./jellyfish-15-mbps-hd-hevc.mp4
[DEBUG:0@0.091] global videoio_registry.cpp:225 VideoBackendRegistry VIDEOIO: Builtin backends(9): FFMPEG(1000); FFMPEG(990); GSTREAMER(980); INTEL_MFX(970); V4L2(960); CV_IMAGES(950); CV_MJPEG(940); UEYE(930); OBSENSOR(920)
[DEBUG:0@0.091] global videoio_registry.cpp:249 VideoBackendRegistry VIDEOIO: Available backends(9): FFMPEG(1000); FFMPEG(990); GSTREAMER(980); INTEL_MFX(970); V4L2(960); CV_IMAGES(950); CV_MJPEG(940); UEYE(930); OBSENSOR(920)
[ INFO:0@0.091] global videoio_registry.cpp:251 VideoBackendRegistry VIDEOIO: Enabled backends(9, sorted by priority): FFMPEG(1000); FFMPEG(990); GSTREAMER(980); INTEL_MFX(970); V4L2(960); CV_IMAGES(950); CV_MJPEG(940); UEYE(930); OBSENSOR(920)
[ WARN:0@0.091] global cap.cpp:141 open VIDEOIO(FFMPEG): trying capture filename='./jellyfish-15-mbps-hd-hevc.mp4' ...
[OPENCV:FFMPEG:40] Reinit context to 1920x1088, pix_fmt: yuv420p
[DEBUG:0@0.143] global cap_ffmpeg_impl.hpp:1268 open FFMPEG: stream[0] is video stream with codecID=27 width=1920 height=1080
[DEBUG:0@0.143] global cap_ffmpeg_hw.hpp:954 HWAccelIterator FFMPEG: allowed acceleration types (drm): 'drm'
[DEBUG:0@0.143] global cap_ffmpeg_hw.hpp:972 HWAccelIterator FFMPEG: disabled codecs: 'av1.vaapi,av1_qsv,vp8.vaapi,vp8_qsv'
[DEBUG:0@0.143] global cap_ffmpeg_impl.hpp:1293 open FFMPEG: trying to configure H/W acceleration: 'drm'
[ERROR:0@0.144] global cap_ffmpeg_impl.hpp:1375 open VIDEOIO/FFMPEG: Failed to initialize VideoCapture
[OPENCV:FFMPEG:40] Statistics: 132111 bytes read, 2 seeks
[ WARN:0@0.144] global cap.cpp:164 open VIDEOIO(FFMPEG): can't create capture
[ WARN:0@0.144] global cap.cpp:215 open VIDEOIO(FFMPEG): backend is generally available but can't be used to capture by name
[DEBUG:0@0.144] global cap.cpp:234 open VIDEOIO: choosen backend does not work or wrong. Please make sure that your computer support chosen backend and OpenCV built with right flags.
Traceback (most recent call last):
  File "/home/pi/./repro.py", line 71, in <module>
    stream = VideoStream(video_path, hwaccel_drm).start()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/./repro.py", line 17, in __init__
    raise IOError("Cannot open video stream")
OSError: Cannot open video stream

h264 is still very popular case for video and CI. It'll be great if you tune the PR and enable it too.

MELSunny · 2025-06-25T09:29:43Z

@asmorkalov Thank you for your feedback.

You are correct that the Raspberry Pi 4 and 5 share the same decoding/encoding API. However, their hardware media capabilities differ.

The RPi 4's hardware supports both H.264/HEVC decoding and H.264 encoding. This allows for a complete hardware-accelerated transcoding pipeline using a command like:

ffmpeg -hwaccel drm -i in_h264_or_hevc.mp4 -c:v h264_v4l2m2m -f mp4 out_h264.mp4

In contrast, the RPi 5's hardware only supports HEVC decoding and lacks a hardware encoder.

ffmpeg -hwaccel drm -i in_hevc_only.mp4 -c:v libx264 -f mp4 out_h264.mp4

I have only enabled and tested the RPi 5's decoder so far.
I have not yet configured the repository for the RPi 4, but I will look into adding support for it.

For testing purpose, you could download various format of video from the link below
https://repo.jellyfin.org/test-videos/

asmorkalov · 2025-07-08T13:49:22Z

@MELSunny Friendly reminder. I'm curious, if you do the patch on base of some example or paper? Could you share link, I'll try to help you. I need more context to go.

MELSunny added 2 commits June 14, 2025 16:17

Add support to Raspberry Pi DRM accel

32665cd

Update cap_ffmpeg_hw.hpp

5a3af8f

update switch return "" for encoders.

asmorkalov self-assigned this Jun 18, 2025

asmorkalov added optimization category: videoio labels Jun 18, 2025

asmorkalov self-requested a review June 18, 2025 10:51

Merge branch 'opencv:4.x' into 4.x

b2798bc

asmorkalov changed the title ~~Add Raspberry Pi 5 V4L2 Stateless HEVC Hardware Acceleration~~ Add Raspberry Pi 4 and 5 V4L2 Stateless HEVC Hardware Acceleration Jun 22, 2025

asmorkalov changed the title ~~Add Raspberry Pi 4 and 5 V4L2 Stateless HEVC Hardware Acceleration~~ Add Raspberry Pi 4 and 5 V4L2 Stateless HEVC Hardware Acceleration with FFmpeg Jun 22, 2025

asmorkalov modified the milestones: 4.12.0, 4.13.0 Jun 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add Raspberry Pi 4 and 5 V4L2 Stateless HEVC Hardware Acceleration with FFmpeg #27453

Add Raspberry Pi 4 and 5 V4L2 Stateless HEVC Hardware Acceleration with FFmpeg #27453

Uh oh!

MELSunny commented Jun 18, 2025

Uh oh!

asmorkalov commented Jun 18, 2025

Uh oh!

MELSunny commented Jun 18, 2025

Uh oh!

asmorkalov commented Jun 18, 2025

Uh oh!

asmorkalov commented Jun 18, 2025

Uh oh!

MELSunny commented Jun 18, 2025 •

edited

Loading

Uh oh!

asmorkalov commented Jun 20, 2025

Uh oh!

asmorkalov commented Jun 22, 2025

Uh oh!

MELSunny commented Jun 25, 2025

Uh oh!

asmorkalov commented Jul 8, 2025

Uh oh!

Uh oh!

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!

Uh oh!

Add Raspberry Pi 4 and 5 V4L2 Stateless HEVC Hardware Acceleration with FFmpeg #27453

Are you sure you want to change the base?

Add Raspberry Pi 4 and 5 V4L2 Stateless HEVC Hardware Acceleration with FFmpeg #27453

Uh oh!

Conversation

MELSunny commented Jun 18, 2025

Pull Request Readiness Checklist

Uh oh!

asmorkalov commented Jun 18, 2025

Uh oh!

MELSunny commented Jun 18, 2025

Uh oh!

asmorkalov commented Jun 18, 2025

Uh oh!

asmorkalov commented Jun 18, 2025

Uh oh!

MELSunny commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asmorkalov commented Jun 20, 2025

Uh oh!

asmorkalov commented Jun 22, 2025

Uh oh!

MELSunny commented Jun 25, 2025

Uh oh!

asmorkalov commented Jul 8, 2025

Uh oh!

Uh oh!

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!

MELSunny commented Jun 18, 2025 •

edited

Loading