[WIP] modules/videoio: add libcamera backend for OpenCV Video I/O #27155

advait-0 · 2025-03-26T06:19:55Z

Adds libcamera backend for OpenCV's Video I/O module. Provides integration with the libcamera stack, enabling native support for modern Linux camera pipelines, especially on embedded platforms like Raspberry Pi.
Implements the functionality to be used by the VideoCapture class for reading and displaying supported streams.

What works

Displaying frames from common formats: MJPEG, YUYV, NV12, and more.
Setting camera properties:
- cv::CAP_PROP_FRAME_WIDTH – Set frame width
- cv::CAP_PROP_FRAME_HEIGHT – Set frame height
- cv::CAP_PROP_MODE – Select PixelFormat
  (e.g., FMT_MJPEG, FMT_YUYV, FMT_RGB888, FMT_BGR888, FMT_NV12, FMT_YUV420)
- cv::CAP_PROP_FORMAT – Select StreamRole
  (e.g., ROLE_RAW, ROLE_VIDEO, ROLE_STILL, ROLE_VIEWFINDER)
Reading some set camera properties (may be unstable):
- cv::CAP_PROP_FRAME_WIDTH
- cv::CAP_PROP_FRAME_HEIGHT

Known issues

Multiple camera support works, but it is laggy for the second camera at the moment (tested with 2 cameras)
~~cv::CAP_PROP_MODE must be explicitly set, for USB and CSI cameras, to avoid crashes.~~

Detailed usage instructions can be found here.
Sample testing program can be found here.

Attempts to close #21653

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

asmorkalov · 2025-03-26T07:46:20Z

Let's use FMT_BGR888 by default, if CAP_PROP_MODE is not set by user. It's default OpenCV behaviour.

asmorkalov · 2025-03-26T15:23:59Z

Raspberry Pi 4, libcamera from git default branch as in gist.
CMake:

--   Video I/O:
--     FFMPEG:                      NO
--       avcodec:                   NO
--       avformat:                  NO
--       avutil:                    NO
--       swscale:                   NO
--       avresample:                NO
--     GStreamer:                   YES (1.18.4)
--     v4l/v4l2:                    YES (linux/videodev2.h)
--     libcamera:                   YES (0.4.0)

Build:

[ 53%] Linking CXX shared library ../../lib/libopencv_videoio.so
/usr/bin/ld: CMakeFiles/opencv_videoio.dir/src/cap_libcamera.cpp.o: in function `cv::CvCapture_libcamera_proxy::getProperty(int) const':
cap_libcamera.cpp:(.text._ZNK2cv25CvCapture_libcamera_proxy11getPropertyEi+0x38): undefined reference to `libcamera::Request::sequence() const'
/usr/bin/ld: CMakeFiles/opencv_videoio.dir/src/cap_libcamera.cpp.o: in function `cv::CvCapture_libcamera_proxy::open()':
cap_libcamera.cpp:(.text._ZN2cv25CvCapture_libcamera_proxy4openEv+0x260): undefined reference to `libcamera::Request::addBuffer(libcamera::Stream const*, libcamera::FrameBuffer*, std::unique_ptr<libcamera::Fence, std::default_delete<libcamera::Fence> >)'
/usr/bin/ld: CMakeFiles/opencv_videoio.dir/src/cap_libcamera.cpp.o: in function `cv::CvCapture_libcamera_proxy::grabFrame()':
cap_libcamera.cpp:(.text._ZN2cv25CvCapture_libcamera_proxy9grabFrameEv+0x4c): undefined reference to `libcamera::Camera::generateConfiguration(libcamera::Span<libcamera::StreamRole const, 18446744073709551615ul>)'
collect2: error: ld returned 1 exit status
make[2]: *** [modules/videoio/CMakeFiles/opencv_videoio.dir/build.make:324: lib/libopencv_videoio.so.5.0.0] Error 1

asmorkalov · 2025-03-26T16:54:37Z

Similar issue with system wide (libcamera-dev) package:

[ 45%] Linking CXX shared library ../../lib/libopencv_videoio.so
/usr/bin/ld: CMakeFiles/opencv_videoio.dir/src/cap_libcamera.cpp.o: in function `cv::CvCapture_libcamera_proxy::getProperty(int) const':
cap_libcamera.cpp:(.text._ZNK2cv25CvCapture_libcamera_proxy11getPropertyEi+0x38): undefined reference to `libcamera::Request::sequence() const'
/usr/bin/ld: CMakeFiles/opencv_videoio.dir/src/cap_libcamera.cpp.o: in function `cv::CvCapture_libcamera_proxy::open()':
cap_libcamera.cpp:(.text._ZN2cv25CvCapture_libcamera_proxy4openEv+0x260): undefined reference to `libcamera::Request::addBuffer(libcamera::Stream const*, libcamera::FrameBuffer*, std::unique_ptr<libcamera::Fence, std::default_delete<libcamera::Fence> >)'
/usr/bin/ld: CMakeFiles/opencv_videoio.dir/src/cap_libcamera.cpp.o: in function `cv::CvCapture_libcamera_proxy::grabFrame()':
cap_libcamera.cpp:(.text._ZN2cv25CvCapture_libcamera_proxy9grabFrameEv+0x4c): undefined reference to `libcamera::Camera::generateConfiguration(libcamera::Span<libcamera::StreamRole const, 18446744073709551615ul>)'
collect2: error: ld returned 1 exit status
make[2]: *** [modules/videoio/CMakeFiles/opencv_videoio.dir/build.make:324: lib/libopencv_videoio.so.5.0.0] Error 1
make[1]: *** [CMakeFiles/Makefile2:2737: modules/videoio/CMakeFiles/opencv_videoio.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

asmorkalov · 2025-03-26T11:17:44Z

modules/videoio/src/cap_libcamera.cpp

+    int error;
+    if (buffer->planes().empty())
+    {
+        std::cerr << "Buffer has no planes " << std::endl;


Use CV_LOG_XXX macro for reporting.

asmorkalov · 2025-03-26T11:18:02Z

modules/videoio/src/cap_libcamera.cpp

+                    std::cerr << "plane is out of buffer: "
+                     << "buffer length=" << length
+                     << ", plane offset=" << plane.offset
+                     << ", plane length=" << plane.length << std::endl;


The same for CV_LOG_XXX

advait-0 · 2025-03-26T17:37:41Z

Raspberry Pi 4, libcamera from git default branch as in gist. CMake:

--   Video I/O:
--     FFMPEG:                      NO
--       avcodec:                   NO
--       avformat:                  NO
--       avutil:                    NO
--       swscale:                   NO
--       avresample:                NO
--     GStreamer:                   YES (1.18.4)
--     v4l/v4l2:                    YES (linux/videodev2.h)
--     libcamera:                   YES (0.4.0)

Build:

[ 53%] Linking CXX shared library ../../lib/libopencv_videoio.so
/usr/bin/ld: CMakeFiles/opencv_videoio.dir/src/cap_libcamera.cpp.o: in function `cv::CvCapture_libcamera_proxy::getProperty(int) const':
cap_libcamera.cpp:(.text._ZNK2cv25CvCapture_libcamera_proxy11getPropertyEi+0x38): undefined reference to `libcamera::Request::sequence() const'
/usr/bin/ld: CMakeFiles/opencv_videoio.dir/src/cap_libcamera.cpp.o: in function `cv::CvCapture_libcamera_proxy::open()':
cap_libcamera.cpp:(.text._ZN2cv25CvCapture_libcamera_proxy4openEv+0x260): undefined reference to `libcamera::Request::addBuffer(libcamera::Stream const*, libcamera::FrameBuffer*, std::unique_ptr<libcamera::Fence, std::default_delete<libcamera::Fence> >)'
/usr/bin/ld: CMakeFiles/opencv_videoio.dir/src/cap_libcamera.cpp.o: in function `cv::CvCapture_libcamera_proxy::grabFrame()':
cap_libcamera.cpp:(.text._ZN2cv25CvCapture_libcamera_proxy9grabFrameEv+0x4c): undefined reference to `libcamera::Camera::generateConfiguration(libcamera::Span<libcamera::StreamRole const, 18446744073709551615ul>)'
collect2: error: ld returned 1 exit status
make[2]: *** [modules/videoio/CMakeFiles/opencv_videoio.dir/build.make:324: lib/libopencv_videoio.so.5.0.0] Error 1

Hmm, I'm not sure why you are facing this issue. Raspberry pi also ships with its custom fork of libcamera, could that be causing an issue?

I have tested this on a Raspberry Pi 5 with a Picamera 2 and on my x84_64 Linux machine and it builds fine on both without any errors.

advait-0 · 2025-03-26T17:41:36Z

Let's use FMT_BGR888 by default, if CAP_PROP_MODE is not set by user. It's default OpenCV behaviour.

Yes will do, I wasn't sure what to set as the default as many UVC cameras just sendsMJPEG and YUYV streams, so the BGR888 format crashes it. We will probably need to check the supported camera pixel formats and then update the default before configuring the stream.

Lookplanes · 2025-04-10T06:15:55Z

Thank you for your excellent initiative and contribution in developing a libcamera backend for OpenCV! We have been closely following your work and are very impressed with your progress.
Our team has also been working on a similar effort to integrate libcamera into OpenCV's Video I/O module. Our approach is based on the LCCV project and aims to provide robust and efficient libcamera support for OpenCV users.
While we see great value in your current implementation, we wanted to briefly share some of the key differences in our approach, which might be of interest and potentially beneficial to the project as a whole.
The main distinction in our design lies in how we handle the interaction with the libcamera backend and manage requests. We have adopted a multi-threaded architecture to maintain and interact with the libcamera request queue.
We believe this multi-threaded approach offers several potential advantages:

Improved Responsiveness: Preventing blocking of the main OpenCV thread for a more reactive API.
Enhanced Performance: Better resource utilization and potentially higher frame rates, leveraging libcamera's asynchronous nature.
Greater Flexibility: More refined control over request management for advanced features and customizations.

We have achieved a highly complete implementation with this architecture, and it demonstrates promising performance in our testing. However, we are currently addressing some bugs, particularly related to rapid release and restart scenarios, before we are fully confident in submitting a pull request.

advait-0 · 2025-04-10T06:50:17Z

Thank you for your kind words and insights @Lookplanes. It's nice to hear that your team has made significant progress with your architecture, I'm looking forward to your approach. If you’re open to it, would you be willing to share your source code or repository? I would love to collaborate on developing a solution.

QueenofUSSR · 2025-04-10T16:00:16Z

Thank you for your kind words and insights @Lookplanes. It's nice to hear that your team has made significant progress with your architecture, I'm looking forward to your approach. If you’re open to it, would you be willing to share your source code or repository? I would love to collaborate on developing a solution.

Certainly! Please check opencv-RPiCamera

advait-0 mentioned this pull request Mar 26, 2025

Add support for libcamera #21653

Open

asmorkalov added feature category: videoio(camera) labels Mar 26, 2025

asmorkalov added this to the 5.0-release milestone Mar 26, 2025

asmorkalov reviewed Mar 26, 2025

View reviewed changes

advait-0 added 11 commits April 7, 2025 12:31

additions to CMake for libcamera backend

d67d639

add source file for libcamera videoio

0aef474

add header file for libcamera videoio

4979a90

add libcamera to videoio registry

eaba735

update build_js.py

50cd9f9

add libcamera interface

def9f63

add libcamera specific changes to videoio header

b065cb6

update license to Apache 2.0

b2a44fb

update std:cerr to CV_LOG_XX for logging

3d3b3d9

fix default pixelformat selection

385cd4e

add initial multi-cam support

8be783f

advait-0 force-pushed the libcamera-backend branch from b3e9351 to 8be783f Compare April 7, 2025 07:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] modules/videoio: add libcamera backend for OpenCV Video I/O #27155

[WIP] modules/videoio: add libcamera backend for OpenCV Video I/O #27155

advait-0 commented Mar 26, 2025 •

edited

Loading

asmorkalov commented Mar 26, 2025

asmorkalov commented Mar 26, 2025

asmorkalov commented Mar 26, 2025

asmorkalov Mar 26, 2025

asmorkalov Mar 26, 2025

advait-0 commented Mar 26, 2025

advait-0 commented Mar 26, 2025 •

edited

Loading

Lookplanes commented Apr 10, 2025

advait-0 commented Apr 10, 2025 •

edited

Loading

QueenofUSSR commented Apr 10, 2025

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

[WIP] modules/videoio: add libcamera backend for OpenCV Video I/O #27155

Are you sure you want to change the base?

[WIP] modules/videoio: add libcamera backend for OpenCV Video I/O #27155

Conversation

advait-0 commented Mar 26, 2025 • edited Loading

What works

Known issues

Pull Request Readiness Checklist

asmorkalov commented Mar 26, 2025

asmorkalov commented Mar 26, 2025

asmorkalov commented Mar 26, 2025

asmorkalov Mar 26, 2025

Choose a reason for hiding this comment

asmorkalov Mar 26, 2025

Choose a reason for hiding this comment

advait-0 commented Mar 26, 2025

advait-0 commented Mar 26, 2025 • edited Loading

Lookplanes commented Apr 10, 2025

advait-0 commented Apr 10, 2025 • edited Loading

QueenofUSSR commented Apr 10, 2025

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

advait-0 commented Mar 26, 2025 •

edited

Loading

advait-0 commented Mar 26, 2025 •

edited

Loading

advait-0 commented Apr 10, 2025 •

edited

Loading