@mshabunin @opencv-alalek Could you take a look?

If I remember correctly there might be issues with building FFmpeg plugin on Windows because of dependency on OpenCV.

Looks like VideoWriter behaviour is not consistent here:

• FFmpeg throws exception, if writer config and image does not match
• GStreamer prints warning and ignores frame, if writer config and image does not match:

  opencv/modules/videoio/src/cap_gstreamer.cpp

  Line 2694 in 09c71ae

  else if (input_pix_fmt == GST_VIDEO_FORMAT_GRAY8) {

• MS media foundation just converts input image to target format independently on writer config:

  opencv/modules/videoio/src/cap_msmf.cpp

  Line 2700 in 09c71ae

  cv::cvtColor(img.getMat(), cv::Mat(videoHeight, videoWidth, CV_8UC4, pData, cbWidth), img.channels() > 1 ? cv::COLOR_BGR2BGRA : cv::COLOR_GRAY2BGRA);

• Android writer ignores configuration and just convert data to expected format:

  opencv/modules/videoio/src/cap_android_mediandk.cpp

  Line 565 in 09c71ae

  #if __ANDROID_API__ >= 26

@asmorkalov @mshabunin @opencv-alalek

Thank you for the feedback. I agree that consistency across backends is important, and I think the best approach is to convert data/input to the desired format, as done in MS media foundation and Android.

Key changes:

FFmpeg:

• Instead of what I previously suggested in my commit, we should perform the needed conversions in writeFrame() in cap_ffmpeg_impl.hpp when checking parameters. (See conversion logic below).
• Also, since the only supported formats are CV_8UC1/CV_8UC3/CV_16UC1, we should add a verification for that in write() at cap_ffmpeg.cpp and do a warning in case of non-supported formats.
• In writeFrame(), instead of receiving only the number of channels (cn), we receive the input depth/type to make the conversions easier.

GStreamer:

• Implement similar conversion logic.

What the conversions would look like in both backends:

cv::Mat mat(height, width, type, (void*)data, step);
if (input_pix_fmt == AV_PIX_FMT_BGR24) {
        if (type != CV_8UC3) {
            if (type == CV_16UC1) // CV_16UC1 -> CV_8UC1
            	mat.convertTo(mat, CV_8UC1, 1.0 / 256); // scale down 16-bit to 8-bit
            cv::cvtColor(mat, mat, COLOR_GRAY2BGR); // CV_8UC1 -> CV_8UC3
            data = mat.data;
            step = (int)mat.step;
            type = CV_8UC3;
        }
}
else if (input_pix_fmt == AV_PIX_FMT_GRAY8) {
	if (type != CV_8UC1) {
            if (type == CV_8UC3) // CV_8UC3 -> CV_8UC1
            	cv::cvtColor(mat, mat, COLOR_BGR2GRAY);
            else // CV_16UC1 -> CV_8UC1
            	mat.convertTo(mat, CV_8UC1, 1.0 / 256); // scale down 16-bit to 8-bit
     	    data = mat.data;
            step = (int)mat.step;
            type = CV_8UC1;
	}
}
else if (input_pix_fmt == AV_PIX_FMT_GRAY16LE) {
	if (type != CV_16UC1) {
    	    if (type == CV_8UC3)
    	    	cv::cvtColor(mat, mat, COLOR_BGR2GRAY);  // CV_8UC3 -> CV_8UC1
    	    mat.convertTo(mat, CV_16UC1);  // CV_8UC1 -> CV_16UC1
    	    data = mat.data;
            step = (int)mat.step;
            type = CV_16UC1;
	}
}
else {
	CV_LOG_WARNING(NULL, "Input data does not match selected pixel format: "
        	<< av_get_pix_fmt_name(input_pix_fmt)
                << ", depth: " << type);
        CV_Assert(false);
}

Test Cases to Add:

• isColor=false with CV_8UC3 (already done in my PR)
• isColor=true with CV_8UC1
• isColor=true with CV_16UC1
• ...

Does this approach seem reasonable? I'm ready to implement these changes.

Hi @asmorkalov, is there anything I should address or clarify to help move this PR forward?

In my last message, I suggested aligning the behavior of the FFmpeg and GStreamer backends with that of MS Media Foundation and Android — handling input conversion internally.

Specifically, the idea is to perform the necessary conversions inside the writeFrame() function for FFmpeg (and similarly for GStreamer). Since only three input types are supported (CV_8UC1, CV_8UC3, and CV_16UC1), these conversions should be straightforward.

I also plan to add test cases to cover these scenarios and make sure the conversions behave as expected.

Please let me know if anything is unclear or if you’d prefer a different approach. I’m ready to make any needed adjustments and move forward with the implementation.

Hi @asmorkalov @mshabunin @opencv-alalek,

Since I didn’t hear back, I decided to move forward and implement the changes I previously proposed. The update improves consistency between the FFmpeg backend and others like MS Media Foundation and Android.

Specifically:

• The conversion logic has been moved into CvVideoWriter_FFMPEG::writeFrame().
• All relevant format conversion cases are now handled.
• Test cases were added to cover all supported conversion scenarios.
• The fix for AVERROR(EAGAIN) handling remains, along with a warning when writeFrame returns false.

Please let me know if you’d like any changes or have further feedback.

@asmorkalov @opencv-alalek @mshabunin friendly reminder

Silent ignorance of the error is a design bug.

But implicit data manipulation is an evil too. There are no safe operations, except may be COLOR_GRAY2BGR. So required conversions should be left on the user side.

.write() should fail properly instead if input is not expected.

Given that there are no safe operations, the best approach is indeed to leave format handling to the user. Since .write() is a void function, changing its return type would require a broader change (though it could be beneficial). Instead, this will be addressed through clearer warnings:

• One when writeFrame() fails to write the frame (already in previous commits)
• Another explicitly reports a mismatch between expected and received input types, similar to the GStreamer backend. (Will add this in the next commit)
  This avoids silent failures and makes the frame skipping behaviour more transparent to users.

Hi @opencv-alalek @mshabunin,

I've pushed a new version addressing @opencv-alalek’s feedback. All required conversions should now be handled by the user. I kept the type argument in the writeFrame() function to allow for more specific warnings. I’ve also added three test cases to validate that VideoWriter fails in case of mismatched types and that appropriate warnings are logged.

However, on Win64, one test case fails: when the input is CV_8UC1 and the expected depth is CV_16UC1. Despite the mismatch, the writer still successfully writes the video, and the file can be read. I suspect this is because, in this particular case, encode_video = false, which skips the input/expected types check and directly encapsulates the frame data, or some other implicit conversion is happening, etc.

I would appreciate your input, as the other two test cases behave as expected by failing due to the mismatch. Is this platform-specific behaviour expected, or am I missing something?

Thanks!

Edit:
The CI failed on the Linux/Ubuntu 22.04 job in two videoio_ffmpeg_16bit.basic test cases. However, I tested this locally and all tests passed. Because I can't reproduce the failure on my machine, it's difficult to diagnose or fix the issue.

My System Information:
OS: Pop!_OS 22.04 LTS (Jammy)
OpenCV version: 4.12.0 (local build)
Compiler: GCC 15.1.0