Skip to content

Prevent image buffer exhaustion on Android using Qt #938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 28, 2025

Conversation

m7913d
Copy link
Contributor

@m7913d m7913d commented Apr 21, 2025

This commit fixes the following exception:
java.lang.IllegalStateException: maxImages (10) has already been acquired, call #close before acquiring more.

This exception is thrown when the device cannot process the images fast enough.

Note that Qt captures the images in a separate (Java) thread (on Android) and sends them asynchronously to the main (GUI) thread.

This fix moves the processing to a separate thread and discards all new images while another image (frame) is still being processed. This allows Qt to close the Android resources before the buffer is full.

In recent versions of Qt, Qt catches this exception itself (see https:// bugreports.qt.io/browse/QTBUG-116526) by restarting the camera, but frequent restarts may degrade performance.

On non-Android platforms, this fix has the advantage that the GUI remains responsive even if the barcode processing is slow, and that always the most recently captured frame is processed.

Related to #743

This commit fixes the following exception:
java.lang.IllegalStateException: maxImages (10) has already been
acquired, call #close before acquiring more.

This exception is thrown when the device cannot process the images fast
enough.

Note that Qt captures the images in a separate (Java) thread (on
Android) and sends them asynchronously to the main (GUI) thread.

This fix moves the processing to a separate thread and discards all new
images while another image (frame) is still being processed. This allows
Qt to close the Android resources before the buffer is full.

In recent versions of Qt, Qt catches this exception itself (see https://
bugreports.qt.io/browse/QTBUG-116526) by restarting the camera, but
frequent restarts may degrade performance.

On non-Android platforms, this fix has the advantage that the GUI
remains responsive even if the barcode processing is slow, and that
always the most recently captured frame is processed.

Related to zxing-cpp#743
@m7913d
Copy link
Contributor Author

m7913d commented Apr 21, 2025

Note that this PR only fixes the Qt6 implementation. However, I assume Qt5 is affected too by this bug. I'm not sure whether we should still put some effort in Qt5 as Qt5 will be EoS very soon.

@axxel
Copy link
Collaborator

axxel commented Apr 22, 2025

Thanks for your contribution. I'd like to suggest though, to not 'force' the use of a multi-threaded approach on everyone using that header file. What about adding some pre-processor macro like ZXING_QT_READER_MULTITHREADING that needs to be defined before including the header to enable this feature?

@m7913d
Copy link
Contributor Author

m7913d commented Apr 22, 2025

I'm not really against such an approach, but

  • on Android, one should (almost) always use the multi-threaded approach, unless one can ensure the software will only run on hardware that is able to process the barcodes as fast as the frame rate. This is almost impossible to guarantee, as other processes may be running on the device and the barcode processing time itself is not constant (I think).
  • in general, I think it is good practice to avoid running CPU intensive tasks on the GUI thread to keep the GUI responsive at all times. An unresponsive GUI results in a bad user experience. If the user device is busy (with another task), I'm ok with a slower barcode processing, but I would expect the GUI itself being responsive.
  • I wonder why one would want to disable the multi-threading approach. Do you think they are using a device with only a single core? Even cheap smartphones has multi-core processors. Do you want to save battery? I think the frame-skipping feature of the multi-threaded implementation may save more battery than the small overhead of another thread. I wonder if anyone is using Qt / QVideoSink on a platform for which the single threaded approach is desirable.

Therefore, I would rather add an option to disable the multi-threaded approach than expecting the user to enable the multi-threaded approach (especially on Android) or just enable the multi-threaded approach always. Note that for rare cases, one can also call BarcodeReader::Process himself.

What do you prefer?

@axxel
Copy link
Collaborator

axxel commented Apr 22, 2025

Alright. First up: it is actually nice to see that someone takes any interest in this Qt wrapper stuff. Until now, it seemed hardly anyone is using that at all.

Second: I agree with all your points above. My hesitation was mainly caused by a lack of active memory what this code I wrote years ago actually does, plus I didn't properly look at what your change was doing. I somehow thought you made the BarcodeReader::process() slot mandatorily switching the thread context.

After freshening up my memory, I see the situation as follows. BarcodeReader currently serves 3 use cases:

  1. provide the ReaderOptions API for initial configuation
  2. the process(QVideoFrame) slot for synchronously scanning one frame for barcodes
  3. the setVideoSink() function to setup contiguous scanning

I agree that the 3rd use-case should be asynchronous by default, and also be limited to processing only the most recent frame(s) by default. This gives the best out-of-the-box user experience. With that established, I wonder whether there is room for improving your PR by maybe making use of a QThreadPool? Ether the "global" application wide one or an instance owned by the BarcodeReader. This should then allow to be more flexible with respect to how many frames might be processed in parallel. This might be overkill, but I wanted to float the idea. Either way, I'd like to propose to look into doing away with the BarcodeProcessor class, at least not make it a publicly visible entity, and merge it's functionality with the BarcodeReader.

If you would look into how this could be implemented neatly, that would be much appreciated.

@m7913d
Copy link
Contributor Author

m7913d commented Apr 22, 2025

Concerning the interest in this repo by the Qt community, I think a lot of them are still using QZXing (based on the original zxing repo). It may be interesting to better highlight that zxing-cpp supports Qt too (and probably has a more performant barcode parser under the hood).

Using QThreadPool is an option too, although I think that (by default) a single processing thread is a good trade off between performance and not exhausting all resources (memory / cpu / battery) or maybe at most two by default to ensure at least one frame is being processed all the time in case the processing of a frame is slightly slower than the frame rate.

Concerning the hiding of implementation details, is it your intention to keep the Qt wrapper header-only?

@axxel
Copy link
Collaborator

axxel commented Apr 23, 2025

I think a lot of them are still using QZXing

In case you are interested in ancient history, you might want to have a look at #75. I had another quick glance at the repo and it seems the project can safely be considered as abandoned.

It may be interesting to better highlight that zxing-cpp supports Qt too

That is true. I was thinking about mentioning Qt in the "wrappers" section of the readme for quite some time. But with the feeling that this is not really used, I did not want to set something in stone. The example code I can change however I see fit without worrying about backward compatibility.

(and probably has a more performant barcode parser under the hood).

Oh, it definitively does ;).

Using QThreadPool is an option too, although I think that (by default) a single processing thread is a good trade off

Sure, the default should be 1. My point was to make it flexible.

Concerning the hiding of implementation details, is it your intention to keep the Qt wrapper header-only?

Not necessarily, if there is a clear advantage of doing something else. The header-only approach makes it trivial to build and distribute the wrapper. If we'd consider the ZXingQtReader.h to be 'releasable', I could simply add it to the core package and everyone could use it out of the box without any build dependency on Qt for the library itself.

@m7913d
Copy link
Contributor Author

m7913d commented Apr 26, 2025

@axxel I changed the implementation to use a QThreadPool with 1 thread by default and removed the BarcodeProcessor class.

@m7913d m7913d force-pushed the pr_fix_android_resource_exhausting branch from a15036a to efb08a0 Compare April 26, 2025 21:04
@axxel
Copy link
Collaborator

axxel commented Apr 27, 2025

That looks really neat. One thing though: I fear ~BarcodeReader() could dead-lock if maxThreadCount() >= 2 and the VideoSink is feeding new frames faster than they can be processed. I believe simply calling setMaxThreadCount(0) before calling waitForDone() should do the trick?

@m7913d
Copy link
Contributor Author

m7913d commented Apr 27, 2025

@axxel I added the requested changes, although I think a dead-lock could not happen as the destruction is called in the same thread as the onVideoFrameChanged method. Nevertheless, setting maxThreadCount to 0 doesn't harm. An alternative may be the clear the sink before calling waitForDone.

@axxel axxel merged commit 7d6d8bb into zxing-cpp:master Apr 28, 2025
18 checks passed
@axxel
Copy link
Collaborator

axxel commented Apr 28, 2025

Thanks a lot for your time. If you have more suggestions on how to improve this qt-thingy or its visibility inside the qt-zxing community, I am all ears.

@m7913d
Copy link
Contributor Author

m7913d commented May 5, 2025

Concerning the Qt visibility, I would:

  • create a new Qt wrapper folder based on ZXingQtReader.h (mark it as experimental if you are not yet happy with the interface) and add relevant "Getting Started" information.
  • mention Qt / QML support in your README
  • create a PR to the original ZXing library to include Qt / QML in the description. It may be useful to also include that QZXing is based on the deprecated C++ port and that SCodes is an alternative Qt/QML wrapper that uses zxing-cpp

However, note that I'm not an expert in managing visibility of open source libraries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy