-
Notifications
You must be signed in to change notification settings - Fork 28.6k
flutter test: concurrency and performance is very low #168268
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
experiment - putting tmp in RAM- we wouldn't expect much change as no iowait, but just to confirm.
|
Actually - maybe the answer is in the execsnoop output? If the test submission loop is running
Here we can see that the test main process is started as PID 3105579, and that the PPID of all the |
ok - so to give this theory a bit of a try - a small hack. captured the output of the command:
i renamed dart to dart.real, and made dart this small script:
so noticable speedup... |
and putting all the things together:
but we still are a long way away from CPU saturation...
seems there is still an inefficient thing somewhere in the serial section of the code... i tried replacing the bash wrapper script with a rust version but it wasn't much faster. hope that is useful. |
here's the rust for reference - its just vibe-coded!
|
Hi thanks for your reply. The mentioned issue seems to be about something completely different? The issue here is about the test runner, I would like to understand what aspect would be different under fvm? Thanks! |
Since it is an external tool, the way it handles the sdk could be different, hence I would like to know if the same reported behavior / tests you ran also persists without using fvm, ie, just using |
hiya - re-tested using just flutter rather than |
Thanks for the update and test. I'll keep the issue open and label for team's tracking based on the report and behavior observed. |
You can also try --experimental-faster-testing - which is definitely faster but lacks coverage support atm. |
Yes definitely faster, but has some different behaviours due to handling of async in widget tests. It's an aside but enforced fake async isn't necessarily ideal... There is a bug with the faster testing, which is that if the path to the source code includes a "-" character compilation fails. It's due to this character missing in the replacement regex. The faster testing is faster and potentially good, but concerned it's abandoned due to no commits for a long time (like 1year?) I personally don't believe in test coverage reporting and measures, so lack of support is no issue. |
@jonahwilliams are you the right person to loop in for testing perf and --experimental-faster-testing? |
There is also #168657 if useful. |
--experimental-faster-testing is not being worked on. |
and in general the testing stuff is on the backburner for now :( |
Steps to reproduce
Running flutter tests is very slow. tests that take only milliseconds to run, are run slowly in the test runner.
It seems that as well as a pretty inefficient run process, the concurrency settings are not honoured.
Lets take a look at the run process - as i understand it - firstly from the entry point:
flutter test
On different machines that I have access to, for the same test suite, the tests (about 2500) take between 1m30 and 10m to run.
Using my desktop as an example, its an Intel i9-14900KF - It has 24cores, 32 threads, peak freq of 6GHz. This machine also has 128GB RAM, and for storage, NVMe Samsung 990 Pro - its no slouch.
However, when running the tests with any concurrency setting, its impossible to make the CPU use more than 2 cores.
It seems firstly that concurrency has no impact at all, but actually the performance hits a maximum at 4
Running the tests we can see that there is some disk io, but nowhere near any sort of limit- disk reads are basically 0 (cached), disk writes are quite a bit, but still no real issue. On this machine /tmp is on nvme0n1.
We can also see that there is little I/O Wait, and that the CPU is just chilling:
Using
execsnoop
andtcpconnect
we can see what's happening under the hood:So for each test file - we are running
pub deps
, creating a source file in /tmp, compiling it, and running it, and this program is then connecting to a local webserver to do something..Finding 1 - The test runner or the compiler is running
pub deps
many many times - it runs it 3 times beforedartaotruntime
and then once for each test file. This takes a long time - even on a very fast computer.However, this isn't a limiting factor: we just are not running the right number of test processes:
(with flutter test -j 32)
Finding 2 - For whatever reason, the specified concurrency is not honoured. or, Ahmdahl getting in the way, and a thing that should be done in parallel is being done serially by the job submission loop.
Even if we accept that each test must be independently compiled, and forked and connect using web sockets back to the main test co-ordinator, each of which things is a source of slowness... there is something happening in the test runner that means the tests are not running efficiently.
Expected results
Running tests should saturate either CPU or disk if its running properly.
Actual results
System is idle - tests take far too long to run.
Code sample
lots of system outputs here- the specific test code is not very relevant - its applicable to all tests.
Screenshots or Video
No response
Logs
Logs
[Paste your logs here]
Flutter Doctor output
✓] Flutter (Channel stable, 3.29.2, on Ubuntu 24.04.2 LTS 6.11.0-21-generic, locale en_GB.UTF-8) [40ms]
• Flutter version 3.29.2 on channel stable at /home/richja/fvm/versions/3.29.2
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision c236373 (7 weeks ago), 2025-03-13 16:17:06 -0400
• Engine revision 18b71d6
• Dart version 3.7.2
• DevTools version 2.42.3
[!] Android toolchain - develop for Android devices (Android SDK version 34.0.0) [53ms]
• Android SDK at /home/richja/Android/Sdk
✗ cmdline-tools component is missing
Run
path/to/sdkmanager --install "cmdline-tools;latest"
See https://developer.android.com/studio/command-line for more details.
✗ Android license status unknown.
Run
flutter doctor --android-licenses
to accept the SDK licenses.See https://flutter.dev/to/linux-android-setup for more details.
[✓] Chrome - develop for the web [15ms]
• Chrome at google-chrome
[✗] Linux toolchain - develop for Linux desktop [44ms]
✗ clang++ is required for Linux development.
It is likely available from your distribution (e.g.: apt install clang), or can be downloaded from https://releases.llvm.org/
✗ CMake is required for Linux development.
It is likely available from your distribution (e.g.: apt install cmake), or can be downloaded from https://cmake.org/download/
✗ ninja is required for Linux development.
It is likely available from your distribution (e.g.: apt install ninja-build), or can be downloaded from
https://github.com/ninja-build/ninja/releases
• pkg-config version 1.8.1
✗ GTK 3.0 development libraries are required for Linux development.
They are likely available from your distribution (e.g.: apt install libgtk-3-dev)
[!] Android Studio (not installed) [11ms]
• Android Studio not found; download from https://developer.android.com/studio/index.html
(or visit https://flutter.dev/to/linux-android-setup for detailed instructions).
[✓] IntelliJ IDEA Ultimate Edition (version 2025.1) [11ms]
• IntelliJ at /home/richja/.local/share/JetBrains/Toolbox/apps/intellij-idea-ultimate
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
[✓] Connected device (2 available) [68ms]
• Linux (desktop) • linux • linux-x64 • Ubuntu 24.04.2 LTS 6.11.0-21-generic
• Chrome (web) • chrome • web-javascript • Google Chrome 135.0.7049.114
[✓] Network resources [206ms]
• All expected network resources are available.
The text was updated successfully, but these errors were encountered: