Skip to content

flutter test: concurrency and performance is very low #168268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
time4tea opened this issue May 3, 2025 · 16 comments
Open

flutter test: concurrency and performance is very low #168268

time4tea opened this issue May 3, 2025 · 16 comments
Labels
a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. P3 Issues that are less important to the Flutter project perf: speed Performance issues related to (mostly rendering) speed team-framework Owned by Framework team triaged-framework Triaged by Framework team

Comments

@time4tea
Copy link

time4tea commented May 3, 2025

Steps to reproduce

Running flutter tests is very slow. tests that take only milliseconds to run, are run slowly in the test runner.

It seems that as well as a pretty inefficient run process, the concurrency settings are not honoured.

Lets take a look at the run process - as i understand it - firstly from the entry point:

flutter test

On different machines that I have access to, for the same test suite, the tests (about 2500) take between 1m30 and 10m to run.

Using my desktop as an example, its an Intel i9-14900KF - It has 24cores, 32 threads, peak freq of 6GHz. This machine also has 128GB RAM, and for storage, NVMe Samsung 990 Pro - its no slouch.

However, when running the tests with any concurrency setting, its impossible to make the CPU use more than 2 cores.

time flutter test
real	1m38.025s
user	4m9.351s
sys	1m1.180s

time flutter test -j 32
real	1m38.110s
user	4m9.876s
sys	1m1.392s

time flutter test -j 100
real	1m37.919s
user	4m8.537s
sys	1m1.240s

It seems firstly that concurrency has no impact at all, but actually the performance hits a maximum at 4

time flutter test -j 1
real	3m44.300s
user	3m52.690s
sys	0m59.870s

time flutter test -j 4
real	1m39.793s
user	4m7.636s
sys	1m0.708s

Running the tests we can see that there is some disk io, but nowhere near any sort of limit- disk reads are basically 0 (cached), disk writes are quite a bit, but still no real issue. On this machine /tmp is on nvme0n1.

03/05/25 13:35:29
     r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util Device
    0.00      0.0k     0.00   0.0%    0.00     0.0k 1442.50    170.9M     1.50   0.1%   12.28   121.3k    0.00      0.0k     0.00   0.0%    0.00     0.0k    0.00    0.00   17.72   7.6% nvme0n1

We can also see that there is little I/O Wait, and that the CPU is just chilling:

sar -u 1
Linux 6.11.0-21-generic (minnow) 	03/05/25 	_x86_64_	(32 CPU)

14:01:08        CPU     %user     %nice   %system   %iowait    %steal     %idle
14:01:09        all      6.34      0.00      2.16      0.00      0.00     91.50
14:01:10        all      6.59      0.00      2.23      0.03      0.00     91.15
14:01:11        all      5.67      0.00      2.22      0.00      0.00     92.11
14:01:12        all      7.71      0.00      1.86      0.00      0.00     90.43
14:01:13        all      8.22      0.00      2.17      0.00      0.00     89.61
14:01:14        all      6.14      0.00      1.83      0.00      0.00     92.04
14:01:15        all      5.55      0.00      2.17      0.00      0.00     92.28
14:01:16        all      6.22      0.00      1.92      0.00      0.00     91.86
14:01:17        all      6.12      0.00      1.98      0.00      0.00     91.90
14:01:18        all      5.90      0.00      2.23      0.00      0.00     91.87

Using execsnoop and tcpconnect we can see what's happening under the hood:

87.130  dart             3066280 3060994   0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json 
87.133  flutter_tester   3066284 3060994   0 /home/richja/fvm/versions/3.29.2/bin/cache/artifacts/engine/linux-x64/flutter_tester --disable-vm-service --enable-checked-mode --verify-entry-points --enable-software-rendering --skia-deterministic-rendering --enable-dart-profiling --non-interactive --use-test-fonts --disable-asset-fonts --packages=/home/richja/dev/app-ui/templates/app_app/.dart_tool/package_config.json --flutter-assets-dir=/home/richja/dev/app-ui/templates/app_app/build/unit_test_assets /tmp/flutter_tools.VFBOSQ/flutter_test_listener.QRNYMJ/listener.dart.dill 
87.453  dart             3066298 3060994   0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json 
87.456  flutter_tester   3066300 3060994   0 /home/richja/fvm/versions/3.29.2/bin/cache/artifacts/engine/linux-x64/flutter_tester --disable-vm-service --enable-checked-mode --verify-entry-points --enable-software-rendering --skia-deterministic-rendering --enable-dart-profiling --non-interactive --use-test-fonts --disable-asset-fonts --packages=/home/richja/dev/app-ui/templates/app_app/.dart_tool/package_config.json --flutter-assets-dir=/home/richja/dev/app-ui/templates/app_app/build/unit_test_assets /tmp/flutter_tools.VFBOSQ/flutter_test_listener.IQKCTK/listener.dart.dill 
87.774  dart             3066318 3060994   0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json 
87.777  flutter_tester   3066320 3060994   0 /home/richja/fvm/versions/3.29.2/bin/cache/artifacts/engine/linux-x64/flutter_tester --disable-vm-service --enable-checked-mode --verify-entry-points --enable-software-rendering --skia-deterministic-rendering --enable-dart-profiling --non-interactive --use-test-fonts --disable-asset-fonts --packages=/home/richja/dev/app-ui/templates/app_app/.dart_tool/package_config.json --flutter-assets-dir=/home/richja/dev/app-ui/templates/app_app/build/unit_test_assets /tmp/flutter_tools.VFBOSQ/flutter_test_listener.SLQPWE/listener.dart.dill 
3049415 flutter_test 4  127.0.0.1        127.0.0.1        42051
3049439 flutter_test 4  127.0.0.1        127.0.0.1        43303
3049456 flutter_test 4  127.0.0.1        127.0.0.1        33745
3049476 flutter_test 4  127.0.0.1        127.0.0.1        36363
3049495 flutter_test 4  127.0.0.1        127.0.0.1        46323
3049514 flutter_test 4  127.0.0.1        127.0.0.1        34249
3049538 flutter_test 4  127.0.0.1        127.0.0.1        35085
3049556 flutter_test 4  127.0.0.1        127.0.0.1        41743
3049576 flutter_test 4  127.0.0.1        127.0.0.1        35867
3049594 flutter_test 4  127.0.0.1        127.0.0.1        35403

So for each test file - we are running pub deps, creating a source file in /tmp, compiling it, and running it, and this program is then connecting to a local webserver to do something..

Finding 1 - The test runner or the compiler is running pub deps many many times - it runs it 3 times before dartaotruntime and then once for each test file. This takes a long time - even on a very fast computer.

time /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json > /dev/null
real	0m0.218s
user	0m0.303s
sys	0m0.060s

However, this isn't a limiting factor: we just are not running the right number of test processes:
(with flutter test -j 32)

while true; do count=$(ps -elf | grep flutter_tester | grep -v grep | wc -l ); echo $(date +"%T.%3N") Count = $count; sleep 0.1; done
13:55:16.578 Count = 0
13:55:16.703 Count = 1
13:55:16.829 Count = 1
13:55:16.950 Count = 2
13:55:17.076 Count = 2
13:55:17.201 Count = 2
13:55:17.326 Count = 2
13:55:17.448 Count = 1
13:55:17.582 Count = 2
13:55:17.712 Count = 2

Finding 2 - For whatever reason, the specified concurrency is not honoured. or, Ahmdahl getting in the way, and a thing that should be done in parallel is being done serially by the job submission loop.

Even if we accept that each test must be independently compiled, and forked and connect using web sockets back to the main test co-ordinator, each of which things is a source of slowness... there is something happening in the test runner that means the tests are not running efficiently.

Expected results

Running tests should saturate either CPU or disk if its running properly.

Actual results

System is idle - tests take far too long to run.

Code sample

lots of system outputs here- the specific test code is not very relevant - its applicable to all tests.

Screenshots or Video

No response

Logs

Logs
[Paste your logs here]

Flutter Doctor output

✓] Flutter (Channel stable, 3.29.2, on Ubuntu 24.04.2 LTS 6.11.0-21-generic, locale en_GB.UTF-8) [40ms]
• Flutter version 3.29.2 on channel stable at /home/richja/fvm/versions/3.29.2
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision c236373 (7 weeks ago), 2025-03-13 16:17:06 -0400
• Engine revision 18b71d6
• Dart version 3.7.2
• DevTools version 2.42.3

[!] Android toolchain - develop for Android devices (Android SDK version 34.0.0) [53ms]
• Android SDK at /home/richja/Android/Sdk
✗ cmdline-tools component is missing
Run path/to/sdkmanager --install "cmdline-tools;latest"
See https://developer.android.com/studio/command-line for more details.
✗ Android license status unknown.
Run flutter doctor --android-licenses to accept the SDK licenses.
See https://flutter.dev/to/linux-android-setup for more details.

[✓] Chrome - develop for the web [15ms]
• Chrome at google-chrome

[✗] Linux toolchain - develop for Linux desktop [44ms]
✗ clang++ is required for Linux development.
It is likely available from your distribution (e.g.: apt install clang), or can be downloaded from https://releases.llvm.org/
✗ CMake is required for Linux development.
It is likely available from your distribution (e.g.: apt install cmake), or can be downloaded from https://cmake.org/download/
✗ ninja is required for Linux development.
It is likely available from your distribution (e.g.: apt install ninja-build), or can be downloaded from
https://github.com/ninja-build/ninja/releases
• pkg-config version 1.8.1
✗ GTK 3.0 development libraries are required for Linux development.
They are likely available from your distribution (e.g.: apt install libgtk-3-dev)

[!] Android Studio (not installed) [11ms]
• Android Studio not found; download from https://developer.android.com/studio/index.html
(or visit https://flutter.dev/to/linux-android-setup for detailed instructions).

[✓] IntelliJ IDEA Ultimate Edition (version 2025.1) [11ms]
• IntelliJ at /home/richja/.local/share/JetBrains/Toolbox/apps/intellij-idea-ultimate
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] Connected device (2 available) [68ms]
• Linux (desktop) • linux • linux-x64 • Ubuntu 24.04.2 LTS 6.11.0-21-generic
• Chrome (web) • chrome • web-javascript • Google Chrome 135.0.7049.114

[✓] Network resources [206ms]
• All expected network resources are available.

@time4tea
Copy link
Author

time4tea commented May 3, 2025

experiment - putting tmp in RAM- we wouldn't expect much change as no iowait, but just to confirm.

sudo mkdir /media/ramdisk
sudo mount -t tmpfs -o size=32G tmpfs /media/ramdisk
export TMPDIR=/media/ramdisk
time fvm flutter test --concurrency 32
real	1m36.530s
user	4m16.528s
sys	0m57.552s

@time4tea
Copy link
Author

time4tea commented May 3, 2025

Actually - maybe the answer is in the execsnoop output?

If the test submission loop is running /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json for each test (suite) and that takes approx 200ms, then we would expect to see 4 to 5 max test suites started per second, which is what we see?

3.419   dart             3105579 3105578   0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart --packages=/home/richja/fvm/versions/3.29.2/packages/flutter_tools/.dart_tool/package_config.json /home/richja/fvm/versions/3.29.2/bin/cache/flutter_tools.snapshot test --concurrency 32 
3.455   uname            3105602 3105579   0 /usr/bin/uname -m 
3.461   git              3105605 3105579   0 /usr/bin/git -c log.showSignature=false log HEAD -n 1 --pretty=format:%ad --date=iso 
3.543   dart             3105608 3105579   0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json 
3.846   dart             3105618 3105579   0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json 
4.558   dart             3105637 3105579   0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json 
4.804   dartaotruntime   3105646 3105579   0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dartaotruntime /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/snapshots/frontend_server_aot.dart.snapshot --sdk-root /home/richja/fvm/versions/3.29.2/bin/cache/artifacts/engine/common/flutter_patched_sdk/ --incremental --no-print-incremental-dependencies --target=flutter --experimental-emit-debug-metadata --output-dill /media/ramdisk/flutter_tools.LDDGGJ/flutter_test_compiler.AGNWRX/output.dill --packages /home/richja/dev/app-ui/templates/app_app/.dart_tool/package_config.json -Ddart.vm.profile=false -Ddart.vm.product=false --enable-asserts --track-widget-creation --initialize-from-dill /home/richja/dev/app-ui/templates/app_app/build/test_cache/build/cache.dill.track.dill --verbosity=error 
5.713   dart             3105656 3105579   0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json 
5.715   flutter_tester   3105657 3105579   0 /home/richja/fvm/versions/3.29.2/bin/cache/artifacts/engine/linux-x64/flutter_tester --disable-vm-service --enable-checked-mode --verify-entry-points --enable-software-rendering --skia-deterministic-rendering --enable-dart-profiling --non-interactive --use-test-fonts --disable-asset-fonts --packages=/home/richja/dev/app-ui/templates/app_app/.dart_tool/package_config.json --flutter-assets-dir=/home/richja/dev/app-ui/templates/app_app/build/unit_test_assets /media/ramdisk/flutter_tools.LDDGGJ/flutter_test_listener.MEPDDW/listener.dart.dill 
6.005   dart             3105674 3105579   0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json 
6.007   flutter_tester   3105676 3105579   0 /home/richja/fvm/versions/3.29.2/bin/cache/artifacts/engine/linux-x64/flutter_tester --disable-vm-service --enable-checked-mode --verify-entry-points --enable-software-rendering --skia-deterministic-rendering --enable-dart-profiling --non-interactive --use-test-fonts --disable-asset-fonts --packages=/home/richja/dev/app-ui/templates/app_app/.dart_tool/package_config.json --flutter-assets-dir=/home/richja/dev/app-ui/templates/app_app/build/unit_test_assets /media/ramdisk/flutter_tools.LDDGGJ/flutter_test_listener.UFTYZJ/listener.dart.dill 
6.290   dart             3105699 3105579   0 /home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json 
6.292   flutter_tester   3105701 3105579   0 /home/richja/fvm/versions/3.29.2/bin/cache/artifacts/engine/linux-x64/flutter_tester --disable-vm-service --enable-checked-mode --verify-entry-points --enable-software-rendering --skia-deterministic-rendering --enable-dart-profiling --non-interactive --use-test-fonts --disable-asset-fonts --packages=/home/richja/dev/app-ui/templates/app_app/.dart_tool/package_config.json --flutter-assets-dir=/home/richja/dev/app-ui/templates/app_app/build/unit_test_assets /media/ramdisk/flutter_tools.LDDGGJ/flutter_test_listener.WRCJUU/listener.dart.dill 

Here we can see that the test main process is started as PID 3105579, and that the PPID of all the dart pub is that same process... ??

@time4tea
Copy link
Author

time4tea commented May 3, 2025

ok - so to give this theory a bit of a try - a small hack.

captured the output of the command:

/home/richja/fvm/versions/3.29.2/bin/cache/dart-sdk/bin/dart pub --suppress-analytics deps --json > hackdeps.txt

i renamed dart to dart.real, and made dart this small script:

#!/bin/bash

script_path="$0"
script_dir="$(dirname "$script_path")"


if [[ "$1" == "pub" && "$2" == "--suppress-analytics" && "$3" == "deps" && "$4" == "--json" && $# -eq 4 ]]; then
        if [[ "$(pwd)" == "/home/richja/dev/app-ui/templates/app_app" ]]; then
                exec cat hackdeps.txt
        fi
fi

exec $script_dir/dart.real "$@"
time fvm flutter test
real	0m43.378s
user	3m0.204s
sys	0m51.929s

so noticable speedup...

@time4tea
Copy link
Author

time4tea commented May 3, 2025

and putting all the things together:

export TMPDIR=/media/ramdisk
time fvm flutter test -j 32

real	0m39.272s
user	3m2.901s
sys	0m46.583s

but we still are a long way away from CPU saturation...

18:37:23        CPU     %user     %nice   %system   %iowait    %steal     %idle
18:37:27        all     30.62      0.00      5.23      0.00      0.00     64.15

seems there is still an inefficient thing somewhere in the serial section of the code... i tried replacing the bash wrapper script with a rust version but it wasn't much faster.

hope that is useful.

@time4tea
Copy link
Author

time4tea commented May 3, 2025

here's the rust for reference - its just vibe-coded!

use std::env;
use std::fs;
use std::os::unix::process::CommandExt;
use std::path::Path;
use std::process::{Command, exit};

fn main() {
    let args: Vec<String> = env::args().collect();

    // Check for exact match of arguments
    if args.len() == 5
        && args[1] == "pub"
        && args[2] == "--suppress-analytics"
        && args[3] == "deps"
        && args[4] == "--json"
    {
        if let Ok(cwd) = env::current_dir() {
            if cwd == Path::new("/home/richja/dev/app-ui/templates/app_app") {
                     match File::open("hackdeps.txt") {
                    Ok(file) => {
                        let mut reader = BufReader::new(file);
                        let stdout = io::stdout();
                        let mut handle = stdout.lock();

                        match io::copy(&mut reader, &mut handle) {
                            Ok(_) => exit(0),
                            Err(e) => {
                                eprintln!("Error writing to stdout: {}", e);
                                exit(1);
                            }
                        }
                    }
                    Err(e) => {
                        eprintln!("Error opening hackdeps.txt: {}", e);
                        exit(1);
                    }
                }
            }
        }
    }

    // Determine the script directory
    let exec_path = match env::current_exe() {
        Ok(path) => path,
        Err(e) => {
            eprintln!("Failed to determine executable path: {}", e);
            exit(1);
        }
    };

    let exec_dir = exec_path.parent().unwrap_or_else(|| Path::new("."));
    let dart_path = exec_dir.join("dart.real");

    // Exec dart.real with the original arguments (excluding program name)
    let err = Command::new(&dart_path)
        .args(&args[1..])
        .exec(); // replaces current process

    // If exec fails, report the error
    eprintln!("Failed to exec {:?}: {}", dart_path, err);
    exit(1);
}

@darshankawar darshankawar added the in triage Presently being triaged by the triage team label May 5, 2025
@darshankawar
Copy link
Member

Thanks for the detailed report and analysis @time4tea
I see that you are using fvm, not sure if it could be related, but does the same behavior occur without using fvm setup ?
Also, can you check this issue to see if it helps further in your case or not ?

@darshankawar darshankawar added the waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds label May 5, 2025
@time4tea
Copy link
Author

time4tea commented May 5, 2025

Hi thanks for your reply.

The mentioned issue seems to be about something completely different?

The issue here is about the test runner, I would like to understand what aspect would be different under fvm?

Thanks!

@github-actions github-actions bot removed the waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds label May 5, 2025
@darshankawar
Copy link
Member

I would like to understand what aspect would be different under fvm?

Since it is an external tool, the way it handles the sdk could be different, hence I would like to know if the same reported behavior / tests you ran also persists without using fvm, ie, just using flutter test instead of fvm flutter test. This is to narrow down the behavior.

@darshankawar darshankawar added the waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds label May 6, 2025
@time4tea
Copy link
Author

time4tea commented May 6, 2025

hiya - re-tested using just flutter rather than fvm and as expected the behaviour is identical.

@github-actions github-actions bot removed the waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds label May 6, 2025
@darshankawar
Copy link
Member

Thanks for the update and test. I'll keep the issue open and label for team's tracking based on the report and behavior observed.

@darshankawar darshankawar added a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. team-framework Owned by Framework team perf: speed Performance issues related to (mostly rendering) speed and removed in triage Presently being triaged by the triage team labels May 7, 2025
@cmkweber
Copy link

cmkweber commented May 7, 2025

You can also try --experimental-faster-testing - which is definitely faster but lacks coverage support atm.

@time4tea
Copy link
Author

time4tea commented May 7, 2025

Yes definitely faster, but has some different behaviours due to handling of async in widget tests. It's an aside but enforced fake async isn't necessarily ideal...

There is a bug with the faster testing, which is that if the path to the source code includes a "-" character compilation fails. It's due to this character missing in the replacement regex.

The faster testing is faster and potentially good, but concerned it's abandoned due to no commits for a long time (like 1year?)

I personally don't believe in test coverage reporting and measures, so lack of support is no issue.

@justinmc
Copy link
Contributor

@jonahwilliams are you the right person to loop in for testing perf and --experimental-faster-testing?

@time4tea
Copy link
Author

There is also #168657 if useful.

@jonahwilliams
Copy link
Member

--experimental-faster-testing is not being worked on.

@jonahwilliams
Copy link
Member

and in general the testing stuff is on the backburner for now :(

@justinmc justinmc added P3 Issues that are less important to the Flutter project triaged-framework Triaged by Framework team labels May 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. P3 Issues that are less important to the Flutter project perf: speed Performance issues related to (mostly rendering) speed team-framework Owned by Framework team triaged-framework Triaged by Framework team
Projects
None yet
Development

No branches or pull requests

5 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy