Skip to content

[integration_test] If tests fail, integration_response_data.json is not created #74391

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
vrnvorona opened this issue Jan 21, 2021 · 12 comments
Open

[integration_test] If tests fail, integration_response_data.json is not created #74391

vrnvorona opened this issue Jan 21, 2021 · 12 comments
Labels
a: tests "flutter test", flutter_test, or one of our tests f: integration_test The flutter/packages/integration_test plugin found in release: 3.19 Found to occur in 3.19 found in release: 3.22 Found to occur in 3.22 framework flutter/packages/flutter repository. See also f: labels. has reproducible steps The issue has been confirmed reproducible and is ready to work on P2 Important issues not at the top of the work list team-framework Owned by Framework team triaged-framework Triaged by Framework team

Comments

@vrnvorona
Copy link

vrnvorona commented Jan 21, 2021 •
edited by TahaTesser
Loading

Steps to Reproduce

  1. Add this test to example test in master branch of plugins repo:
testWidgets('failure 2', (WidgetTester tester) async {
    (tester.binding as IntegrationTestWidgetsFlutterBinding).reportData = {'1': 1};
    expect(1 + 1, 2);
  });

and integration_test.dart:

import 'package:integration_test/integration_test_driver.dart' as integration_test_driver;

Future<void> main() {
  integration_test_driver.testOutputsDirectory = 'integration_test/gherkin/reports';

  return integration_test_driver.integrationDriver();
}
  1. Run tests
  2. Change expect(1 + 1, 2); to expect(1 + 1, 3);
  3. Delete integration_response_data.json
  4. Run tests

Expected results: On step 2 i expect {"1": 1} json file. On step 4 i expect same file to be created again

Actual results: On step 2 everything is fine, on step 4 file is not created.

Reason: Because responseDataCallback is called only if response.allTestsPassed is true (which is not the case if tests fail)(https://github.com/flutter/plugins/blob/f3024731b090659edaa92d01416549c690f65678/packages/integration_test/lib/integration_test_driver.dart#L81). But even if we call that callback here and fix "file not created" problem, there is another one:
response.data will be null. Exact reason is unknown to me yet, but this callback is called earlier than tests or frameworks can write to binding.reportData. So, if i add await responseDataCallback(response.data); to else statement on L82, file generated will contain just null despite what we passed in tests. This blocks reporting ability of testing frameworks when tests fail, because they can't write files themselves and rely on integration_test_driver to do it.

Logs 
Using device iPhone 8 Plus.
Starting application: test_driver/failure.dart
Warning: Missing build name (CFBundleShortVersionString).
Warning: Missing build number (CFBundleVersion).
Action Required: You must set a build name and number in the pubspec.yaml file version field before submitting to the App Store.
Running Xcode build...                                                  
 └─Compiling, linking and signing...                         4.3s
Xcode build done.                                           16.9s
flutter: Observatory listening on http://127.0.0.1:57170/CD-CwTHQPAM=/
VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:57170/CD-CwTHQPAM=/
VMServiceFlutterDriver: Isolate found with number: 2615774461649659
VMServiceFlutterDriver: Isolate is paused at start.
VMServiceFlutterDriver: Attempting to resume isolate
flutter: 00:00 +0: failure 2
VMServiceFlutterDriver: Connected to Flutter application.
flutter: (The following exception is now available via WidgetTester.takeException:)
flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
flutter: The following TestFailure object was thrown running a test:
flutter:   Expected: <3>
flutter:   Actual: <2>
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #4      main.<anonymous closure> (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/failure.dart:26:5)
flutter: #5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:146:29)
flutter: <asynchronous suspension>
flutter: #6      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart)
flutter: #7      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:796:19)
flutter: <asynchronous suspension>
flutter: #10     TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:777:14)
flutter: #11     LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1605:12)
flutter: #12     IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17)
flutter: #13     testWidgets.<anonymous closure> (package:flutter_test/src/widget_tester.dart:138:24)
flutter: #14     Declarer.test.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/declarer.dart:175:19)
flutter: <asynchronous suspension>
flutter: #15     Declarer.test.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/declarer.dart)
flutter: #20     Declarer.test.<anonymous closure> (package:test_api/src/backend/declarer.dart:173:13)
flutter: #21     Invoker.waitForOutstandingCallbacks.<anonymous closure> (package:test_api/src/backend/invoker.dart:231:15)
flutter: #26     Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5)
flutter: #27     Invoker._onRun.<anonymous closure>.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/invoker.dart:383:17)
flutter: <asynchronous suspension>
flutter: #28     Invoker._onRun.<anonymous closure>.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/invoker.dart)
flutter: #33     Invoker._onRun.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/invoker.dart:370:9)
flutter: #34     Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15)
flutter: #35     Invoker._onRun.<anonymous closure> (package:test_api/src/backend/invoker.dart:369:7)
flutter: #42     Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11)
flutter: #43     LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11)
flutter: (elided 31 frames from dart:async and package:stack_trace)
flutter:
flutter: This was caught by the test expectation on the following line:
flutter:   file:///Users/voronin/plugins/packages/integration_test/example/test_driver/failure.dart line 26
flutter: The test description was:
flutter:   failure 2
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════
flutter: (If WidgetTester.takeException is called, the above exception will be ignored. If it is not, then the above exception will be dumped when another exception is caught by the framework or when the test ends, whichever happens
first, and then the test will fail due to having not caught or expected the exception.)
flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
flutter: The following TestFailure object was thrown running a test:
flutter:   Expected: <3>
flutter:   Actual: <2>
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #4      main.<anonymous closure> (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/failure.dart:26:5)
flutter: #5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:146:29)
flutter: <asynchronous suspension>
flutter: #6      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart)
flutter: #7      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:796:19)
flutter: <asynchronous suspension>
flutter: #10     TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:777:14)
flutter: #11     LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1605:12)
flutter: #12     IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17)
flutter: #13     testWidgets.<anonymous closure> (package:flutter_test/src/widget_tester.dart:138:24)
flutter: #14     Declarer.test.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/declarer.dart:175:19)
flutter: <asynchronous suspension>
flutter: #15     Declarer.test.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/declarer.dart)
flutter: #20     Declarer.test.<anonymous closure> (package:test_api/src/backend/declarer.dart:173:13)
flutter: #21     Invoker.waitForOutstandingCallbacks.<anonymous closure> (package:test_api/src/backend/invoker.dart:231:15)
flutter: #26     Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5)
flutter: #27     Invoker._onRun.<anonymous closure>.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/invoker.dart:383:17)
flutter: <asynchronous suspension>
flutter: #28     Invoker._onRun.<anonymous closure>.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/invoker.dart)
flutter: #33     Invoker._onRun.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/invoker.dart:370:9)
flutter: #34     Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15)
flutter: #35     Invoker._onRun.<anonymous closure> (package:test_api/src/backend/invoker.dart:369:7)
flutter: #42     Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11)
flutter: #43     LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11)
flutter: (elided 31 frames from dart:async and package:stack_trace)
flutter:
flutter: This was caught by the test expectation on the following line:
flutter:   file:///Users/voronin/plugins/packages/integration_test/example/test_driver/failure.dart line 26
flutter: The test description was:
flutter:   failure 2
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════
flutter: 00:00 +0: failure 2 [E]
flutter:   Test failed. See exception logs above.
  The test description was: failure 2
flutter:
flutter: 00:00 +0 -1: (tearDownAll)
flutter: Warning: integration_test test plugin was not detected.
flutter: 00:00 +1 -1: Some tests failed.
Failure Details:
Failure in method: failure 2
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
The following TestFailure object was thrown running a test:
  Expected: <3>
  Actual: <2>

When the exception was thrown, this was the stack:
#4      main.<anonymous closure> (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/failure.dart:26:5)
#5      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:146:29)
<asynchronous suspension>
#6      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart)
#7      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:796:19)
<asynchronous suspension>
#10     TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:777:14)
#11     LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1605:12)
#12     IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17)
#13     testWidgets.<anonymous closure> (package:flutter_test/src/widget_tester.dart:138:24)
#14     Declarer.test.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/declarer.dart:175:19)
<asynchronous suspension>
#15     Declarer.test.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/declarer.dart)
#20     Declarer.test.<anonymous closure> (package:test_api/src/backend/declarer.dart:173:13)
#21     Invoker.waitForOutstandingCallbacks.<anonymous closure> (package:test_api/src/backend/invoker.dart:231:15)
#26     Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5)
#27     Invoker._onRun.<anonymous closure>.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/invoker.dart:383:17)
<asynchronous suspension>
#28     Invoker._onRun.<anonymous closure>.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/invoker.dart)
#33     Invoker._onRun.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/invoker.dart:370:9)
#34     Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15)
#35     Invoker._onRun.<anonymous closure> (package:test_api/src/backend/invoker.dart:369:7)
#42     Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11)
#43     LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11)
(elided 31 frames from dart:async and package:stack_trace)

This was caught by the test expectation on the following line:
  file:///Users/voronin/plugins/packages/integration_test/example/test_driver/failure.dart line 26
The test description was:
  failure 2
═════════════════════════════════════════════════════════════════

end of failure 1



Stopping application instance.
Driver tests failed: 1


[✓] Flutter (Channel stable, 1.22.5, on Mac OS X 10.15.7 19H2 darwin-x64, locale en-RU)
    • Flutter version 1.22.5 at /usr/local/flutter
    • Framework revision 7891006299 (6 weeks ago), 2020-12-10 11:54:40 -0800
    • Engine revision ae90085a84
    • Dart version 2.10.4

 
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at /Users/voronin/Library/Android/sdk
    • Platform android-30, build-tools 30.0.2
    • ANDROID_HOME = /Users/voronin/Library/Android/sdk
    • Java binary at: /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9+11)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.1)
    • Xcode at /Applications/Xcode-12.1.app/Contents/Developer
    • Xcode 12.1, Build version 12A7403
    • CocoaPods version 1.10.0

[!] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.dev/docs/get-started/install/macos#android-setup for detailed instructions).

[✓] IntelliJ IDEA Ultimate Edition (version 2020.1)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin version 46.0.3
    • Dart plugin version 201.7223.43

[!] VS Code (version 1.52.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    ✗ Flutter extension not installed; install from
      https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (1 available)
    • iPhone 8 Plus (mobile) • AD6DDB6C-44AB-4655-81C4-AAC5C241F4A2 • ios • com.apple.CoreSimulator.SimRuntime.iOS-13-7 (simulator)

! Doctor found issues in 2 categories.
@vrnvorona vrnvorona mentioned this issue Jan 21, 2021
Open
@TahaTesser
Copy link
Member

TahaTesser commented Jan 21, 2021 •
edited
Loading

Hi @vrnvorona
Adding integration_test_driver.testOutputsDirectory = 'integration_test/gherkin/reports'; and the fail test to the master branch example , step 2 still didn't create the file for me.

flutter doctor -v 
[✓] Flutter (Channel stable, 1.22.5, on macOS 11.1 20C69 darwin-x64, locale en-GB)
    • Flutter version 1.22.5 at /Users/tahatesser/Code/flutter_stable
    • Framework revision 7891006299 (6 weeks ago), 2020-12-10 11:54:40 -0800
    • Engine revision ae90085a84
    • Dart version 2.10.4

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/SDK
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/SDK
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.1

[!] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.1

[✓] Connected device (2 available)
    • RMX2001 (mobile)     • EUYTFEUSQSRGDA6D          • android-arm64 • Android 10 (API 29)
    • Taha’s iPad (mobile) • 00008020-000255113EE8402E • ios           • iOS 14.3

! Doctor found issues in 1 category.
[✓] Flutter (Channel master, 1.26.0-2.0.pre.448, on macOS 11.1 20C69 darwin-x64, locale en-GB)
    • Flutter version 1.26.0-2.0.pre.448 at /Users/tahatesser/Code/flutter_master
    • Framework revision eeb7c261f2 (6 hours ago), 2021-01-20 23:04:03 -0500
    • Engine revision dde6b7e8ea
    • Dart version 2.12.0 (build 2.12.0-246.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/SDK
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/SDK
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.1

[✓] Connected device (4 available)
    • RMX2001 (mobile)     • EUYTFEUSQSRGDA6D          • android-arm64  • Android 10 (API 29)
    • Taha’s iPad (mobile) • 00008020-000255113EE8402E • ios            • iOS 14.3
    • macOS (desktop)      • macos                     • darwin-x64     • macOS 11.1 20C69 darwin-x64
    • Chrome (web)         • chrome                    • web-javascript • Google Chrome 87.0.4280.141

• No issues found!

Can you please provide more details and command you using to run the test?
Thank you

@TahaTesser TahaTesser added in triage Presently being triaged by the triage team waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds labels Jan 21, 2021
@vrnvorona
Copy link
Author

vrnvorona commented Jan 21, 2021 •
edited by TahaTesser
Loading

@TahaTesser I have only integration_test_driver.dart with

import 'package:integration_test/integration_test_driver.dart'
    as integration_test_driver;

Future<void> main() {
  // The Gherkin report data send back to this runner by the app after
  // the tests have run will be saved to this directory
  integration_test_driver.testOutputsDirectory =
      'integration_test/gherkin/reports';

  return integration_test_driver.integrationDriver(
    timeout: const Duration(minutes: 120)
  );
}

and failure.dart with

import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';

void main() {
  IntegrationTestWidgetsFlutterBinding.ensureInitialized();

  testWidgets('failure 2', (WidgetTester tester) async {
    (tester.binding as IntegrationTestWidgetsFlutterBinding).reportData = {'gherkin_reports': [[1]]};
    expect(1 + 1, 2); // This should fail
  });
}

I run them as flutter drive --driver=test_driver/integration_test_driver.dart --target=test_driver/failure.dart

Be ware that report will be created only if expect is expect(1 + 1, 2);, not expect(1 + 1, 3)

@no-response no-response bot removed the waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds label Jan 21, 2021
@TahaTesser
Copy link
Member

Hi @vrnvorona
Thanks for the details! I can reproduce the issue, step 4 doesn't create the file after making changes in the test.

flutter doctor -v 
[✓] Flutter (Channel master, 1.26.0-13.0.pre.118, on macOS 11.1 20C69 darwin-x64, locale en-GB)
    • Flutter version 1.26.0-13.0.pre.118 at /Users/tahatesser/Code/flutter_master
    • Framework revision 08068fd94c (3 minutes ago), 2021-01-21 22:51:03 -0800
    • Engine revision 90641fa923
    • Dart version 2.12.0 (build 2.12.0-248.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/SDK
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/SDK
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.18.1

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-x64     • macOS 11.1 20C69 darwin-x64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 88.0.4324.96

• No issues found!

Thank you

@TahaTesser TahaTesser added found in release: 1.26 Found to occur in 1.26 has reproducible steps The issue has been confirmed reproducible and is ready to work on p: first party f: integration_test The flutter/packages/integration_test plugin and removed in triage Presently being triaged by the triage team labels Jan 22, 2021
@stuartmorgan-g stuartmorgan-g added a: tests "flutter test", flutter_test, or one of our tests and removed plugin labels Mar 15, 2021
@ipowell
Copy link

ipowell commented Oct 14, 2021 •
edited
Loading

I've also been looking at this, I would like to add a way to call responseDataCallback even when tests fail. The easiest option would be to add a boolean parameter to integrationDriver. I think a better option would be to add a flag to flutter drive, but I'm not sure yet how to propagate that value yet. I'd be happy to give it a go if either of these solutions are acceptable.

@jiteshmupl
Copy link

This issue still persists, any solution for it?

@vrnvorona
Copy link
Author

This issue still persists, any solution for it?

Ugly workaround is to meddle with local version of flutter or fork it to add it.

@darshankawar darshankawar added the framework flutter/packages/flutter repository. See also f: labels. label Apr 26, 2022
@Piinks Piinks added the P2 Important issues not at the top of the work list label Jun 9, 2022
@jgmGit
Copy link

jgmGit commented Jul 29, 2022

Any update on this ? The issue still remains with Flutter version 3.0+. Any fix planned on such a basic feature ? Or is there some other way to get the tests reports from the CI/CD server ?

@vrnvorona
Copy link
Author

Any update on this ? The issue still remains with Flutter version 3.0+. Any fix planned on such a basic feature ? Or is there some other way to get the tests reports from the CI/CD server ?

Try my approach with this. Instead of using default integration driver, make custom one with slight tweak to it. Works for me for now.

Future<void> fixedIntegrationDriver({
  Duration timeout = const Duration(minutes: 20),
  integration_test_driver.ResponseDataCallback? responseDataCallback =
      integration_test_driver.writeResponseData,
}) async {
  final FlutterDriver driver = await FlutterDriver.connect();
  final String jsonResult = await driver.requestData(null, timeout: timeout);
  final Response response = Response.fromJson(jsonResult);

  await driver.close();
  // moved callback outside of allTestsPassed check
  if (responseDataCallback != null) {
    await responseDataCallback(response.data);
  }
  if (response.allTestsPassed) {
    //ignore: avoid_print
    print('All tests passed.');
    exit(0);
  } else {
    //ignore: avoid_print
    print('Failure Details:\n${response.formattedFailureDetails}');
    exit(1);
  }
}

@jgmGit
Copy link

jgmGit commented Jul 31, 2022

Any update on this ? The issue still remains with Flutter version 3.0+. Any fix planned on such a basic feature ? Or is there some other way to get the tests reports from the CI/CD server ?

Try my approach with this. Instead of using default integration driver, make custom one with slight tweak to it. Works for me for now.

Future<void> fixedIntegrationDriver({
  Duration timeout = const Duration(minutes: 20),
  integration_test_driver.ResponseDataCallback? responseDataCallback =
      integration_test_driver.writeResponseData,
}) async {
  final FlutterDriver driver = await FlutterDriver.connect();
  final String jsonResult = await driver.requestData(null, timeout: timeout);
  final Response response = Response.fromJson(jsonResult);

  await driver.close();
  // moved callback outside of allTestsPassed check
  if (responseDataCallback != null) {
    await responseDataCallback(response.data);
  }
  if (response.allTestsPassed) {
    //ignore: avoid_print
    print('All tests passed.');
    exit(0);
  } else {
    //ignore: avoid_print
    print('Failure Details:\n${response.formattedFailureDetails}');
    exit(1);
  }
}

@vrnvorona This seems to be bit better, but still the json response doesn't seem to contain the total test count and other information. Could you also fix that part ?

@vrnvorona
Copy link
Author

@jgmGit to that I have no information so I can't help.

@flutter-triage-bot flutter-triage-bot bot added team-framework Owned by Framework team triaged-framework Triaged by Framework team labels Jul 8, 2023
@huycozy
Copy link
Member

huycozy commented Apr 16, 2024

Reproduced the issue on the latest Flutter stable and master channels.

flutter doctor -v (stable and master) 
[✓] Flutter (Channel stable, 3.19.5, on macOS 14.1 23B74 darwin-x64, locale en-VN)
    • Flutter version 3.19.5 on channel stable at /Users/huynq/Documents/GitHub/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 300451adae (8 days ago), 2024-03-27 21:54:07 -0500
    • Engine revision e76c956498
    • Dart version 3.3.3
    • DevTools version 2.31.1

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.9+0-17.0.9b1087.7-11185874)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15C500b
    • CocoaPods version 1.15.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • android-studio-dir = /Applications/Android Studio.app/
    • Java version OpenJDK Runtime Environment (build 17.0.9+0-17.0.9b1087.7-11185874)

[✓] VS Code (version 1.87.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.84.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-x64     • macOS 14.1 23B74 darwin-x64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 123.0.6312.106

[✓] Network resources
    • All expected network resources are available.

• No issues found!
[!] Flutter (Channel master, 3.22.0-11.0.pre.14, on macOS 14.1 23B74 darwin-x64, locale en-VN)
    • Flutter version 3.22.0-11.0.pre.14 on channel master at /Users/huynq/Documents/GitHub/flutter_master
    ! Warning: `flutter` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    ! Warning: `dart` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/dart, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 817630ccf1 (51 minutes ago), 2024-04-15 22:56:25 -0400
    • Engine revision 76fbe31a96
    • Dart version 3.5.0 (build 3.5.0-61.0.dev)
    • DevTools version 2.35.0-dev.8
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.9+0-17.0.9b1087.7-11185874)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.3)
    • Xcode at /Applications/Xcode15.3.app/Contents/Developer
    • Build 15E204a
    • CocoaPods version 1.15.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • android-studio-dir = /Applications/Android Studio.app/
    • Java version OpenJDK Runtime Environment (build 17.0.9+0-17.0.9b1087.7-11185874)

[✓] VS Code (version 1.88.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.86.0

[✓] Connected device (4 available)
    • RMX2001 (mobile) • EUYTFEUSQSRGDA6D                         • android-arm64  • Android 11 (API 30)
    • iPhone (mobile)  • d9a94afe2b649fef56ba0bfeb052f0f2a7dae95e • ios            • iOS 15.8 19H370
    • macOS (desktop)  • macos                                    • darwin-x64     • macOS 14.1 23B74 darwin-x64
    • Chrome (web)     • chrome                                   • web-javascript • Google Chrome 123.0.6312.124

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

@huycozy huycozy added found in release: 3.19 Found to occur in 3.19 and removed found in release: 1.26 Found to occur in 1.26 labels Apr 16, 2024
@huycozy huycozy added the found in release: 3.22 Found to occur in 3.22 label Apr 16, 2024
@ricardodalarme
Copy link
Contributor

It looks like this was already fixed:

You just need to set writeResponseOnFailure to true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
a: tests "flutter test", flutter_test, or one of our tests f: integration_test The flutter/packages/integration_test plugin found in release: 3.19 Found to occur in 3.19 found in release: 3.22 Found to occur in 3.22 framework flutter/packages/flutter repository. See also f: labels. has reproducible steps The issue has been confirmed reproducible and is ready to work on P2 Important issues not at the top of the work list team-framework Owned by Framework team triaged-framework Triaged by Framework team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@stuartmorgan-g @ipowell @vrnvorona @jgmGit @Piinks @ricardodalarme @TahaTesser @darshankawar @jiteshmupl @huycozy
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy