Note: fixing this will enable us to fix the web HHH bot from failing every time the kernel version changes.

Examples: https://github.com/flutter/flutter/blob/master/packages/flutter_tools/test/integration.shard/command_output_test.dart#L21
https://github.com/flutter/flutter/blob/master/packages/flutter_tools/test/integration.shard/test_driver.dart#L477

\cc @jmagman any pointers that you can give us here?

@yjbanov was investigating something similar in #78072.

Never mind, this is requesting the possibility to --local-engine=android_debug instead of host_debug, not that the local engine isn't being used at all in some tests.

The original impetus for this seems to be:

We are seeing issues on HHH configuration of dart-lang/sdk,
flutter/engine and flutter/flutter where the web tests are using new
frontend_server tooling (which expects current kernel version) but is
given kernel file from an older version.

That may be fixed by #78496.

However if we still need this, the integration tests could have a getLocalEngineArguments() for each desired variant like getLocalEngineAndroidArguments() that replaces the injected flag values. Something like:

List<String> getLocalEngineAndroidArguments() => 
  getLocalEngineArguments().map((String argument) => argument.replaceAll('host_', 'android_')).toList();

but something more robust.

@jmagman I was under impression that we needed to set local engine environment variables to make sure the bots run tests with the local engine, as was attempted in #71721? I think that PR was reverted. Is it no longer needed?

@annagrin You're right, it's not related to #78496. web_tool_tests is calling pub without the local engine flags, not flutter test.
https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8852389365860202592/+/steps/flutter_test_web_tool_tests/0/stdout

@jmagman @godofredoc I would appreciate if someone could take this over from me, as I am not familiar with the flutter test infrastructure enough to make the right call.

The problem I think being described is that the tool assumes that --local-engine points to a platform/build mode variant. If you were locally developing for iOS, say, it's expecting:

--local-engine=ios_debug --local-engine-src-path=/path/to/engine/src

And if it needs to grab something from host_debug the artifacts know how to do that mapping, it can map ios_debug → host_debug

But it doesn't bother with host_debug → ios_debug, because it assumes if you wanted iOS artifacts you would have passed in ios_debug. _getEngineArtifactsPath isn't trying to figure out /path/to/engine/src/ios_debug here, it's just trusting --local-engine is pointing to the appropriate variant:

However, the testing infrastructure always passes in --local-engine host_*.

--local-engine=host_debug --local-engine-src-path=/b/s/w/ir/cache/builder/src

So when --local-engine --local-engine-src-path was passed through to pub test in #71721, and a tool test needed an Android artifact, it failed:

The file /b/s/w/ir/cache/builder/src/out/host_debug/flutter_embedding_release.pom wasn't found in the local engine out directory.

#71721 (comment)
https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8861844867680645408/+/steps/flutter_test_web_tool_tests/0/stdout

This was easy to reproduce locally:

$ cd examples/hello_world
$ flutter --local-engine=host_debug_unopt --local-engine-src-path=/path/to/engine/src build apk --analyze-size --target-platform=android-arm64

On top of all that, the pom/jar gradle artifact symlinking bypasses the above artifacts logic and does the path construction itself:

The most robust thing to do would be to handle --local-engine=host_ and do the mapping to target platforms engine out directories like ios_debug or whatever. The artifacts location code is already pretty hairy though, and our CI/tool subshard limitations of requiring more than one target platforms engine may not be worth potential regressions.

The easier thing to do would be to do something cheesy like getLocalEngineAndroidArguments as described in #72368 (comment)

Note: fixing this will enable us to fix the web HHH bot from failing every time the kernel version changes.

We had another kernel version change 4 days ago: https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket.appspot.com/8851986851676848432/+/steps/flutter_test_tool_tests/0/stdout

cc @johnniwinther

@athomas would it be possible to build other flavors of engine in dart HHH bot to make sure all tests run correctly if/when pub runs with local engine flag (will need to reland #71721 for it, and add more flags probably)? See @jmagman investigation above.

cc @sigmundch

Some observations after looking at this more closely and discussing it with @jonahwilliams:

1. Tests that use getLocalEngineArguments are in the integration.shard. We'll attempt to set the local-engine environment flags only for the web.shard to unblock the 3H-web bot on the Dart side.
2. We may not need a big general change to handle the uses of getLocalEngineArguments, but a one by one manual review instead. Currently only 12 files (total of 47 unit tests) use it. Many don't need a local engine, for example, so it may be OK to remove the use of the flag in those cases (see [flutter_tools] remove getLocalEngineArtifacts from integration tests that cant use it #83282). Individual tests could be changed to ignore the local engine flag or skipped the test if the specified engine is not compatible with the test.

@sigmundch Can you comment if this is something that needs to be done or otherwise close this issue? Thanks!

It's been a while, so trying to refresh my memory here. Since we landed #83308, we are no longer affected in the web HHH boots (item 1 in my previous comment), so that part is complete.

I am not aware that any work was done to address item 2. However, I'm not aware of anything being blocked by it either. As such, maybe the remaining work here could be categorized as tech debt/test hygiene and prioritized accordingly?

Got it, thanks!