25x nonparallelized speedup by avoiding MSBuildWorkspace in example tester #1338
Conversation
…ulti-project examples
|
I'd like to add the ExampleTester and ExampleExtractor projects to the filter for the GitHub Actions .yaml that runs ExampleExtractor and ExampleTester, so that we can see how the run is affected by code changes and not just by changes under the (Update: this is handled in #1336) |
|
Also, I need to investigate a failing example. The reason I didn't investigate this example before pushing is that this example also fails for me on my local machine with the latest commit on
(Update: this is handled in #1336) |
CS1661 is listed twice in the Expected list, otherwise the lists are the same – cause of problem? |
|
@Nigel-Ecma Interesting, thank you. Sounds likely to be a change in compiler behavior. Investigating at #1336 (comment), since I realized that is the PR that will first bring on the change. (Update: this is handled in #1336) |
|
Can I check, is the intention that this replaces #1337? I'd personally rather avoid parallelization unless we actually need it. |
|
I would want to keep #1337 since it allows working with each example as a unit test in addition to running via the CLI. Then, parallelization across the unit tests could be disabled if we didn't want it to be possible. What's the nature of your concern about parallelizing? |
Only that my experience is it can make life more complicated (and harder to diagnose) for very little benefit. I'll try to review both PRs in the next couple of days. |
|
I'm happy to make life easier by removing anything worrisome. 👍 |
Running ExampleTester on my machine takes 8 minutes. With the changes in this PR, it takes 19 seconds. That's a 25x speedup!
That's not even parallelizing- parallelized together with #1337, the entire suite of example tests runs in 8 seconds.
The goal is twofold:
This PR speeds up all tests except the six or so multi-csproj examples which include project references. The way it speeds up the tests is by reading and interpreting the csproj XML directly. Hopefully, the new code is straightforward enough to extend as needed.
It's important to preserve the fidelity of the testing. Because of this, the fast path will bail if it sees any csproj element that it does not fully understand, falling back to the existing MSBuildWorkspace code which invokes full MSBuild out-of-process. When an example starts using an unrecognized csproj property or item, the tester will quietly fall back to the slow path without logging any notices about this. If you're running the tester with a debugger attached which has opted in to stopping on handled exceptions, you'll hit a handled NotImplementedException that will be the only clue that you're falling back to the slow path.