Content-Length: 368052 | pFad | https://github.com/fastplotlib/fastplotlib/pull/447

EA simplify screenshot testing by kushalkolar · Pull Request #447 · fastplotlib/fastplotlib · GitHub
Skip to content

simplify screenshot testing #447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Mar 31, 2024
Merged

simplify screenshot testing #447

merged 15 commits into from
Mar 31, 2024

Conversation

kushalkolar
Copy link
Member

@kushalkolar kushalkolar commented Mar 23, 2024 •
edited
Loading

use ideas from pygfx/pygfx#684

closes #446
A threshold of 0.2 seemed like it was too lenient, I was able to change the color of one of the circles in the line collection examples and the RMSE was 0.09

@kushalkolar kushalkolar marked this pull request as ready for review March 23, 2024 04:24
@kushalkolar kushalkolar requested a review from clewis7 as a code owner March 23, 2024 04:24
@kushalkolar
Copy link
Member Author

kushalkolar commented Mar 23, 2024 •
edited
Loading

This seems to work quite well. The ground truths were generated on my RX 570 at home, the largest diff was with the scatter_size example, due to the text. The diff on github actions was slightly over 0.02. I also tested on the GPUs on the Flatiron cluster, the diff for that example is 0.020826.

There seems to be no difference between the diffs on Nvidia Tesla V100 and Nvidia A100 on the cluster, at least Nvidia is consistent about one think I guess 😂

EDIT: Just tested on the Nvidia H100 and I get a max diff of 0.02077 on the scatter_size example. The diffs are a bit difference than on the V100 and A100 so I guess Nvidia actually isn't consistent.

diffs for all screenshots on a Tesla V100 on the Flatiron cluster for reference:
This is the diff between the Rx 570 and the overpowered Nvidia GPUs

examples/tests/test_examples.py::test_that_we_are_on_lavapipe DiscreteGPU Vulkan
examples/tests/test_examples.py::test_example_screenshots[gridplot] 0.0025418061938043534
examples/tests/test_examples.py::test_example_screenshots[gridplot_non_square] 0.0017253794384335269
examples/tests/test_examples.py::test_example_screenshots[heatmap] 0.0026005766798322047
examples/tests/test_examples.py::test_example_screenshots[heatmap_cmap] 0.002662877302239744
examples/tests/test_examples.py::test_example_screenshots[heatmap_data] 0.0020797810741957933
examples/tests/test_examples.py::test_example_screenshots[heatmap_vmin_vmax] 0.002914066889570422
examples/tests/test_examples.py::test_example_screenshots[image_cmap] 0.0019563093643064503
examples/tests/test_examples.py::test_example_screenshots[image_rgb] 0.0014489276911573798
examples/tests/test_examples.py::test_example_screenshots[image_rgbvminvmax] 0.0005612692156754262
examples/tests/test_examples.py::test_example_screenshots[image_simple] 0.0016309417498936698
examples/tests/test_examples.py::test_example_screenshots[image_vminvmax] 0.0005612692156754262
examples/tests/test_examples.py::test_example_screenshots[line] 0.0029302534153462386
examples/tests/test_examples.py::test_example_screenshots[line_cmap] 0.003085553070868819
examples/tests/test_examples.py::test_example_screenshots[line_colorslice] 0.0031247686572509034
examples/tests/test_examples.py::test_example_screenshots[line_dataslice] 0.0023580007672577286
examples/tests/test_examples.py::test_example_screenshots[line_present_scaling] 0.0027367853208208497
examples/tests/test_examples.py::test_example_screenshots[line_collection] 0.003779845416274792
examples/tests/test_examples.py::test_example_screenshots[line_collection_cmap_values] 0.0023366316256708897
examples/tests/test_examples.py::test_example_screenshots[line_collection_cmap_values_qualitative] 0.002891149865417338
examples/tests/test_examples.py::test_example_screenshots[line_collection_colors] 0.0`022398405285301453
examples/tests/test_examples.py::test_example_screenshots[line_stack] 0.005911900712291164
examples/tests/test_examples.py::test_example_screenshots[scatter] 0.012353324462660456
examples/tests/test_examples.py::test_example_screenshots[scatter_cmap] 0.0030825225024386587
examples/tests/test_examples.py::test_example_screenshots[scatter_colorslice] 0.004020251842124368
examples/tests/test_examples.py::test_example_screenshots[scatter_dataslice] 0.013067512988075274
examples/tests/test_examples.py::test_example_screenshots[scatter_present] 0.013810446602977897
examples/tests/test_examples.py::test_example_screenshots[scatter_size] 0.020826465412837777

kushalkolar
kushalkolar commented Mar 23, 2024
View reviewed changes
@kushalkolar
Copy link
Member Author

So on Mac there seems to be large diffs with the heatmaps, no sure why, will figure out later.

diff-heatmap-rgb
diff-heatmap_cmap-rgb
diff-heatmap_data-rgb
diff-heatmap_vmin_vmax-rgb

@kushalkolar
Copy link
Member Author

kushalkolar commented Mar 23, 2024 •
edited
Loading

I think that we should figure out how to make the heatmap test screenshot have more of the actual heatmap, not sure why it's small in the top left corner.

Or maybe think about getting rid of heatmap since the Texture dimension limit seems to no longer apply with WGPU for the past while, need to investigate this 🤔

@kushalkolar kushalkolar requested a review from almarklein March 23, 2024 05:17
@kushalkolar kushalkolar merged commit 9a1bfbe into main Mar 31, 2024
@kushalkolar kushalkolar deleted the simplify-screenshot-testing branch March 31, 2024 09:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@clewis7 clewis7 Awaiting requested review from clewis7 clewis7 is a code owner

@almarklein almarklein Awaiting requested review from almarklein

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

better image comparison for tests
2 participants
@kushalkolar @almarklein








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: https://github.com/fastplotlib/fastplotlib/pull/447

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy