Skip to content

add ImageVolumeGraphic #791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from
Draft

add ImageVolumeGraphic #791

wants to merge 14 commits into from

Conversation

kushalkolar
Copy link
Member

@kushalkolar kushalkolar commented Apr 12, 2025

actually going to merge it this time

  • TextureArrays supports 3D textures
  • implement simple ImageVolumeGraphic
  • implement graphic features for all the different volume materials
  • check and document sharing the data buffer between an object that uses mip/ray material and another object that uses slice material.
  • examples
    • basic for all material types
    • 3D video, maybe generated calcium toy data?
    • 2D video over time, maybe from the calcium example I added to imageio
    • other toy data, does sklearn have any relevant ones?
  • make hlut tool work with this, account for the dims so that the cmap bar shows up
  • make this work with ImageWidget, image widget is a mess, need to refactor iw before adding volume support
  • tests: screenshots, API, new graphic, TextureArray for 2d and 3d, and any new graphic features.
  • test RGB(A)
  • fix some typos in graphics.image
  • scripts/generate_add_graphic_methods.py is better, removed Graphic.type attribute which was only used in the generate mixin script and use regex instead to create the add_<graphic> method names.
  • improve docstring for quick_min_max()

@almarklein are RGB volumes supported? Is it even a thing?

selector tools for slicing, cube slicing, etc., non-orthogonal slicing to come later

note: implementing ImageVolumeGraphic as a subclass of ImageGraphic wasn't a good idea because the __init__ gets messy. Might make a BaseImageGraphic class which just has properties which are almost identical between ImageGraphic and ImageVolumeGraphic, but not a big deal right now.

@kushalkolar kushalkolar requested a review from clewis7 as a code owner April 12, 2025 06:20
@kushalkolar kushalkolar mentioned this pull request Apr 12, 2025
@kushalkolar
Copy link
Member Author

took some 3d movie gen code from caiman to generate a 3d movie:

@FlynnOConnell

3d-2025-04-12_03.09.08.mp4

@FlynnOConnell
Copy link
Collaborator

File [~\repos\fastplotlib\fastplotlib\graphics\features\_image.py:147](http://localhost:8888/lab/tree/demos/notebooks/~/repos/fastplotlib/fastplotlib/graphics/features/_image.py#line=146), in TextureArray._fix_data(self, data)
    145 def _fix_data(self, data):
    146     if data.ndim not in (2, 3):
--> 147         raise ValueError(
    148             "image data must be 2D with or without an RGB(A) dimension, i.e. "
    149             "it must be of shape [rows, cols], [rows, cols, 3] or [rows, cols, 4]"
    150         )
    152     # let's just cast to float32 always
    153     return data.astype(np.float32)

ValueError: image data must be 2D with or without an RGB(A) dimension, i.e. it must be of shape [rows, cols], [rows, cols, 3] or [rows, cols, 4]
data = np.random.randn(50, 2, 448, 448)
data.shape
>> (50, 2, 448, 448)

The ability to view a time-series with a 4th dimension would be nice here

@FlynnOConnell
Copy link
Collaborator

Also how difficult is something like scale=[1, 4, 1] to account for anisotropy in XY and Z? Here I have a volume of [14, 448, 448] ZXY, but from the side its a flat plane. This dataset is [2um x 2um x 17um]

image

@FlynnOConnell
Copy link
Collaborator

Also how difficult is something like scale=[1, 4, 1] to account for anisotropy in XY and Z? Here I have a volume of [14, 448, 448] ZXY, but from the side its a flat plane. This dataset is [2um x 2um x 17um]

image

nevermind :)

volume.world_object.world.scale_z = 15.0

Some other helpful tidbits:

hlut = fpl.HistogramLUTTool(volume.data.value, volume)
volume.interpolation = "linear"
volume.cmap = "gnuplot2"

fig[0, 0].docks["right"].add_graphic(hlut)
fig[0, 0].docks["right"].size = 80
fig[0, 0].docks["right"].controller.enabled = False
fig[0, 0].docks["right"].camera.maintain_aspect = False
fig[0, 0].docks["right"].auto_scale(maintain_aspect=False)

@almarklein
Copy link
Collaborator

@almarklein are RGB volumes supported? Is it even a thing?

Yes, looks like it is; it uses the same logic (in the shader) as the image. I have never encountered it in the wild though.

@kushalkolar kushalkolar mentioned this pull request Apr 17, 2025
67 tasks
@kushalkolar kushalkolar marked this pull request as draft June 4, 2025 04:58
@kushalkolar
Copy link
Member Author

ok all the basics are down, switchable materials, can change material props:

vol-2025-06-04_03.34.28.mp4

Need to figure out why regex is turning VolumeSliceMaterial into slic instead of slice

@kushalkolar
Copy link
Member Author

In other news, the latest pycharm is full of bugs, wtf is this. Also it wanted to auto insert the old rst structured docstring template even though I configured it to do numpydoc format 🤮

image

@kushalkolar
Copy link
Member Author

kushalkolar commented Jun 5, 2025

All that's left is tests, 2D over time example, and RGB(A) examples. I also think I want to merge #849 and then rebase this so the examples will be much simpler.

Also pygfx has non-orthogonal slicing:

vol_slicing-2025-06-05_00.53.40.mp4

@kushalkolar
Copy link
Member Author

ok there's a bug in the TextureArray or tiling somewhere, because when a dim is greater than the limit it is tiling along the wrong dim 😆

image

@kushalkolar
Copy link
Member Author

The rendering at the edges is weird if one of the dims is larger than the 3d texture size limit so let's just not support this, I think the uses are very much rare edge cases. Also if the 3d texture size limit is 16,383, then 16384x16384x16384 with float32 is 17 TB which is more than any GPU that I know of can handle 😂 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy