Skip to content

align/concat/combine_nested implementation #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 27 commits into from
Jun 16, 2025
Merged

align/concat/combine_nested implementation #18

merged 27 commits into from
Jun 16, 2025

Conversation

dcherian
Copy link
Collaborator

@dcherian dcherian commented Apr 17, 2025 •
edited
Loading

Closes #20

TODO:

  • add 2D combine_nested test
  • Add better error messages for the new tests

@benbovy feel free to take over if you have time.

This is ugly BUT for the example in https://github.com/dcherian/rasterix/issues/16#issuecomment-2811585219

xr.concat([ds40, ds50], dim=("x", "y"))

now gets as far as

ile [~/miniforge3/envs/rasterix/lib/python3.12/site-packages/xarray/structure/alignment.py:511](http://localhost:8888/lab/tree/repos/workshop-cng-2025-zarr/~/miniforge3/envs/rasterix/lib/python3.12/site-packages/xarray/structure/alignment.py#line=510), in Aligner._get_dim_pos_indexers(self, matching_indexes)
    509     if obj_idx is not None:
    510         if self.reindex[key]:
--> 511             indexers = obj_idx.reindex_like(aligned_idx, **self.reindex_kwargs)
    512             dim_pos_indexers.update(indexers)
    514 return dim_pos_indexers

File [~/miniforge3/envs/rasterix/lib/python3.12/site-packages/xarray/core/indexes.py:305](http://localhost:8888/lab/tree/repos/workshop-cng-2025-zarr/~/miniforge3/envs/rasterix/lib/python3.12/site-packages/xarray/core/indexes.py#line=304), in Index.reindex_like(self, other)
    289 def reindex_like(self, other: Self) -> dict[Hashable, Any]:
    290     """Query the index with another index of the same type.
    291 
    292     Implementation is optional but required in order to support alignment.
   (...)    303         indexers.
    304     """
--> 305     raise NotImplementedError(f"{self!r} doesn't support re-indexing labels")

NotImplementedError: RasterIndex
'x':
    <rasterix.raster_index.AxisAffineTransformIndex object at 0x10e146510>
'y':
    <rasterix.raster_index.AxisAffineTransformIndex object at 0x10e146a50> doesn't support re-indexing labels

dcherian
dcherian commented Apr 17, 2025
View reviewed changes
@dcherian
Copy link
Collaborator Author

dcherian commented Apr 17, 2025 •
edited
Loading

Aha, here's a concrete feature request

This index is shared between dims x and y.

So I cannot do

xr.concat([ds50, ds40], dim="x")

because I get

ValueError: cannot exclude dimension(s) x from alignment because these are used by an index together with non-excluded dimensions y

So I cannot do

xr.concat([ds50, ds40], dim=("x", "y"))

Then the alignment does not exclude "x" and "y" individually, it excludes the tuple ("x", "y") so we end up in a bad situation with terrible reindexing.

Q: If asked to concat along a dim with a multi-dim index, then should we exclude all of those dims?

Using combine_nested seems promising, perhaps that is the solution

xr.combine_nested([[ds50, ds40]], concat_dim=("x", "y"))

@dcherian
Copy link
Collaborator Author

dcherian commented Apr 17, 2025 •
edited
Loading

Nope, combine_nested is just concat in a loop so we are still blocked.

Q: If asked to concat along a dim with a multi-dim index, then should we exclude all of those dims?

I don't see any other option at the moment. especially if we want concat_nested to magically work.

@benbovy
Copy link
Member

benbovy commented Apr 18, 2025

If asked to concat along a dim with a multi-dim index, then should we exclude all of those dims?

It might be worth trying, although I don't know much what implications this rule would have in the general case.

Or maybe trying to fit this case into concat or combine_nested is not really worth the extra complexity and it would be fine redirecting users to a more specific method similar to Rioxarray's merge_arrays and merge_datasets?

From a UX perspective, however, the alignment error above while technically correct does not give a chance of providing more context. We should probably give this chance, e.g., via some dedicated Xarray Index API (the alignment error is raised from a loop iterating over index objects).

@benbovy benbovy mentioned this pull request Apr 18, 2025
Open
@dcherian dcherian mentioned this pull request Apr 18, 2025
Closed
dcherian
dcherian commented May 16, 2025
View reviewed changes
@dcherian dcherian changed the title WIP align implementation WIP align/combine implementation May 16, 2025
@dcherian dcherian mentioned this pull request May 29, 2025
Merged
@benbovy benbovy mentioned this pull request Jun 5, 2025
Merged
@dcherian dcherian changed the title WIP align/combine implementation align/concat/combine implementation Jun 12, 2025
@dcherian dcherian requested a review from benbovy June 12, 2025 03:56
@dcherian dcherian marked this pull request as ready for review June 12, 2025 03:56
dcherian
dcherian commented Jun 12, 2025
View reviewed changes
benbovy
benbovy reviewed Jun 12, 2025
View reviewed changes
dcherian
dcherian commented Jun 13, 2025
View reviewed changes
@dcherian dcherian changed the title align/concat/combine implementation align/concat/combine_nested implementation Jun 13, 2025
@dcherian
Copy link
Collaborator Author

OK this should be good to go!

@dcherian dcherian merged commit d6242e1 into main Jun 16, 2025
9 checks passed
@dcherian dcherian deleted the wip-align branch June 16, 2025 15:42
benbovy
benbovy reviewed Jun 26, 2025
View reviewed changes
src/rasterix/raster_index.py
Comment on lines +341 to +344
self._shape = {
"x": self._wrapped_indexes["x"].axis_transform.size,
"y": self._wrapped_indexes["y"].axis_transform.size,
}
Copy link
Member

@benbovy benbovy Jun 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should avoid hard-coding "x" and "y" coordinate names.

We should also fix self._shape for the following cases:

  • There's only one wrapped index, either AxisAffineTransformIndex or PandasIndex (e.g., result from indexing with a scalar on one of the raster dimensions)
  • There's two wrapped PandasIndex instances (e.g., result from indexing with arbitrary array values)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I got this on the PR branch with docs 😓

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm refactoring RasterIndex in #43. I think it will make our lives much easier... Supporting the cases above in RasterIndex was a (my) bad idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@benbovy benbovy benbovy left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

concat and combine
2 participants
@dcherian @benbovy
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy