Skip to content

Allow setting (or skipping) new indexes in open_dataset #8051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Jul 8, 2025
Merged
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
c703ebc
add set_indexes parameter to open_dataset
benbovy Aug 7, 2023
6f54cd5
implement set_indexes in (zarr) backend store
benbovy Aug 7, 2023
f77aac7
Merge branch 'main' into backend-set-indexes
dcherian Nov 13, 2023
eae983b
Merge branch 'main' into backend-set-indexes
TomNicholas Feb 3, 2024
dfe6496
Merge branch 'main' into backend-set-indexes
keewis Jun 29, 2025
145ae1c
replace `set_indexes` with `create_default_indexes`
keewis Jun 30, 2025
192c367
make sure indexes set by the backend survive
keewis Jun 30, 2025
f5823a7
also add the parameter to `open_datatree`
keewis Jun 30, 2025
2ff8402
share the implementation of the default indexes creation
keewis Jun 30, 2025
82d629b
Merge branch 'main' into backend-set-indexes
keewis Jun 30, 2025
de5ce26
Merge branch 'main' into backend-set-indexes
keewis Jul 1, 2025
294b2f7
check that the store backend entrypoint does not create default indexes
keewis Jul 1, 2025
5c3a843
actually do not create default indexes in the backends
keewis Jul 1, 2025
08939de
rename the helper
keewis Jul 1, 2025
0f281b1
Merge branch 'main' into backend-set-indexes
dcherian Jul 2, 2025
4620490
Merge branch 'main' into backend-set-indexes
dcherian Jul 3, 2025
95dbf8e
move the handling of `create_default_indexes` up the call stack
keewis Jul 3, 2025
eb4f866
Merge branch 'main' into backend-set-indexes
keewis Jul 3, 2025
d7e6daa
what's new
keewis Jul 3, 2025
687f0c2
Merge branch 'main' into backend-set-indexes
benbovy Jul 8, 2025
3d483d3
Fix
dcherian Jul 8, 2025
741564e
fix again
dcherian Jul 8, 2025
8889eda
also create default indexes without chunks
keewis Jul 8, 2025
804db4c
also copy `_close`
keewis Jul 8, 2025
75c1dd6
reuse the code for copying `_close`
keewis Jul 8, 2025
a0d94fb
refactor
dcherian Jul 8, 2025
bbc263b
Merge branch 'main' into backend-set-indexes
dcherian Jul 8, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
share the implementation of the default indexes creation
  • Loading branch information
keewis committed Jun 30, 2025
commit 2ff8402192a3afd443e155e5a3c10d592475dce7
34 changes: 24 additions & 10 deletions xarray/backends/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,18 @@ def _chunk_ds(
return backend_ds._replace(variables)


def _create_default_indexes(ds, create_default_indexes):
if not create_default_indexes:
return ds

to_index = {
name: coord.variable
for name, coord in ds.coords.items()
if coord.dims == (name,) and name not in ds.xindexes
}
return ds.assign_coords(Coordinates(to_index))


def _dataset_from_backend_dataset(
backend_ds,
filename_or_obj,
Expand All @@ -400,15 +412,7 @@ def _dataset_from_backend_dataset(

_protect_dataset_variables_inplace(backend_ds, cache)

if create_default_indexes:
to_index = {
name: coord.variable
for name, coord in backend_ds.coords.items()
if coord.dims == (name,) and name not in backend_ds.xindexes
}
indexed = backend_ds.assign_coords(Coordinates(to_index))
else:
indexed = backend_ds
indexed = _create_default_indexes(backend_ds, create_default_indexes)

if chunks is None:
ds = indexed
Expand Down Expand Up @@ -447,6 +451,7 @@ def _datatree_from_backend_datatree(
inline_array,
chunked_array_type,
from_array_kwargs,
create_default_indexes,
**extra_tokens,
):
if not isinstance(chunks, int | dict) and chunks not in {None, "auto"}:
Expand All @@ -461,7 +466,7 @@ def _datatree_from_backend_datatree(
tree = DataTree.from_dict(
{
path: _chunk_ds(
node.dataset,
node.dataset.pipe(_create_default_indexes, create_default_indexes),
filename_or_obj,
engine,
chunks,
Expand Down Expand Up @@ -977,6 +982,7 @@ def open_datatree(
concat_characters: bool | Mapping[str, bool] | None = None,
decode_coords: Literal["coordinates", "all"] | bool | None = None,
drop_variables: str | Iterable[str] | None = None,
create_default_indexes: bool = True,
inline_array: bool = False,
chunked_array_type: str | None = None,
from_array_kwargs: dict[str, Any] | None = None,
Expand Down Expand Up @@ -1086,6 +1092,13 @@ def open_datatree(
A variable or list of variables to exclude from being parsed from the
dataset. This may be useful to drop variables with problems or
inconsistent values.
create_default_indexes : bool, default: True
If True, create pandas indexes for :term:`dimension coordinates <dimension coordinate>`,
which loads the coordinate data into memory. Set it to False if you want to avoid loading
data into memory.

Note that backends can still choose to create other indexes. If you want to control that,
please refer to the backend's documentation.
inline_array: bool, default: False
How to include the array in the dask task graph.
By default(``inline_array=False``) the array is included in a task by
Expand Down Expand Up @@ -1179,6 +1192,7 @@ def open_datatree(
chunked_array_type,
from_array_kwargs,
drop_variables=drop_variables,
create_default_indexes=create_default_indexes,
**decoders,
**kwargs,
)
Expand Down
Loading
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy