Skip to content

Potential problem with a wsc method during multiprocessing #18147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Richardjmorton opened this issue May 14, 2025 · 1 comment
Open

Potential problem with a wsc method during multiprocessing #18147

Richardjmorton opened this issue May 14, 2025 · 1 comment

Comments

@Richardjmorton
Copy link

Richardjmorton commented May 14, 2025

Description

I have been trying to write a multiprocessing function that works with headers and WCS, however I am running into an issue. I think the issue is with the method wcs.array_index_to_world, which returns something different to what I expect.

Expected behavior

No response

How to Reproduce

Here is a reduced version of the code I am running:

def example_func(header_str, dim_len):
      header = Header.fromstring(header_str,sep="\n")
      wcs = WCS(header)
      xy = wcs.array_index_to_world(0,np.arange(dim_len),0)


# multiprocessing part
dim_len = value
tasks = [(hdr.tostring(sep="\n",padding=False, endcard=True), dim_len) for hdr in hdrs]

with ProcessPoolExecutor() as executor:
        results = list(texecutor.map(example_func, *zip(*tasks)))

To use concurrent.ProcessPoolExecutor I am having to convert headers to strings (so they can be pickled) and then try rebuildling the header in the functions. The header is passed to WCS as I want to get spatial coordinates of the data.

The wcs object appears to be as it should (at least from a naiive print(wcs) inside the example function). However, after calling the method array_index_to_world I get a different result than when running the code without multiprocessing.

In single processing, xy has length 2 and xy[1] is a SkyCoord. The xy[0] is a SpectralCoord.

During multiprocessing, the variable xy is of length 3 and and the elements of the list are now Quantity.

The issue goes away if I swap concurrent.ProcessPoolExecutor for concurrent.ThreadPoolExecutor.

I don't really have a minimum working example of the issue, but happy to provide a fits file that this occurs on.

Versions

The version of astropy I am using is 6.17

@Richardjmorton Richardjmorton changed the title Problem with creating fits headers during mutliprocessing Potential problem with a wsc method during mutliprocessing May 14, 2025
@pllim pllim added the io.fits label May 14, 2025
@pllim
Copy link
Member

pllim commented May 14, 2025

If you cannot provide minimally reproducible example, would still be nice to give us the input/output values for one problematic case you are seeing. Since the pseudocode provided is not really the whole picture, maybe the actual problem is not even presented here. My suspicion is that something you thought was pickled is not behaving as you thought it is, since it gives problem in process pool but not thread pool.

But before that, I recommend you first upgrade to use astropy v7.1rc1 (pip install astropy --pre -U) as we no longer support the v6.x series.

Thank you.

@pllim pllim changed the title Potential problem with a wsc method during mutliprocessing Potential problem with a wsc method during multiprocessing May 14, 2025
@pllim pllim added the wcs label May 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy