Skip to content

gh-136421: Load _datetime static types during interpreter initialization #136583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

ZeroIntensity
Copy link
Member

@ZeroIntensity ZeroIntensity commented Jul 12, 2025

_datetime is a special module, because it's the only non-builtin C extension that contains static types. As such, it would initialize static types in the module's execution function, which can run concurrently. Since static type initialization is not thread-safe, this caused crashes. This fixes it by moving the initialization of _datetime's static types to interpreter startup (where all other static types are initialized), which is already properly protected through other locks.

I have no idea if this will fix the Windows build, but let's hope.
@ZeroIntensity ZeroIntensity requested a review from a team as a code owner July 12, 2025 14:22
@neonene
Copy link
Contributor

neonene commented Jul 12, 2025

My concern is unnecessarily losing portability between the builtin module and the non-builtin one.

This PR avoids a crash not by locking but forcing the main interpreter initialize the _datetime static types, which I think is almost equivalent to the following change in _datetimemodule.c:

PyMODINIT_FUNC
PyInit__datetime(void)
{
+   if (PyDateTime_DateType.tp_subclasses == NULL) {
+       PyInterpreterState *interp = PyInterpreterState_Get();
+       // main interp currently checks a module before subinterp imports
+       assert(_Py_IsMainInterpreter(interp));
+       init_static_types(interp, 0);
+   }
    return PyModuleDef_Init(&datetimemodule);
}

@ZeroIntensity
Copy link
Member Author

Hmm, that feels really hacky.

I wouldn't call the movement of _datetime to static being totally unnecessary, because _datetime exposes a C API via a capsule. If _datetime isn't included with the build of Python, things using the capsule are very likely to crash, so it makes more sense to me to require it. Are there people actually disabling _datetime in their builds?

@neonene
Copy link
Contributor

neonene commented Jul 12, 2025

Hmm, that feels really hacky.

The crash can be avoided even on pure Python level, and managed static extsension types are already special only for _datetime. The C API is likely to be redesigned in the future, which will be a standard way to shared extensions including _datetime.

Are there people actually disabling _datetime in their builds?

Not sure about disabling. On Windows, _datetime is a built-in module that can be switched to a DLL. I know someone who want to use such a DLL as a plugin.

@ZeroIntensity
Copy link
Member Author

Right, there are several ways to fix this that work, but I'm trying to find the fix that is the most correct. Relying on implementation details like tp_subclasses is a little more maintenance if we ever change type initialization, and it also muddies our general rules about how static types should work. I see several options:

  1. Make _datetime static, as it is in this PR. This makes the most sense, but I guess it could compromise compatibility in some niche cases.
  2. Initialize static types in the PyInit_ function. This works, but will probably lead to some other issues because it mixes single and multi-phase initialization, which isn't supposed to be work.
  3. Import _datetime in _interpreters. I really don't like this fix because it won't fix subinterpreters created via the C API.
  4. Import _datetime in new_interpreter. This slows down initialization and is also a bit of a hack.
  5. Make _PyStaticType_InitForExtension thread-safe. I think this will be needlessly complex.
  6. Serialize calls to _PyStaticType_InitForExtension (gh-136421: Fix crash when _datetime is been initialized in multiple sub-interpreters #136422). This works but again hurts the story about static types, and is technically more of a bottleneck.

We have to find a balance between correctness and simplicity, and I'm not yet sure which that is.

@neonene
Copy link
Contributor

neonene commented Jul 12, 2025

  1. Make _datetime static, as it is in this PR. This makes the most sense, but I guess it could compromise compatibility in some niche cases.

This is at least a feature change more than a bug fix.

@ZeroIntensity
Copy link
Member Author

Yeah, you're right. I'd be fine with a bandaid fix for 3.14 if you want to make a PR. The PyInit_ approach probably works best, but I don't fully understand why we need the tp_subclasses check.

@neonene
Copy link
Contributor

neonene commented Jul 12, 2025

Still rough example, but checking tp_subclasses is an alternative to PyType_HasFeature(PyDateTime_DateType, Py_TPFLAGS_READY). I'm not sure which is better on free-threading. Both must be kept in the main interpreter in its life, which means interp_count is at least 1 in the runtime state and the subinterp shutdown cannot invoke fini_static_type() as final.

During the _datetime exec phase, the main interpreter probably needs to skip init_static_types() to not increase interp_count.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy