Skip to content

Support for building python-tcod with Python 3.13 --disable-gil (no-GIL) build #161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
coldwarrl opened this issue Apr 19, 2025 · 9 comments

Comments

@coldwarrl
Copy link

coldwarrl commented Apr 19, 2025

Hi there 👋

First off, thank you for maintaining python-tcod — it's a really valuable library for roguelike development!

I've been experimenting with Python 3.13's new --disable-gil feature (based on the free-threaded no-GIL branch), and I wanted to see how well python-tcod builds and works in that environment.

🛠 What I tried:

  • Cloned the repository and initialized submodules.

  • Patched build_libtcod.py to conditionally skip Py_LIMITED_API when Py_GIL_DISABLED is detected via sysconfig.get_config_var("Py_GIL_DISABLED").

  • Removed py-limited-api entries from pyproject.toml.

  • Attempted to install in editable mode with:

    pip install -e .

⚠️ What fails:

The build still fails due to low-level C header issues during the CFFI extension build:

error: "The limited API is not currently supported in the free-threaded build"
error: unknown type name 'PyMutex'
error: call to undeclared function '_Py_atomic_*'

🧠 My understanding:

It seems that, despite disabling Py_LIMITED_API, some core Python headers or assumptions in libtcod or tcod._libtcod still expect GIL-based behavior and are not currently compatible with the free-threaded interpreter.


🤛 Feature Request:

Would it be possible to support building python-tcod on GIL-disabled Python versions in the future?

Even limited or experimental support (e.g. via conditional compilation) would be amazing for those of us exploring Python's no-GIL future.

But I cannot judge the effort.

@HexDecimal
Copy link
Collaborator

I'm interested in free-threaded no-GIL builds.

From what little research I've done, I suspect the issue is in CFFI itself and that issue might've already been fixed but not in an official release. So I suspect that you don't need to remove references to Py_LIMITED_API from tcod's build files, instead you need to fetch and built cffi directly from its latest source and then tcod will build normally.

Otherwise I'll need to install a GIL-less version of Python on the GitHub workflows in order to verify any fixes.

@HexDecimal HexDecimal added the bug label Apr 19, 2025
@coldwarrl
Copy link
Author

I tried

pip install git+https://github.com/python-cffi/cffi.git

but did not help if you have meant that ...

@HexDecimal
Copy link
Collaborator

I tried

pip install git+https://github.com/python-cffi/cffi.git

but did not help if you have meant that ...

I did mean that. I'm unsure what to try next.

@coldwarrl
Copy link
Author

coldwarrl commented Apr 19, 2025

clang -UNDEBUG -UPy_LIMITED_API ...
/Users/xxx/.pyenv/versions/3.13-dev/include/python3.13t/Python.h:51:4: error: "The limited API is not currently supported in the free-threaded build"

somehow it remains, although I changed the build script.

This limitied api is only needed due to older python versions, right ? Probably you do not want to discard those...

@HexDecimal
Copy link
Collaborator

I assume you've also removed the py_limited_api=True line from build_libtcod.py?

@coldwarrl
Copy link
Author

yes

py_limited_api=not sysconfig.get_config_var("Py_GIL_DISABLED"),

i will look again tomorrow...maybe AI has also some further hints

@coldwarrl
Copy link
Author

coldwarrl commented Apr 20, 2025

this is what the AI analzed:

⚠️ Python-tcod Build Issue with Python 3.13 No-GIL

Problem Summary

Even though the project no longer sets Py_LIMITED_API in the ffi.set_source call, building python-tcod against Python 3.13 (No-GIL variant) still triggers errors related to the limited API. Specifically, compilation fails with:

#error "The limited API is not currently supported in the free-threaded build"

Why This Happens

  1. CFFI Always Includes <Python.h>

    • CFFI-generated .c files (like tcod._libtcod.c) always include the Python C API via #include <Python.h>, regardless of your build flags.
  2. Python 3.13 No-GIL Is Strict About the Limited API

    • Python 3.13’s headers explicitly disallow use of the limited API in no-GIL builds.
    • Even without Py_LIMITED_API being defined, some internal logic or missing defines can still trigger the "limited API" checks.
  3. The Issue Is Triggered Inside Python’s Headers

    • This is not due to an explicit mistake in your own code, but due to changes in Python’s internal headers and how CFFI-generated code interacts with them.

Workarounds and Suggestions

  • Use the standard (GIL-enabled) CPython 3.13 to build python-tcod, if you're not specifically targeting No-GIL compatibility.
  • Add a check in build_libtcod.py to detect unsupported Python builds and fail early with a clear message.
  • Investigate CFFI-level solutions: If possible, suppress or replace #include <Python.h> in the generated C code (this is non-trivial).
  • Upstream Watch: Monitor the status of CFFI and CPython 3.13 development to see if proper No-GIL compatibility will be added.

Conclusion

This issue isn’t caused by incorrect flags in your project, but by deeper incompatibilities between how CFFI-generated files interact with the Python C API headers in No-GIL builds. A future-proof fix will likely require changes in either Python itself or CFFI.


Sounds reasonable to me, but i cannot judge the cffi/c stuff...

A workaround for my project is now to use Rust for multi-core algorithms and call it from python...should be doable, I guess

@HexDecimal
Copy link
Collaborator

HexDecimal commented Apr 20, 2025

This is not due to an explicit mistake in your own code, but due to changes in Python’s internal headers and how CFFI-generated code interacts with them.

There is a blatant lack of specificity here from the AI. The AI seems to state stuff that is relevant but is unable to properly explain how everything interacts with each other.

CFFI needs to include #include <Python.h> in its generated code _libtcod.c. The issue is that CFFI might set Py_LIMITED_API before it includes that header. Looking at _libtcod.c shows how this actually happens as well as some minor documentation on how CFFI uses Py_LIMITED_API.

What I notice is that CFFI internally uses _CFFI_NO_LIMITED_API to disable adding Py_LIMITED_API. Perhaps you can try adding a define for _CFFI_NO_LIMITED_API which should prevent it from defining Py_LIMITED_API in the generated code.

@coldwarrl
Copy link
Author

I have tried that also but no success....so I guess there is no easy solution for now...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy