Skip to content

Fix slotted reference cycles on 3.14 #1446

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jul 28, 2025
Merged

Fix slotted reference cycles on 3.14 #1446

merged 6 commits into from
Jul 28, 2025

Conversation

hynek
Copy link
Member

@hynek hynek commented Jul 22, 2025

@@ -845,6 +857,13 @@ def _create_slots_class(self):
if k not in (*tuple(self._attr_names), "__dict__", "__weakref__")
}

if PY_3_14_PLUS:
# Clean up old dict to avoid leaks.
old_cls_dict = self._cls.__dict__ | _deproxier
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd recommend putting all this in a try-except in case we break this trick in a later version of CPython; if so, it feels better for your users to leak the original class than to get a mysterious error about mappingproxies.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JelleZijlstra given context from Discord, we’re kinda weighting “broken introspection” vs “things explode if someone adds another decorator at the wrong position” anyways?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this variant definitely has some questionable edge cases. I think you can be a bit less conservative in CPython in terms of judging whether this hack is worth it.

As I understand from Brandt, a problem with the hack is that it mutates the class dict without incrementing the internal version tag for the type object. A workaround could then be to perform some no-op that increments the version tag anyway. For example, setting cls.__abstractmethods__ to its existing value could work.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can be a bit less conservative in CPython in terms of judging whether this hack is worth it.

I'm afraid we have to be more conservative, because in attrs slots are on by default. :|

But looking again at the code, which part is actually dangerous here? It's just the wekreaf part, no? We do not modify self._cls.__dict__ – we just use your escape hatch to break cycles.

E.g. I tried this:

def test_no_references_to_original_when_using_cached_property(self):
        """
        When subclassing a slotted class and using cached property, there are
        no stray references to the original class.
        """

        @attr.s(slots=True)
        class C:
            pass

        tmp = None

        def decorator(cls):
            nonlocal tmp
            tmp = cls
            return cls

        @attr.s(slots=True)
        @decorator
        class C2(C):
            @functools.cached_property
            def value(self) -> int:
                return 0

        print(tmp.__dict__)

        # The original C2 is in a reference cycle, so force a collect:
        gc.collect()

        print(tmp.__dict__)

        assert [C2] == C.__subclasses__()

and it obviously fails, because there's a legit reference to the original class within tmp. But both prints work and print what would be expected without segfaulting.

Doesn't this mean that the only downside/edge case (aside from it potentially breaking eventually) is that the original class can't be weakref'ed? I'm not super deep in this topic, so happy to be enlightened. That would def be a downside I'd be OK to accept.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the risky part is removing "__dict__" from the __dict__. The interpreter uses various caches and optimizations that are keyed on the "type version", a number that gets incremented when the type is modified. Because we use this trick to get to the type dict (which usually can't be modified directly from Python code), we're modifying the type dict without bumping the version number. That can lead to crashes if interpreter code assumes the dict is unchanged from before.

There is a separate risk that you break users of the original class who rely on the __dict__/__weakref__ slots. But that is Python-level breakage (things throw exceptions), not C-level breakage (the interpreter crashes).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit more color on what exactly this breaks on the affected class (the original class that we want to be garbage collected):

  • Removing __weakref__ actually breaks very little that matters. Even after the slot is deleted, weakref.ref() still works on these objects. It's just the Python-visible obj.__weakref__ attribute that is gone, but as far as I can tell basically nothing relies on that.
  • Removing __dict__ means that obj.__dict__ no longer works, but direct attribute access (obj.x = 1) still works.

del self._cls.__weakref__

# Manually bump internal version tag.
self._cls.__name__ = self._cls.__name__
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JelleZijlstra so __abstractmethods__ didn't work, because it's not omnipresent – will this work too?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, modifying the name doesn't bump the type version. I think if there's no __abstractmethods__, you can set it to a nonempty value and then del it.

hynek added a commit that referenced this pull request Jul 23, 2025
Comment on lines 867 to 872
# Manually bump internal version tag.
if hasattr(self._cls, "__abstractmethods__"):
self._cls.__abstractmethods__ = self._cls.__abstractmethods__
else:
self._cls.__abstractmethods__ = frozenset({"__init__"})
del self._cls.__abstractmethods__
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better now @JelleZijlstra? i have to admit i'm fishing a bit in the dark here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it should work! Note this is still technically unsafe in the presence of threading: if somebody in another thread interacts with the class while you are doing this, things may explode.

@hynek hynek requested a review from Tinche July 23, 2025 15:08
hynek added a commit that referenced this pull request Jul 23, 2025
hynek added a commit that referenced this pull request Jul 25, 2025
hynek added a commit that referenced this pull request Jul 25, 2025
hynek added a commit that referenced this pull request Jul 27, 2025
hynek and others added 5 commits July 27, 2025 12:39
Ref python/cpython#136893 &
python/cpython#135228

Co-authored-by: Jelle Zijlstra <906600+JelleZijlstra@users.noreply.github.com>
Co-authored-by: Brandt Bucher
<40968415+brandtbucher@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra
<906600+JelleZijlstra@users.noreply.github.com>
@hynek hynek added this pull request to the merge queue Jul 28, 2025
Merged via the queue into main with commit 000c563 Jul 28, 2025
19 checks passed
@hynek hynek deleted the fix-slots-314 branch July 28, 2025 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy