-
-
Notifications
You must be signed in to change notification settings - Fork 387
Fix slotted reference cycles on 3.14 #1446
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@@ -845,6 +857,13 @@ def _create_slots_class(self): | |||
if k not in (*tuple(self._attr_names), "__dict__", "__weakref__") | |||
} | |||
|
|||
if PY_3_14_PLUS: | |||
# Clean up old dict to avoid leaks. | |||
old_cls_dict = self._cls.__dict__ | _deproxier |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd recommend putting all this in a try-except in case we break this trick in a later version of CPython; if so, it feels better for your users to leak the original class than to get a mysterious error about mappingproxies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@JelleZijlstra given context from Discord, we’re kinda weighting “broken introspection” vs “things explode if someone adds another decorator at the wrong position” anyways?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this variant definitely has some questionable edge cases. I think you can be a bit less conservative in CPython in terms of judging whether this hack is worth it.
As I understand from Brandt, a problem with the hack is that it mutates the class dict without incrementing the internal version tag for the type object. A workaround could then be to perform some no-op that increments the version tag anyway. For example, setting cls.__abstractmethods__
to its existing value could work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you can be a bit less conservative in CPython in terms of judging whether this hack is worth it.
I'm afraid we have to be more conservative, because in attrs slots are on by default. :|
But looking again at the code, which part is actually dangerous here? It's just the wekreaf part, no? We do not modify self._cls.__dict__
– we just use your escape hatch to break cycles.
E.g. I tried this:
def test_no_references_to_original_when_using_cached_property(self):
"""
When subclassing a slotted class and using cached property, there are
no stray references to the original class.
"""
@attr.s(slots=True)
class C:
pass
tmp = None
def decorator(cls):
nonlocal tmp
tmp = cls
return cls
@attr.s(slots=True)
@decorator
class C2(C):
@functools.cached_property
def value(self) -> int:
return 0
print(tmp.__dict__)
# The original C2 is in a reference cycle, so force a collect:
gc.collect()
print(tmp.__dict__)
assert [C2] == C.__subclasses__()
and it obviously fails, because there's a legit reference to the original class within tmp. But both prints work and print what would be expected without segfaulting.
Doesn't this mean that the only downside/edge case (aside from it potentially breaking eventually) is that the original class can't be weakref'ed? I'm not super deep in this topic, so happy to be enlightened. That would def be a downside I'd be OK to accept.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, the risky part is removing "__dict__"
from the __dict__
. The interpreter uses various caches and optimizations that are keyed on the "type version", a number that gets incremented when the type is modified. Because we use this trick to get to the type dict (which usually can't be modified directly from Python code), we're modifying the type dict without bumping the version number. That can lead to crashes if interpreter code assumes the dict is unchanged from before.
There is a separate risk that you break users of the original class who rely on the __dict__
/__weakref__
slots. But that is Python-level breakage (things throw exceptions), not C-level breakage (the interpreter crashes).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A bit more color on what exactly this breaks on the affected class (the original class that we want to be garbage collected):
- Removing
__weakref__
actually breaks very little that matters. Even after the slot is deleted,weakref.ref()
still works on these objects. It's just the Python-visibleobj.__weakref__
attribute that is gone, but as far as I can tell basically nothing relies on that. - Removing
__dict__
means thatobj.__dict__
no longer works, but direct attribute access (obj.x = 1
) still works.
src/attr/_make.py
Outdated
del self._cls.__weakref__ | ||
|
||
# Manually bump internal version tag. | ||
self._cls.__name__ = self._cls.__name__ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@JelleZijlstra so __abstractmethods__
didn't work, because it's not omnipresent – will this work too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, modifying the name doesn't bump the type version. I think if there's no __abstractmethods__
, you can set it to a nonempty value and then del
it.
src/attr/_make.py
Outdated
# Manually bump internal version tag. | ||
if hasattr(self._cls, "__abstractmethods__"): | ||
self._cls.__abstractmethods__ = self._cls.__abstractmethods__ | ||
else: | ||
self._cls.__abstractmethods__ = frozenset({"__init__"}) | ||
del self._cls.__abstractmethods__ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
better now @JelleZijlstra? i have to admit i'm fishing a bit in the dark here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like it should work! Note this is still technically unsafe in the presence of threading: if somebody in another thread interacts with the class while you are doing this, things may explode.
Ref python/cpython#136893 & python/cpython#135228 Co-authored-by: Jelle Zijlstra <906600+JelleZijlstra@users.noreply.github.com>
Co-authored-by: Brandt Bucher <40968415+brandtbucher@users.noreply.github.com> Co-authored-by: Jelle Zijlstra <906600+JelleZijlstra@users.noreply.github.com>
Ref python/cpython#136893 & python/cpython#135228