Skip to content

gh-92203: Add closure support to exec(). #92204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
May 6, 2022
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion Doc/library/functions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -496,7 +496,7 @@ are always available. They are listed here in alphabetical order.
n += 1


.. function:: eval(expression[, globals[, locals]])
.. function:: eval(expression[, globals[, locals]], *, closure=None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the docs for eval, but you actually changed only exec.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. Fixed.

I put the description of the closure argument correctly in the docs for exec, I just modified the wrong prototype. (I kept making this dumb mistake--originally the branch was called eval_closure.)


The arguments are a string and optional globals and locals. If provided,
*globals* must be a dictionary. If provided, *locals* can be any mapping
Expand Down Expand Up @@ -576,6 +576,11 @@ are always available. They are listed here in alphabetical order.
builtins are available to the executed code by inserting your own
``__builtins__`` dictionary into *globals* before passing it to :func:`exec`.

The *closure* argument specifies a closure--a tuple of cellvars.
It's only valid when the *object* is a code object containing free variables.
The length of the tuple must exactly match the number of free variables
referenced by the code object.

.. audit-event:: exec code_object exec

Raises an :ref:`auditing event <auditing>` ``exec`` with the code object
Expand All @@ -594,6 +599,9 @@ are always available. They are listed here in alphabetical order.
Pass an explicit *locals* dictionary if you need to see effects of the
code on *locals* after function :func:`exec` returns.

.. versionchanged:: 3.11
Added the *closure* parameter.


.. function:: filter(function, iterable)

Expand Down
44 changes: 43 additions & 1 deletion Lib/test/test_builtin.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
from inspect import CO_COROUTINE
from itertools import product
from textwrap import dedent
from types import AsyncGeneratorType, FunctionType
from types import AsyncGeneratorType, FunctionType, CellType
from operator import neg
from test import support
from test.support import (swap_attr, maybe_get_event_loop_policy)
Expand Down Expand Up @@ -772,6 +772,48 @@ def test_exec_redirected(self):
finally:
sys.stdout = savestdout

def test_exec_closure(self):
result = 0
def make_closure_functions():
a = 2
b = 3
c = 5
def two_freevars():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It actually has three, and the other one has four.

(Pdb) p two_freevars.__closure__
(<cell at 0x107c8b710: int object at 0x106518360>, <cell at 0x107c8b6d0: int object at 0x106518380>, <cell at 0x107c8bc50: int object at 0x106518320>)
(Pdb) p three_freevars.__closure__
(<cell at 0x107c8b710: int object at 0x106518360>, <cell at 0x107c8b6d0: int object at 0x106518380>, <cell at 0x107c8b610: int object at 0x1065183c0>, <cell at 0x107c8bc50: int object at 0x106518320>)

More importantly, it would be good to have tests passing in various bad arguments: non-tuples, tuples containing non-cells.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done:

  • Renamed three_freevars to four_freevars.
  • Then, renamed two_freevars to three_freevars.
  • Added tests passing in a correctly-sized list of cell vars instead of a tuple, and a tuple of the right length with one non-cell-var entry.

nonlocal result
nonlocal a
nonlocal b
result = a*b
def three_freevars():
nonlocal result
nonlocal a
nonlocal b
nonlocal c
result = a*b*c
return two_freevars, three_freevars
two_freevars, three_freevars = make_closure_functions()

exec(two_freevars.__code__,
two_freevars.__globals__,
closure=two_freevars.__closure__)
self.assertEquals(result, 6)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self.assertEquals(result, 6)
self.assertEqual(result, 6)

assertEquals is deprecated

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


result = 0
my_closure = list(two_freevars.__closure__)
my_closure[0] = CellType(35)
my_closure[1] = CellType(72)
my_closure = tuple(my_closure)
exec(two_freevars.__code__,
two_freevars.__globals__,
closure=my_closure)
self.assertEquals(result, 2520)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self.assertEquals(result, 2520)
self.assertEqual(result, 2520)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


self.assertRaises(TypeError,
exec,
two_freevars.__code__,
two_freevars.__globals__,
closure=three_freevars.__closure__)


def test_filter(self):
self.assertEqual(list(filter(lambda c: 'a' <= c <= 'z', 'Hello World')), list('elloorld'))
self.assertEqual(list(filter(None, [1, 'hello', [], [3], '', None, 9, 0])), [1, 'hello', [3], 9])
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Add a closure keyword-only parameter to exec(). It can only be specified
when exec-ing a code object that uses free variables. When specified, it
must be a tuple, with exactly the number of cell variables referenced by the
code object. closure has a default value of None, and it must be None if the
code object doesn't refer to any free variables.
60 changes: 52 additions & 8 deletions Python/bltinmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -977,6 +977,8 @@ exec as builtin_exec
globals: object = None
locals: object = None
/
*
closure: object(c_default="NULL") = None

Execute the given source in the context of globals and locals.

Expand All @@ -985,12 +987,14 @@ or a code object as returned by compile().
The globals must be a dictionary and locals can be any mapping,
defaulting to the current globals and locals.
If only globals is given, locals defaults to it.
The closure must be a tuple of cellvars, and can only be used
when source is a code object requiring exactly that many cellvars.
[clinic start generated code]*/

static PyObject *
builtin_exec_impl(PyObject *module, PyObject *source, PyObject *globals,
PyObject *locals)
/*[clinic end generated code: output=3c90efc6ab68ef5d input=01ca3e1c01692829]*/
PyObject *locals, PyObject *closure)
/*[clinic end generated code: output=7579eb4e7646743d input=f13a7e2b503d1d9a]*/
{
PyObject *v;

Expand Down Expand Up @@ -1029,20 +1033,60 @@ builtin_exec_impl(PyObject *module, PyObject *source, PyObject *globals,
return NULL;
}

if (closure == Py_None) {
closure = NULL;
}

if (PyCode_Check(source)) {
Py_ssize_t num_free = PyCode_GetNumFree((PyCodeObject *)source);
if (num_free == 0) {
if (closure) {
PyErr_SetString(PyExc_ValueError,
"code object cannot use a closure");
return NULL;
}
} else {
int closure_is_ok =
closure
&& PyTuple_CheckExact(closure)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not allow subclasses of tuple?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I erred on the side of caution. I don't know if all the code that deals with closure objects inside CPython accept subclasses of tuple, or iterables generally; it seemed to me that the safest route was to require what CPython itself uses, which is exactly a tuple object. If this proves too restrictive we can relax the restriction later, once we prove to ourselves that it's safe to do so.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyFunction_SetClosure() accepts subclasses of tuple.

&& (PyTuple_GET_SIZE(closure) == num_free);
if (closure_is_ok) {
for (Py_ssize_t i = 0; i < num_free; i++) {
PyObject *cell = PyTuple_GET_ITEM(closure, i);
if (!PyCell_Check(cell)) {
closure_is_ok = 0;
break;
}
}
}
if (!closure_is_ok) {
PyErr_Format(PyExc_TypeError,
"code object requires a closure of exactly length %d",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"requires a tuple" or "requires a tuple of closures"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I imagine the textbook definition of a "closure" is different, Python uses the word "closure" to describe this object (a tuple of CellVars) in several places. For example, the __closure__ attribute of a FunctionObject, or the fc_closure field of a FrameContructor struct.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if change it to "closure should be None or a tuple of cells of exactly length %zd"?

Note also %zd instead of %d.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyFunction_SetClosure() does not check the length of the tuple or the type of items, so perhaps it is not necessary here too, if the following code raises an exception of appropriate type.

num_free);
return NULL;
}
}

if (PySys_Audit("exec", "O", source) < 0) {
return NULL;
}

if (PyCode_GetNumFree((PyCodeObject *)source) > 0) {
PyErr_SetString(PyExc_TypeError,
"code object passed to exec() may not "
"contain free variables");
return NULL;
if (!closure) {
v = PyEval_EvalCode(source, globals, locals);
} else {
v = PyEval_EvalCodeEx(source, globals, locals,
NULL, 0,
NULL, 0,
NULL, 0,
NULL,
closure);
}
v = PyEval_EvalCode(source, globals, locals);
}
else {
if (closure != NULL) {
PyErr_SetString(PyExc_ValueError,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not it be a TypeError?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

"closure cannot be used when source is a string");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, the source can be a bytes-like object.

Would not be better to say "closure can only be used when source is a code object"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

}
PyObject *source_copy;
const char *str;
PyCompilerFlags cf = _PyCompilerFlags_INIT;
Expand Down
37 changes: 26 additions & 11 deletions Python/clinic/bltinmodule.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy