[Python-Dev] PEP 554 v2 (new "interpreters" module)

[Antoine Pitrou]

On Tue, 12 Sep 2017 14:43:43 -0700
Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Sat, Sep 9, 2017 at 5:05 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Fri, 8 Sep 2017 16:04:27 -0700, Eric Snow <ericsnowcurrently at gmail.com> wrote:  
> >> ``list()``::  
> >
> > It's called ``enumerate()`` in the threading module.  Not sure there's
> > a point in choosing a different name here.  
> 
> Yeah, in the first version of the PEP it was called "enumerate()".  I
> changed it to "list()" at Raymond's recommendation.  The reasoning is
> that it's less confusing to most people that way.  TBH, I'd rather
> leave it "list()", but could be swayed.  Perhaps it would be enough
> for the PEP to not mention any relationship to "threading"?

Yeah, given the other clarifications you did, I agree not mentioning
threading at all would be better, since the PEP itself does not strive
to provide new concurrency primitives.

As for the naming, let's make it both unconfusing and explicit?
How about three functions: `all_interpreters()`,
`running_interpreters()` and `idle_interpreters()`, for example?

> > And why guarantee that it executes in the "current OS thread"?
> > I would say you don't want to specify where it executes exactly, as it
> > opens the door for more sophisticated implementations (such as
> > automatic assignment of subinterpreters inside a pool of threads).  
> 
> Again, I had explained this poorly in the PEP.  The key thing here is
> that subinterpreters don't do anything special relative to threading.
> If you want to call "Interpreter.run()" in a thread then you stick it
> in a "threading.Thread".  If you want to auto-assign to a pool of
> threads then you treat it like any other function you would
> auto-assign to a pool of threads.

Makes sense, thanks.  Perhaps you want to mention that example quickly
in the PEP? (or perhaps not)

> > Has any thought been given to how FIFOs could integrate with async code
> > driven by an event loop (e.g. asyncio)?  I think the model of executing
> > several asyncio (or Tornado) applications each in their own
> > subinterpreter may prove quite interesting to reconcile multi-core
> > concurrency with ease of programming.  That would require the FIFOs to
> > be able to synchronize on something an event loop can wait on (probably
> > a file descriptor?).  
> 
> Personally I've given pretty much no thought to the relationship with
> async.  TBH, the FIFO parts of the PEP were added only recently and
> haven't fully baked yet.  I'd be interested more feedback on async
> relative to PEP, not just with the FIFO bits; my experience with async
> is pretty limited thus far.

I think the FIFO is the primary point where interaction with event loop
would prove useful, since that's how you would communicate and
synchronize with other interpreters.

Actually, even waiting on an interpreter's state (e.g. running / idle)
could use a "hidden" FIFO sending the right messages (for example the
strings "running" and "idle", for a crude proof-of-concept).

Regards

Antoine.