Skip to content

gh-123152: Add a Concurrency Howto Page #123163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 81 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
4187872
Let "python3" be set at the commandline.
ericsnowcurrently Aug 19, 2024
424dc37
Add a NEWS entry.
ericsnowcurrently Aug 20, 2024
9ab68fa
Add the concurrency howto doc.
ericsnowcurrently Aug 19, 2024
fce455e
Add more explanation.
ericsnowcurrently Aug 20, 2024
28d8ac0
More explanation.
ericsnowcurrently Aug 20, 2024
503d829
Re-structure the comparison table.
ericsnowcurrently Aug 20, 2024
e5d904f
Add basic examples for each concurrency model.
ericsnowcurrently Aug 20, 2024
b7bb9a5
Clarify about "spook action at a distance".
ericsnowcurrently Aug 21, 2024
7efb9b4
Clarify about the "pain" threads can cause.
ericsnowcurrently Aug 21, 2024
daaa02b
Add references to the howto in the library docs.
ericsnowcurrently Aug 21, 2024
ad50bdd
Clean up the Python concurrency models section.
ericsnowcurrently Aug 21, 2024
3a82978
Add a section about concurrent.futures.
ericsnowcurrently Aug 21, 2024
560f26e
Make the examples runnable.
ericsnowcurrently Aug 21, 2024
4f49ed7
Revert changes to Makefile.
ericsnowcurrently Aug 22, 2024
5151e3b
Fix a typo.
ericsnowcurrently Aug 22, 2024
14e59f8
Fix sphinx warnings.
ericsnowcurrently Aug 22, 2024
0b27fd9
Adjust the references.
ericsnowcurrently Aug 22, 2024
0586c72
Fill out and restructure the comparisons.
ericsnowcurrently Aug 22, 2024
8df61ad
Fill out the section about Python theads.
ericsnowcurrently Aug 22, 2024
269eaac
Fill out the section about multiple interpreters.
ericsnowcurrently Aug 22, 2024
7754912
Fill out the section about multprocessing.
ericsnowcurrently Aug 22, 2024
2c9a793
Fill out the section about async/await.
ericsnowcurrently Aug 23, 2024
2975bbb
Shuffle the concurrency model order.
ericsnowcurrently Aug 23, 2024
b82a18b
Fill out the section about distributed concurrency.
ericsnowcurrently Aug 23, 2024
5e7f7c3
Drop an incomplete note.
ericsnowcurrently Aug 23, 2024
c06b843
Fix typos.
ericsnowcurrently Aug 23, 2024
e05e76e
Add a missing divider.
ericsnowcurrently Aug 23, 2024
e7144ec
Add a link to the "overhead" table.
ericsnowcurrently Aug 26, 2024
e131b83
Expand the general explanation of workloads.
ericsnowcurrently Aug 26, 2024
c601e27
Updates for the first two workload examples.
ericsnowcurrently Aug 26, 2024
5c05581
Do not worry about demonstrating a web service.
ericsnowcurrently Aug 27, 2024
a0a8ebb
Move the grep example up.
ericsnowcurrently Aug 27, 2024
c3d1688
Implement threads for the grep example.
ericsnowcurrently Aug 27, 2024
48ac79d
Implement grep using concurrent.futures.
ericsnowcurrently Aug 28, 2024
9a1a81c
Fix nested lists and add quadrants.
ericsnowcurrently Aug 29, 2024
03156b5
Implement grep using multiprocessing.
ericsnowcurrently Aug 29, 2024
e0d833e
Start reorganizing, and dump a bunch of explanation.
ericsnowcurrently Aug 29, 2024
6dd91e3
Clarify about Python-supported concurrency models.
ericsnowcurrently Sep 3, 2024
1e935df
Small clarifications and cleanup.
ericsnowcurrently Sep 3, 2024
4fdf9f5
Move the high-level APIs section down.
ericsnowcurrently Sep 3, 2024
cb843f6
Clarify about the comparisons.
ericsnowcurrently Sep 3, 2024
8a87617
Updates about free-threading caveats.
ericsnowcurrently Sep 3, 2024
288c4f4
Formatting and links for intro section.
ericsnowcurrently Sep 4, 2024
0b768ae
wording tweaks
ericsnowcurrently Sep 4, 2024
ab24e83
Update the section about multiple interpreters.
ericsnowcurrently Sep 4, 2024
f7f1769
Update the section about coroutines.
ericsnowcurrently Sep 4, 2024
fdf46cb
Update the sections about multi-processing.
ericsnowcurrently Sep 4, 2024
4cc62a1
Clear out the section on downsides.
ericsnowcurrently Sep 4, 2024
54bacb6
Update the section about shared resources.
ericsnowcurrently Sep 4, 2024
49ec62f
Normalize TODO comments.
ericsnowcurrently Sep 4, 2024
8fd8406
Move the explanations to the right places.
ericsnowcurrently Sep 4, 2024
fdcc128
Add sub-sections.
ericsnowcurrently Sep 4, 2024
3033875
Move the caveats subsections around.
ericsnowcurrently Sep 4, 2024
3e4ab6a
Fix formatting.
ericsnowcurrently Sep 4, 2024
f07c696
Drop a TODO.
ericsnowcurrently Sep 4, 2024
7a04361
Hide the incomplete portions of text.
ericsnowcurrently Sep 4, 2024
f9815bb
Tweak a table header.
ericsnowcurrently Sep 4, 2024
48551a7
Fix a table header.
ericsnowcurrently Sep 4, 2024
eac23f9
Fix the sidebar table of contents.
ericsnowcurrently Sep 16, 2024
3e1c53e
Expand the grep section and code.
ericsnowcurrently Sep 16, 2024
9f6ea1f
Make the examples more uniform.
ericsnowcurrently Sep 16, 2024
3aaa4d1
Fix some wording.
ericsnowcurrently Sep 17, 2024
3a7b0be
Reorganize the example code a little.
ericsnowcurrently Sep 17, 2024
36edcce
Split up the examples file.
ericsnowcurrently Sep 18, 2024
7b4af12
Tweak CLI.
ericsnowcurrently Sep 18, 2024
bc6e725
Simplify the examples.
ericsnowcurrently Sep 19, 2024
f4749f6
Drop all the context manager stuff.
ericsnowcurrently Sep 19, 2024
eaaecca
Implement the async example.
ericsnowcurrently Sep 20, 2024
ebf3130
Drop the text about do_search().
ericsnowcurrently Sep 20, 2024
21ff9e1
Fix a typo.
ericsnowcurrently Sep 23, 2024
3682348
Move the grep examples to individual files.
ericsnowcurrently Sep 23, 2024
10ee369
Tweak run-examples.py to run the newer example code.
ericsnowcurrently Sep 23, 2024
e6f0088
Drop the combined grep package.
ericsnowcurrently Sep 24, 2024
3f2637b
Fix the highlighting.
ericsnowcurrently Sep 24, 2024
9ce6ccf
cleanups
ericsnowcurrently Sep 24, 2024
10a6571
Fix typos and wording.
ericsnowcurrently Sep 24, 2024
793bfa0
Limit side-by-side example lines to 60 characters.
ericsnowcurrently Sep 24, 2024
3e19d7e
Fix the examples.
ericsnowcurrently Sep 24, 2024
f826c9c
Fix a ref.
ericsnowcurrently Sep 24, 2024
049f1de
lint
ericsnowcurrently Sep 25, 2024
7686e10
Merge branch 'main' into concurrency-howto
AA-Turner Apr 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update the section about coroutines.
  • Loading branch information
ericsnowcurrently committed Sep 4, 2024
commit f7f176994a351922aee6b3b718f34b9f341b5795
73 changes: 18 additions & 55 deletions Doc/howto/concurrency.rst
Original file line number Diff line number Diff line change
Expand Up @@ -575,69 +575,32 @@ As the work on isolation wraps up, improvements will shift to focus
on performance and memory usage. Thus the overhead associated with
using multiple interpreters will drastically decrease over time.

Coroutines are contagious
^^^^^^^^^^^^^^^^^^^^^^^^^

Coroutines can be an effective mechanism for letting a program's
non-blocking code run while simultaneously waiting for blocking code
to finish. The tricky part is that the underlying machinery (the
:ref:`event loop <asyncio-event-loop>`) relies on each coroutine
explicitly yielding control at the appropriate moments.

Normal functions do not follow this pattern, so they cannot take
advantage of that cooperative scheduling to avoid blocking
the program. Thus coroutines and non-coroutines don't mix well.
While there are tools for wrapping normal functions to act like
coroutines, they are often converted into coroutines instead.
At that point, if any non-async code relies on the function then
either you'll need to convert the other code a coroutine or you'll
need to keep the original non-async implementation around along
with the new, almost identical async one.

You can see how that can proliferate, leading to possible extra
maintenance/development costs.

Coroutines (Async/Await)
^^^^^^^^^^^^^^^^^^^^^^^^

The use of :term:`coroutines <coroutine>` for concurrency has been
around a long time and has grown in popularity in the software world,
particularly with the addition of ``async/await`` syntax in
various languages.

Python has supported coroutines to some degree since the beginning.
The best example is :pypi:`twisted`, which has provided this concurrency
model for decades. For most of that time :pypi:`!twisted` did it
primarily through callbacks and a form of "promises"/"futures".

Explicit support for coroutines in Python really started with the
introduction of :term:`generators <generator>` in Python 2.2
(:pep:`255`). In Python 2.5 (:pep:`342`), :term:`!generators` were
tweaked to explicitly support use as coroutines. That went a step
further in Python 3.3 with the addition of ``yield from`` (:pep:`380`)
and the :mod:`asyncio` module (:pep:`3156`). Finally, in Python 3.5
(:pep:`492`), we got dedicated ``async/await`` syntax
and :ref:`a dedicated protocol <coroutine-protocol>`
for :term:`!coroutine` objects.

There are three main pieces to using coroutines:

* coroutines (non-blocking, yield control instead)
* an event loop (schedules coroutines)
* coroutine wrappers around blocking operations

A :term:`coroutine function` looks *almost* the same as a regular
function. It is a non-blocking function that *cooperatively* yields
control of the program to other coroutines, which in turn yield control
back (eventually). At those points of synchronization,
coroutines often provide data to one another.

The event loop is what keeps track of which coroutines have yielded
control and which should get control next.

Generally a coroutine needs to avoid doing anything that takes very long
before yielding control back to the event loop. Any blocking operation
in a coroutine, like waiting on a socket, has to be implemented in a way
that only waits a little while, yields, and then waits again, etc. until
ready. The alternative is to wrap the blocking operation/function
in some sort of "future" coroutine that yields until the blocking
operation completes. The event loop can also fill that role
to an extent.

In addition to support for coroutines in the language, Python's stdlib
provides the :mod:`asyncio` module, which includes:

* an event loop
* a number of useful coroutines
* a variety of helpful APIs that build on coroutines and the event loop

One of the main challenges with using coroutines is that they do not
normally mix well with non-coroutines. As a result, ``async/await``
can be contagious, requiring surrounding code to be async. This can
lead to having the same thing implemented twice, once normal and once
async, with significant code duplication.

Multi-processing
^^^^^^^^^^^^^^^^
Expand Down
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy