Open
Description
Bug report
Bug description:
Code:
from concurrent.futures import ThreadPoolExecutor
data = [
list(range(0, 5)),
list(range(5, 10)),
]
def _f(x):
print(f"Processing {x}")
return True
print("=== 1 - No consumption from the iterator ===")
executor = ThreadPoolExecutor(max_workers=1)
for ints in data:
executor.map(_f, ints)
executor.shutdown(wait=True)
print("=== 2 - Consume all values from the iterator ===")
executor = ThreadPoolExecutor(max_workers=1)
for ints in data:
futures = executor.map(_f, ints)
results = list(futures)
executor.shutdown(wait=True)
print("=== 3 - Consume one value from the iterator ===")
executor = ThreadPoolExecutor(max_workers=1)
for ints in data:
futures = executor.map(_f, ints)
first = next(futures)
executor.shutdown(wait=True)
print("=== 4 - Dropping iterator cancels remaining futures ===")
executor = ThreadPoolExecutor(max_workers=1)
futures = executor.map(_f, range(0, 5))
first = next(futures)
del futures
executor.shutdown(wait=True)Result:
=== 1 - No consumption from the iterator ===
Processing 0
Processing 1
Processing 2
Processing 3
Processing 4
Processing 5
Processing 6
Processing 7
Processing 8
Processing 9
=== 2 - Consume all values from the iterator ===
Processing 0
Processing 1
Processing 2
Processing 3
Processing 4
Processing 5
Processing 6
Processing 7
Processing 8
Processing 9
=== 3 - Consume one value from the iterator ===
Processing 0
Processing 1
Processing 5
Processing 6
Processing 7
Processing 8
Processing 9
=== 4 - Dropping iterator cancels remaining futures ===
Processing 0
Processing 1
The behaviour seems to be:
- If the iterator returned from
mapis never used (case 1), futures are not cancelled - If the iterator returned from
mapis exhausted (case 2), futures are not cancelled - If the iterator returned from
mapis partially consumed and then dropped (cases 3 & 4), the remaining futures are cancelled
We hit this doing a version of case 3, calling any on the iterator, which short-circuited, causing the remaining futures to not execute. This tripped us up and seems like quite a confusing behaviour that is not flagged in the docs.
It looks like this is caused by this code: https://github.com/python/cpython/blob/main/Lib/concurrent/futures/_base.py#L669-L671
Possibly related to #108518
CPython versions tested on:
3.12
Operating systems tested on:
Linux