`UNSUBSCRIBE` hangs with redis-py 5.3.0 #9691

WilliamDEdwards · 2025-05-07T16:04:11Z

WilliamDEdwards
May 7, 2025

With Celery 5.5.2 and redis-py 5.2.1, all is well.

With Celery 5.5.2 and redis-py 5.3.0, some tests are stuck on UNSUBSCRIBE. I can't find a pattern between these tests.

Backtrace

Traceback (most recent call first):
  <built-in method __enter__ of _thread.lock object at remote 0x7ff5a1f3a880>
  File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 858, in execute_command
    with self._lock:
  File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1043, in unsubscribe
    return self.execute_command("UNSUBSCRIBE", *args)
  File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 184, in cancel_for
    self._pubsub.unsubscribe(key)
  File "/usr/local/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 216, in on_result_fulfilled
    self.result_consumer.cancel_for(result.id)
  File "/usr/local/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 208, in remove_pending_result
    self.on_result_fulfilled(result)
  File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 417, in __del__
    self.backend.remove_pending_result(self)
  Garbage-collecting
  File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 894, in _execute
    def _execute(self, conn, command, *args, **kwargs):
  File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 859, in execute_command
    self._execute(connection, connection.send_command, *args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 1043, in unsubscribe
    return self.execute_command("UNSUBSCRIBE", *args)
  File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 184, in cancel_for
    self._pubsub.unsubscribe(key)
  File "/usr/local/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 216, in on_result_fulfilled
    self.result_consumer.cancel_for(result.id)
  File "/usr/local/lib/python3.11/site-packages/celery/backends/asynchronous.py", line 208, in remove_pending_result
    self.on_result_fulfilled(result)
  File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 119, in _on_fulfilled
    self.backend.remove_pending_result(self)
  File "/usr/local/lib/python3.11/site-packages/vine/promises.py", line 158, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 436, in _maybe_set_cache
    self.on_ready(self)
  File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 442, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 503, in state
    return self._get_task_meta()['status']
  File "/builds/core/core-api/app/services/celery/task_collections.py", line 55, in get_results
    original_state = celery_obj.status.lower()
  File "/builds/core/core-api/tests/utilities/task_collections.py", line 52, in get_results
    results, _ = services.celery.celery_task_collection_service.get_results(
  File "/builds/core/core-api/tests/utilities/task_collections.py", line 29, in get_task
    results = get_results(database_session, task_collection_uuid)
  File "/builds/core/core-api/tests/utilities/task_collections.py", line 125, in assert_task_collection
    state, message, task_uuid = get_task(
  File "/builds/core/core-api/tests/feature_tests/app/api/api_v1/controllers/public/test_nodes.py", line ?, in test_create_task_collection
    (failed to get frame line number)
  File "/usr/local/lib/python3.11/site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call

This is in the context of celery.contrib.pytest tests. I haven't tested this with a 'real' worker.

According to the redis-py changelog (https://github.com/redis/redis-py/releases/tag/v5.3.0), one thing related to locking changed: redis/redis-py#3473. This change doesn't seem to have any functional implications, though.

Why a discussion instead of an issue? I'm not sure if this is Redis's fault, or a compatibility issue between Celery and Redis. I do see that a discussion on the Redis PR wasn't resolved (https://github.com/redis/redis-py/pull/3473/files#r1905748019), while the PR is merged - which makes me wonder if the change was tested thoroughly on Redis's end.

Finally, from #9680: "kombu does not support 5.3.0 yet." I don't know why not.

Additional information

(.venv) Williams-MacBook-Air:core-api williamdavidedwards$ pip freeze
alembic==1.15.2
amqp==5.3.1
annotated-types==0.7.0
anyio==4.7.0
bcrypt==4.2.1
beautifulsoup4==4.13.4
billiard==4.2.1
boto3==1.38.9
botocore==1.38.10
bugsnag==4.7.1
celery==5.5.2
certifi==2025.4.26
cffi==1.17.1
cfgv==3.4.0
charset-normalizer==3.4.0
click==8.1.8
click-didyoumean==0.3.1
click-plugins==1.1.1
click-repl==0.3.0
coverage==7.8.0
croniter==6.0.0
cryptography==44.0.3
debugpy==1.8.12
distlib==0.3.9
dnspython==2.7.0
docker==7.1.0
ecdsa==0.19.0
elastic-transport==8.15.1
elasticsearch==8.18.1
email_validator==2.2.0
execnet==2.1.1
factory_boy==3.3.1
Faker==33.1.0
fastapi==0.115.12
fastapi-cli==0.0.6
filelock==3.16.1
flower==2.0.1
freezegun==1.5.1
h11==0.14.0
httpcore==1.0.7
httptools==0.6.4
httpx==0.28.1
humanize==4.11.0
identify==2.6.3
idna==3.10
importlib_metadata==8.6.1
inflection==0.5.1
iniconfig==2.0.0
itsdangerous==2.2.0
Jinja2==3.1.6
jmespath==1.0.1
kombu==5.5.3
Mako==1.3.8
markdown-it-py==3.0.0
MarkupSafe==3.0.2
mdurl==0.1.2
mypy==1.15.0
mypy-extensions==1.0.0
netaddr==1.3.0
nodeenv==1.9.1
orjson==3.10.12
packaging==24.2
passlib==1.7.4
pika==1.3.2
platformdirs==4.3.6
pluggy==1.5.0
polyfactory==2.21.0
pre_commit==4.2.0
prometheus_client==0.21.1
prompt_toolkit==3.0.48
protobuf==3.20.3
psutil==6.1.1
pyasn1==0.4.8
pycparser==2.22
pydantic==2.11.4
pydantic-extra-types==2.10.1
pydantic-settings==2.9.1
pydantic_core==2.33.2
pyfakefs==5.7.4
Pygments==2.18.0
PyJWT==2.9.0
PyMySQL==1.1.1
pyOpenSSL==25.0.0
pytest==8.3.5
pytest-asyncio==0.25.3
pytest-cov==6.1.1
pytest-docker-tools==3.1.3
pytest-factoryboy==2.7.0
pytest-mock==3.14.0
pytest-rerunfailures==15.0
pytest-socket==0.7.0
pytest-timeout==2.4.0
pytest-xdist==3.6.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-jose==3.4.0
python-multipart==0.0.19
python3-cyberfusion-rpc-client==1.0.1
pytz==2025.2
PyYAML==6.0.2
redis==5.3.0
requests==2.32.3
requests-mock==1.12.1
rich==13.9.4
rich-toolkit==0.12.0
rsa==4.9
ruff==0.11.8
s3transfer==0.12.0
shellingham==1.5.4
six==1.17.0
sniffio==1.3.1
soupsieve==2.6
SQLAlchemy==2.0.40
SQLAlchemy-Utils==0.41.2
starlette==0.45.3
tenacity==9.0.0
tornado==6.4.2
typer==0.15.3
types-certifi==2021.10.8.3
types-cffi==1.16.0.20240331
types-croniter==6.0.0.20250411
types-pyOpenSSL==24.1.0.20240722
types-pytz==2025.2.0.20250326
types-PyYAML==6.0.12.20250402
types-requests==2.32.0.20250328
types-setuptools==75.6.0.20241126
typing-inspection==0.4.0
typing_extensions==4.12.2
tzdata==2025.2
ujson==5.10.0
urllib3==2.2.3
uvicorn==0.34.2
uvloop==0.21.0
validators==0.35.0
vine==5.1.0
virtualenv==20.28.0
watchfiles==1.0.3
wcwidth==0.2.13
WebOb==1.8.9
websockets==14.1
zipp==3.21.0

(.venv) Williams-MacBook-Air:core-api williamdavidedwards$ celery -A app.repositories.celery.infrastructure.worker_entrypoint.celery_app report
/Users/williamdavidedwards/PycharmProjects/core-api/.venv/lib/python3.11/site-packages/pydantic_settings/sources/providers/secrets.py:60: UserWarning: directory "/etc/cluster-api" does not exist
  warnings.warn(f'directory "{path}" does not exist')

software -> celery:5.5.2 (immunity) kombu:5.5.3 py:3.11.9
            billiard:4.2.1 redis:5.3.0
platform -> system:Darwin arch:64bit
            kernel version:24.4.0 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://:**@127.0.0.1:1522/14

result_expires: None
task_track_started: True
result_extended: True
broker_url: 'redis://:********@127.0.0.1:1522/15'
result_backend: 'redis://:********@127.0.0.1:1522/14'
deprecated_settings: None

auvipy · 2025-05-08T08:47:11Z

auvipy
May 8, 2025
Maintainer

we will investigate in next big release. until that please stick with the supported version

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

`UNSUBSCRIBE` hangs with redis-py 5.3.0 #9691

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Uh oh!

UNSUBSCRIBE hangs with redis-py 5.3.0 #9691

Uh oh!

Uh oh!

WilliamDEdwards May 7, 2025

Backtrace

Additional information

Replies: 1 comment

Uh oh!

auvipy May 8, 2025 Maintainer

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

`UNSUBSCRIBE` hangs with redis-py 5.3.0 #9691

WilliamDEdwards
May 7, 2025

auvipy
May 8, 2025
Maintainer