Celery Release Notes

Celery v5.5.0 Beta 3 is now available for testing. Please help us test this version and report any issues.

Key Highlights

Soft Shutdown

The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited "warm shutdown" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting.

The soft shutdown is disabled by default, and can be enabled by setting the new configuration option worker_soft_shutdown_timeout. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option worker_enable_soft_shutdown_on_idle is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways. …

Celery v5.5.0 Beta 2 is now available for testing. Please help us test this version and report any issues.

Key Highlights

Pydantic Support

This release introduces support for Pydantic models in Celery tasks. For more info, see the new pydantic example and PR #9023 by @mathiasertl.

After upgrading to this version, please share your feedback on the new Pydantic support.

Previous Beta Highlights

Redis Broker Stability Improvements

The root cause of the Redis broker instability issue has been identified and resolved in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements.

After upgrading to this version, please share your feedback on the Redis broker stability.

Relevant Issues: #7276, #8091, #8030, #8384

Quorum Queues Initial Support

This release introduces the initial support for Quorum Queues with Celery.

See new configuration options for more details:

• task_default_queue_type
• worker_detect_quorum_queues …

Celery v5.5.0 Beta 1 is now available for testing. Please help us test this version and report any issues.

Key Highlights

Redis Broker Stability Improvements

The root cause of the Redis broker instability issue has been identified and resolved in the release-candidate for Kombu v5.4.0. This beta release has been upgraded to use the new Kombu RC version, which should resolve the disconnections bug and offer additional improvements.

After upgrading to this version, please share your feedback on the Redis broker stability.

Relevant Issues: #7276, #8091, #8030, #8384

Quorum Queues Initial Support

This release introduces the initial support for Quorum Queues with Celery.

See new configuration options for more details:

• task_default_queue_type
• worker_detect_quorum_queues

After upgrading to this version, please share your feedback on the Quorum Queues support.

Relevant Issues: #6067, #9121

What's Changed

• (docs): use correct version celery v.5.4.x by @FraCata00 in https://github.com/celery/celery/pull/8975
• Update mypy to 1.10.0 by @pyup-bot in https://github.com/celery/celery/pull/8977 …

Celery v5.4.0 and v5.3.x have consistently focused on enhancing the overall QA, both internally and externally. This effort led to the new pytest-celery v1.0.0 release, developed concurrently with v5.3.0 & v5.4.0.

This release introduces two significant QA enhancements:

• Smoke Tests: A new layer of automatic tests has been added to Celery's standard CI. These tests are designed to handle production scenarios and complex conditions efficiently. While new contributions will not be halted due to the lack of smoke tests, we will request smoke tests for advanced changes where appropriate.
• Standalone Bug Report Script: The new pytest-celery plugin now allows for encapsulating a complete Celery dockerized setup within a single pytest script. Incorporating these into new bug reports will enable us to reproduce reported bugs deterministically, potentially speeding up the resolution process.

Contrary to the positive developments above, there have been numerous reports about issues with the Redis broker malfunctioning upon restarts and disconnections. Our initial attempts to resolve this were not successful (#8796). With our enhanced QA capabilities, we are now prepared to address the core issue with Redis (as a broker) again. …

What's Changed

• feat(daemon): allows daemonization options to be fetched from app settings by @noirbizarre in https://github.com/celery/celery/pull/8553
• Fixed version documentation tag from #8553 in configuration.rst by @Nusnus in https://github.com/celery/celery/pull/8802
• Upgraded Sphinx from v5.3.0 to v7.x.x by @Nusnus in https://github.com/celery/celery/pull/8803
• Update elasticsearch requirement from <=8.11.1 to <=8.12.0 by @dependabot in https://github.com/celery/celery/pull/8810
• Update elastic-transport requirement from <=8.11.0 to <=8.12.0 by @dependabot in https://github.com/celery/celery/pull/8811
• Update cryptography to 42.0.0 by @pyup-bot in https://github.com/celery/celery/pull/8814
• Catch UnicodeDecodeError when opening corrupt beat-schedule.db by @andyzickler in https://github.com/celery/celery/pull/8806
• Update cryptography to 42.0.1 by @pyup-bot in https://github.com/celery/celery/pull/8817
• Limit moto to <5.0.0 until the breaking issues are fixed by @Nusnus in https://github.com/celery/celery/pull/8820
• Enable efficient chord when using dynamicdb as backend store by @dingxiong in https://github.com/celery/celery/pull/8783
• Add a Task class specialised for Django by @browniebroke in https://github.com/celery/celery/pull/8491
• Sync kombu versions in requirements and setup.cfg by @Watkurem in https://github.com/celery/celery/pull/8825 …

Celery v5.4 continues our effort to provide improved stability in production environments. The release candidate version is available for testing. The official release is planned for March-April 2024.

Please share feedback in the discussion below.

What's Changed

• New Config: worker_enable_prefetch_count_reduction by @Nusnus in https://github.com/celery/celery/pull/8581
• Added "Serverless" section to Redis doc (redis.rst) by @Nusnus in https://github.com/celery/celery/pull/8640
• Upstash's Celery example repo link fix due to deprecation of the prev… by @Nusnus in https://github.com/celery/celery/pull/8665
• Update mypy version by @auvipy in https://github.com/celery/celery/pull/8679
• Update cryptography dependency to 41.0.7 by @danyi1212 in https://github.com/celery/celery/pull/8690
• Add type annotations to celery/utils/nodenames.py by @lyzlisa in https://github.com/celery/celery/pull/8667
• Issue 3426. Adding myself to the contributors. by @Mulugruntz in https://github.com/celery/celery/pull/8696
• Bump actions/setup-python from 4 to 5 by @dependabot in https://github.com/celery/celery/pull/8701
• Fixed bug where chord.link_error() throws an exception on a dict type errback object by @Nusnus in https://github.com/celery/celery/pull/8702
• Bump github/codeql-action from 2 to 3 by @dependabot in https://github.com/celery/celery/pull/8725 …

This release is focused mainly to fix AWS SQS new feature compatibility issue and old regressions. The code changes are mostly fix for regressions. More details can be found below.

What's Changed

• Update Minor-Version-Release-Checklist.md by @auvipy in https://github.com/celery/celery/pull/8624
• Increased docker-build CI job timeout from 30m -> 60m by @Nusnus in https://github.com/celery/celery/pull/8635
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/celery/celery/pull/8639
• Incredibly minor spelling fix. by @Asday in https://github.com/celery/celery/pull/8649
• Fix non-zero exit code when receiving remote shutdown by @lyzlisa in https://github.com/celery/celery/pull/8650
• Update task.py get_custom_headers missing 'compression' key by @auvipy in https://github.com/celery/celery/pull/8633
• Update kombu>=5.3.4 to fix SQS request compatibility by @auvipy in https://github.com/celery/celery/pull/8646
• test requirements version update by @auvipy in https://github.com/celery/celery/pull/8655
• Update elasticsearch version by @auvipy in https://github.com/celery/celery/pull/8656
• Propagates more ImportErrors during autodiscovery by @johnjameswhitman in https://github.com/celery/celery/pull/8632
• Re-raise ModuleNotFoundError unless for guessed task by @johnjameswhitman in https://github.com/celery/celery/pull/8660
• Added changelog for v5.3.6 release by @auvipy in https://github.com/celery/celery/pull/8659 …

Main theme of this release is adding Python 3.12 compatibility support all through the projects dependencies. Also lots of bugs were squashed. Dependencies upgraded and docs improved.

What's Changed

• Update test.txt versions by @auvipy in https://github.com/celery/celery/pull/8481
• fix os.getcwd() FileNotFoundError by @mortimer2015 in https://github.com/celery/celery/pull/8448
• Fix typo in CONTRIBUTING.rst by @monteiro-renato in https://github.com/celery/celery/pull/8494
• typo(doc): configuration.rst by @shifenhutu in https://github.com/celery/celery/pull/8484
• assert before raise by @monteiro-renato in https://github.com/celery/celery/pull/8495
• Update GHA checkout version by @auvipy in https://github.com/celery/celery/pull/8496
• Fixed replaced_task_nesting by @Nusnus in https://github.com/celery/celery/pull/8500
• Fix code indentation for route_task() example by @stefmolin in https://github.com/celery/celery/pull/8502
• support redis 5.x by @dulmandakh in https://github.com/celery/celery/pull/8504
• Fix typos in test_canvas.py by @monteiro-renato in https://github.com/celery/celery/pull/8498
• Marked flaky tests by @Nusnus in https://github.com/celery/celery/pull/8508
• Fix typos in calling.rst by @visitorckw in https://github.com/celery/celery/pull/8506
• Added support for replaced_task_nesting in chains by @Nusnus in https://github.com/celery/celery/pull/8501 …

Notice Regarding v5.3.2 and v5.3.3

Versions 5.3.2 and 5.3.3 were identified to contain breaking changes and have been yanked to maintain codebase stability. We appreciate the community's timely feedback on these versions and apologize for any inconvenience caused.

What's Changed

• Bugfix: Removed unecessary stamping code from _chord.run() by @Nusnus in https://github.com/celery/celery/pull/8339
• User guide fix (hotfix for #1755) by @Nusnus in https://github.com/celery/celery/pull/8342
• store children with database backend by @aaronst in https://github.com/celery/celery/pull/8338
• Stamping bugfix with group/chord header errback linking by @Nusnus in https://github.com/celery/celery/pull/8347
• Use argsrepr and kwargsrepr in LOG_RECEIVED by @zhu in https://github.com/celery/celery/pull/8301
• Fixing minor typo in code example in calling.rst by @dwysocki in https://github.com/celery/celery/pull/8366
• add documents for timeout settings by @ooyamatakehisa in https://github.com/celery/celery/pull/8373
• fix: copyright year by @karanganesan in https://github.com/celery/celery/pull/8380
• setup.py: enable include_package_data by @elohmeier in https://github.com/celery/celery/pull/8379
• Fix eager tasks does not populate name field by @KOliver94 in https://github.com/celery/celery/pull/8383
• Update test.txt dependencies by @auvipy in https://github.com/celery/celery/pull/8389
• Update auth.txt deps by @auvipy in https://github.com/celery/celery/pull/8392 …