Skip to content

fix: log configuring when running under spawn and add --log-format #435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 25, 2025

Conversation

ivolnistov
Copy link
Contributor

@ivolnistov ivolnistov commented Mar 25, 2025

Technical explanation of the issue:

  1. When using 'fork', a copy of the current process is created, including memory and state of all objects. This means logging settings are preserved.
  2. When using 'spawn', a new process is created which imports modules fresh. This means all global objects, including loggers, are initialized again.

Problem in taskiq code:

  1. Logging configuration happens in the parent process, but these settings are not passed to child processes when using 'spawn'.
  2. DEBUG logs are not visible because by default loggers in child processes have WARNING or INFO level.

PR fixes:

  1. This PR fixes the logging issue when using the 'spawn' method in multiprocessing
  2. Adds log-format support, which allows more flexible configuration of logger behavior when using spawn

Now DEBUG level logs will be correctly displayed in all child processes, even when using the 'spawn' method, and log formatting settings can be easily customized.

Fixed issue with logging configuration when using multiprocessing 'spawn' method on macOS.
The spawn method reimports all modules which caused logging configuration to be lost.
Added `--log-format` argument to easily customize logging format across all processes,
including spawned child processes.
s3rius
s3rius previously approved these changes Mar 25, 2025
@s3rius
Copy link
Member

s3rius commented Mar 25, 2025

Can you please run pre-commit run -a?

@ivolnistov
Copy link
Contributor Author

Can you please run pre-commit run -a?

sure

@s3rius
Copy link
Member

s3rius commented Mar 25, 2025

Thanks for the pull request. It makes a lot of sense, even though I haven't thought of it.

@s3rius s3rius merged commit bdf446f into taskiq-python:master Mar 25, 2025
33 checks passed
s3rius pushed a commit that referenced this pull request Apr 4, 2025
)

* fix: log configuring when running under spawn and add --log-format

Fixed issue with logging configuration when using multiprocessing 'spawn' method on macOS.
The spawn method reimports all modules which caused logging configuration to be lost.
Added `--log-format` argument to easily customize logging format across all processes,
including spawned child processes.

* fix linting and formatting

---------

Co-authored-by: Ilya Volnistov <i.volnistov@gaijin.team>
s3rius pushed a commit that referenced this pull request Apr 4, 2025
)

* fix: log configuring when running under spawn and add --log-format

Fixed issue with logging configuration when using multiprocessing 'spawn' method on macOS.
The spawn method reimports all modules which caused logging configuration to be lost.
Added `--log-format` argument to easily customize logging format across all processes,
including spawned child processes.

* fix linting and formatting

---------

Co-authored-by: Ilya Volnistov <i.volnistov@gaijin.team>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy