-
Notifications
You must be signed in to change notification settings - Fork 554
Integrate async transport with SDK #4615
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: srothh/async-transport
Are you sure you want to change the base?
Integrate async transport with SDK #4615
Conversation
…ted a sync transport HTTP subclass Moved shared sync/async logic into a new superclass (HttpTransportCore), and moved sync transport specific code into a new subclass(BaseSyncHttpTransport), from which the current transport implementations inherit Fixes GH-4568
Removed an unnecessary TODO message and reverted a class name change for BaseHTTPTransport. GH-4568
Adds test coverage for the error handling path when HTTP requests return error status codes. GH-4568
Restore comments accidentally removed during a previous commit.
Refactored class names such that BaseHttpTransport now has the same functionality as before the hierarchy refactor GH-4568
Add a new flush_async method in the Transport ABC. This is needed for the async transport, as calling it from the client while preserving execution order in close will require flush to be a coroutine, not a function. GH-4568
Move flush_async down to the specific async transport subclass. This makes more sense anyway, as this will only be required by the async transport. If more async transports are expected, another shared superclass can be created. GH-4568
Add necessary type annotations to the core HttpTransport to accomodate for async transport. GH-4568
Add an abstract bass class for implementation of the background worker. This was done to provide a shared interface for the current implementation of a threaded worker in the sync context as well as the upcoming async task-based worker implementation. GH-4578
Add a new factory method instead of direct instatiation of the threaded background worker. This allows for easy extension to other types of workers, such as the upcoming task-based async worker. GH-4578
Add a new flush_async method to worker ABC. This is necessary because the async transport cannot use a synchronous blocking flush. GH-4578
Move the flush_async down to the concrete subclass to not break existing testing. This makes sense, as this will only really be needed by the async worker anyway and therefore is not shared logic. GH-4578
Coroutines have a return value, however the current function signature for the worker methods does not accomodate for this. Therefore, this signature was changed. GH-4578
Add a new implementation of the worker interface, implementing the worker as an async task. This is to be used by the upcoming async transport. GH-4581
Refactor the flush method in the async worker to use the async_flush coroutine. GH-4581
…unctions Add a check to see wheter callbacks are awaitable coroutines or functions, as coroutines need to be awaited. GH-4581
…coroutines Coroutines do not return None, therefore it is necessary to consider this in the callback parameter of the worker. Previously, only callbacks with return Type None were accepted. GH-4581
Enable concurrent callbacks on async task worker by firing them as a task rather than awaiting them. A done callback handles the necessary queue and exception logic. GH-4581
Changed kill to also use the _TERMINATOR sentinel, so the queue is still drained to this point on kill instead of cancelled immediately. This should also fix potential race conditions with flush_async. GH-4581
Add proper type annotation to worker task list to fix linting problems GH-4581
Add an implementation of Transport to work with the async background worker and HTTPCore async. GH-4582
Async Transport now properly checks for the presence of the event loop in capture_envelop, and drops items in case the event loop is no longer running for some reason. GH-4582
Implement a kill method that properly shuts down the async transport. The httpcore async connection pool needs to be explicitly shutdown at the end of its usage. GH-4582
Fix type errors resulting from async override and missing type definition in the async transport. GH-4582
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## srothh/async-transport #4615 +/- ##
==========================================================
+ Coverage 83.42% 84.69% +1.26%
==========================================================
Files 158 158
Lines 15654 15729 +75
Branches 2472 2484 +12
==========================================================
+ Hits 13060 13321 +261
+ Misses 1842 1628 -214
- Partials 752 780 +28
|
Add a try/catch to ensure silent fail on kill in case the event loop shuts down. GH-4582
e5ea8e9
to
57ea658
Compare
Make kill optionally return a task for async transport. This allows for a blocking kill operation if the caller is in an async context. GH-4582
fca8740
to
328d8ad
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some questsions
I am not sure why CI Lint is failing. The issue is in a completely unrelated file that the PR did not change |
295a0e9
to
6cb72ad
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bug: Async Context Manager Exit Issue
The Client.__exit__
method calls self.close()
but does not await the asyncio.Task
returned by close()
when AsyncHttpTransport
is used. Since __exit__
is synchronous, this prevents async flush and cleanup operations from completing, leading to potential data loss or resource leaks when the client is used as a context manager.
sentry_sdk/client.py#L1043-L1045
sentry-python/sentry_sdk/client.py
Lines 1043 to 1045 in 9f226cf
def __exit__(self, exc_type: Any, exc_value: Any, tb: Any) -> None: | |
self.close() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks great now.
ignore the linting problem, we need to fix this in master and potel-base branches.
Integrate the async transport with the rest of the SDK. Provide a new experimental option
transport_async
that enables the async transport if an event loop is running. Otherwise, fall back to the sync transport.Furthermore, adapt the client to work with the async transport. To this end, flush and close were changed to be non blocking and awaitable in an async context to avoid deadlocks, however close enforces a completed flush before shutdown. As there are to my knowledge no background threads running flush/close, these methods are currently not thread-safe/loop-aware for async, which can be changed if necessary.
Atexit issue: The atexit integration used by the SDK runs after the event loop has already closed if asyncio.run() is used. This makes it impossible for the async flush to happen, as atexit calls client.close(), but a loop is no longer present. I attempted to apply this fix by patching the loop close in the asyncio integration, but I am unsure if I did it correctly/put it in the correct spot, or if this is a good idea. From my SDK test however, it seems to fix the flush issue. Note also that this will apparently be patched in Python 3.14, as per the discussion in the linked thread.
As a final note, I added event loop checking. Whenever the event loop is used, the transport/client catch RuntimeErrors, which would be thrown in case the event loop was already shut down. I am not sure if this is a case we need to consider, but I added it for now because I did not want the transport to potentially throw RuntimeError if the event loop is shutdown during a program. If this should be left out currently for simplicity, I can remove it again.
I added the httpcore[asyncio] dependency to requirements-testing, as it is needed for the async httpcore functionality.
GH-4601