S3FeedStorage should use async IO (like txaws) #3845
Labels
Content-Length: 214933 | pFad | https://github.com/scrapy/scrapy/issues/3845
FBFetched URL: https://github.com/scrapy/scrapy/issues/3845
Alternative Proxies:
Currently, S3FeedStorage uses io blocking boto library through deferToThread that converts it to deferred.
However, this is not a real non-blocking execution, as the blocking code runs in a separate python thread. Overhead of thread is still significant. Taking into consideration there might be much more than 1 such thread. More about disadvantages of python threads: http://alexeyvishnevsky.com/2013/10/why-you-should-consider-asynchronous-programming-model-apm-when-writing-web-server-in-python/
Instead, S3FeedStorage should use a real non-blocking AWS library, such as txaws, for example.
The text was updated successfully, but these errors were encountered: