Content-Length: 214933 | pFad | https://github.com/scrapy/scrapy/issues/3845

FB S3FeedStorage should use async IO (like txaws) · Issue #3845 · scrapy/scrapy · GitHub
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3FeedStorage should use async IO (like txaws) #3845

Open
diemacht opened this issue Jun 27, 2019 · 1 comment
Open

S3FeedStorage should use async IO (like txaws) #3845

diemacht opened this issue Jun 27, 2019 · 1 comment

Comments

@diemacht
Copy link

Currently, S3FeedStorage uses io blocking boto library through deferToThread that converts it to deferred.
However, this is not a real non-blocking execution, as the blocking code runs in a separate python thread. Overhead of thread is still significant. Taking into consideration there might be much more than 1 such thread. More about disadvantages of python threads: http://alexeyvishnevsky.com/2013/10/why-you-should-consider-asynchronous-programming-model-apm-when-writing-web-server-in-python/

Instead, S3FeedStorage should use a real non-blocking AWS library, such as txaws, for example.

@Gallaecio
Copy link
Member

https://github.com/twisted/txaws seems dead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: https://github.com/scrapy/scrapy/issues/3845

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy