Scrape Amazon product listings utilising scrapy & residential proxies

To get started with Scrapy you will first need to install it using methods provided in their documentation. Check here for more information

Once you have an active subscription you can find your credentials & proxy addresses in Dashboard > Residential > Proxy Setup

Navigate to settings.py in /amazon/amazon/ folder and modify the following lines to authenticate.

SMARTPROXY_USER = 'SPusername' ## Smartproxy Username (Sub-user)
SMARTPROXY_PASSWORD = 'SPpassword' ## Password for your user
SMARTPROXY_ENDPOINT = 'gate.smartproxy.com' ## Endpoint you'd like to use
SMARTPROXY_PORT = '7000' ## Port of the endpoint you are using.

Navigate to the project folder and run the following command

scrapy crawl amazon_search

Amazon search results will be saved in /amazon/data folder in a .csv format