How to set up the robots protocol in Crawlee for Python? #1059

Closed

Closed

How to set up the robots protocol in Crawlee for Python?#1059

Assignees

Labels

opened

When using the Python version of Crawlee, I didn't see any options to configure the robots protocol, but I found relevant information in the Node.js version's documentation. If such a configuration exists, please help me.

Metadata

Assignees

Mantisus

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.

Alternative Proxies:

Alternative Proxy