Expand description
robots.txt middleware.
RobotsTxtMiddleware fetches and caches robots.txt per origin and blocks
requests that are disallowed for the outgoing user agent.
Structsยง
- Robots
TxtMiddleware - Middleware that enforces
robots.txtrules before download.