Struct RobotsTxtMiddleware 

pub struct RobotsTxtMiddleware { /* private fields */ }

robots.txt enforcement middleware. Middleware that enforces robots.txt rules before download.

Implementations

impl RobotsTxtMiddleware

pub fn new() -> RobotsTxtMiddleware

Creates a new RobotsTxtMiddleware with default settings.

pub fn cache_ttl(self, cache_ttl: Duration) -> RobotsTxtMiddleware

Set the time-to-live for the cache.

pub fn cache_capacity(self, cache_capacity: u64) -> RobotsTxtMiddleware

Set the max capacity for the cache.

pub fn request_timeout(self, request_timeout: Duration) -> RobotsTxtMiddleware

Set the timeout for fetching robots.txt files.

Trait Implementations

impl Debug for RobotsTxtMiddleware

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for RobotsTxtMiddleware

fn default() -> RobotsTxtMiddleware

Returns the “default value” for a type.

impl<C> Middleware<C> for RobotsTxtMiddleware

where C: HttpClient,

fn name(&self) -> &str

Returns a human-readable middleware name for logs and diagnostics.

fn process_request<'life0, 'life1, 'async_trait>( &'life0 self, client: &'life1 C, request: Request, ) -> Pin<Box<dyn Future<Output = Result<MiddlewareAction<Request>, SpiderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, RobotsTxtMiddleware: 'async_trait,

Intercepts an outgoing request before the downloader runs.

fn process_response<'life0, 'async_trait>( &'life0 self, response: Response, ) -> Pin<Box<dyn Future<Output = Result<MiddlewareAction<Response>, SpiderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Intercepts a successful response after download.

fn handle_error<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, _request: &'life1 Request, error: &'life2 SpiderError, ) -> Pin<Box<dyn Future<Output = Result<MiddlewareAction<Request>, SpiderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, Self: 'async_trait,

Handles downloader errors for a request.