Module response

spider_util

Module response 

Source
Expand description

Response types and response-side helpers.

Response wraps the downloaded body together with the final URL, status, headers, and request metadata. It also provides convenience methods for Scrapy-like CSS extraction, parsing JSON, and extracting links.

§Example

ⓘ
use spider_util::response::Response;
use reqwest::StatusCode;
use bytes::Bytes;
use url::Url;

// Create a response (typically done internally by the downloader)
let response = Response {
    url: Url::parse("https://example.com").unwrap(),
    status: StatusCode::OK,
    headers: http::header::HeaderMap::new(),
    body: Bytes::from("<html><body>Hello</body></html>"),
    request_url: Url::parse("https://example.com").unwrap(),
    request_priority: 0,
    meta: None,
    cached: false,
};

// Extract text with the builtin selector API
let heading = response.css("h1::text").unwrap().get();

// Extract links from the response
let links = response.links();

In the crawler lifecycle, a Response is produced by the downloader, optionally rewritten by middleware, and then handed to Spider::parse.

Structs§

Link
A link discovered while extracting URLs from a response.
LinkExtractOptions
Options that control link extraction from a Response.
LinkSource
One selector/attribute pair used during link extraction.
PageMetadata
Structured page metadata extracted from an HTML response.
Response
Represents an HTTP response received from a server.

Enums§

LinkType
Classification for links discovered in a response.