pub struct Response {
pub url: Url,
pub status: StatusCode,
pub headers: HeaderMap,
pub body: Bytes,
pub request_url: Url,
pub request_priority: i32,
pub meta: Option<Arc<DashMap<String, Value>>>,
pub cached: bool,
}Expand description
Shared runtime data types and convenience helpers. Represents an HTTP response received from a server.
Response contains all information about an HTTP response, including
the final URL (after redirects), status code, headers, body content,
and metadata carried over from the original request.
The type is designed for parse-time ergonomics:
Response::cssexposes the recommended Scrapy-like selector APIResponse::jsondeserializes JSON payloadsResponse::linksand related helpers extract follow-up links- [
Response::to_request] reconstructs the originating request context
§Example
use spider_util::response::Response;
use reqwest::StatusCode;
use bytes::Bytes;
use url::Url;
let response = Response {
url: Url::parse("https://example.com").unwrap(),
status: StatusCode::OK,
headers: http::header::HeaderMap::new(),
body: Bytes::from("<html><body>Hello</body></html>"),
request_url: Url::parse("https://example.com").unwrap(),
meta: None,
cached: false,
};
// Extract text using the builtin selector API
let title = response.css("title::text").ok().and_then(|list| list.get());Fields§
§url: UrlThe final URL of the response after any redirects.
status: StatusCodeThe HTTP status code of the response.
headers: HeaderMapThe headers of the response.
body: BytesThe body of the response.
request_url: UrlThe original URL of the request that led to this response.
request_priority: i32The scheduling priority of the original request.
meta: Option<Arc<DashMap<String, Value>>>Metadata associated with the response, carried over from the request. Uses Option to allow lazy initialization.
cached: boolIndicates if the response was served from a cache.
Implementations§
Source§impl Response
impl Response
Sourcepub fn new(
url: Url,
status: StatusCode,
headers: HeaderMap,
body: Bytes,
request_url: Url,
) -> Response
pub fn new( url: Url, status: StatusCode, headers: HeaderMap, body: Bytes, request_url: Url, ) -> Response
Creates a new response with an empty HTML cache.
Most application code receives responses from the runtime rather than constructing them directly. This constructor is mainly useful for custom downloaders and lower-level integrations.
Sourcepub fn request_from_response(&self) -> Request
pub fn request_from_response(&self) -> Request
Reconstructs the original Request that led to this response.
This method creates a new Request with the same URL and metadata
as the request that produced this response. Useful for retry scenarios
or when you need to re-request the same resource.
§Example
let original_request = response.request_from_response();Sourcepub fn meta_value<T>(&self, key: &str) -> Result<Option<T>, Error>where
T: DeserializeOwned,
pub fn meta_value<T>(&self, key: &str) -> Result<Option<T>, Error>where
T: DeserializeOwned,
Deserializes a metadata value into the requested type.
Sourcepub fn discovery_rule_name(&self) -> Option<String>
pub fn discovery_rule_name(&self) -> Option<String>
Returns the runtime discovery rule name attached to this response, if any.
Sourcepub fn matches_discovery_rule(&self, rule_name: &str) -> bool
pub fn matches_discovery_rule(&self, rule_name: &str) -> bool
Returns true when the response was reached through the named discovery rule.
Sourcepub fn insert_meta(&mut self, key: impl Into<String>, value: Value)
pub fn insert_meta(&mut self, key: impl Into<String>, value: Value)
Inserts a metadata value, lazily allocating the map if needed.
Sourcepub fn clone_meta(&self) -> Option<Arc<DashMap<String, Value>>>
pub fn clone_meta(&self) -> Option<Arc<DashMap<String, Value>>>
Returns a clone of the internal metadata map, if present.
Sourcepub fn json<T>(&self) -> Result<T, Error>where
T: DeserializeOwned,
pub fn json<T>(&self) -> Result<T, Error>where
T: DeserializeOwned,
Deserializes the response body as JSON.
§Type Parameters
T: The target type to deserialize into (must implementDeserializeOwned)
§Errors
Returns a serde_json::Error if the body cannot be parsed as JSON
or if it cannot be deserialized into type T.
§Example
let data: Data = response.json()?;Sourcepub fn css(&self, query: &str) -> Result<SelectorList, SpiderError>
pub fn css(&self, query: &str) -> Result<SelectorList, SpiderError>
Applies a builtin CSS selector to the response body using a Scrapy-like API.
Supports standard CSS selectors plus terminal extraction suffixes:
::text::attr(name)
§Example
let heading = response.css("h1::text")?.get().unwrap_or_default();
let next_href = response.css("a::attr(href)")?.get();§Errors
Returns SpiderError::Utf8Error when the body is not valid UTF-8 and
SpiderError::HtmlParseError when the selector is invalid.
Sourcepub fn page_metadata(&self) -> Result<PageMetadata, Utf8Error>
pub fn page_metadata(&self) -> Result<PageMetadata, Utf8Error>
Extracts structured page metadata from HTML responses.
Sourcepub fn links_iter(
&self,
options: LinkExtractOptions,
) -> impl Iterator<Item = Link>
pub fn links_iter( &self, options: LinkExtractOptions, ) -> impl Iterator<Item = Link>
Returns a customizable iterator of links discovered in the response body.
Unlike Response::links, this method does not deduplicate results.
Callers that need uniqueness can collect into a set or use Response::links.
§Example
let links: Vec<_> = response
.links_iter(LinkExtractOptions::default())
.collect();
assert!(!links.is_empty());Sourcepub fn links(&self) -> DashSet<Link>
pub fn links(&self) -> DashSet<Link>
Extracts all unique, same-site links from the response body.
This method discovers links from:
- HTML elements with
hreforsrcattributes (<a>,<link>,<script>,<img>, etc.) - URLs found in text content (using link detection)
Only links pointing to the same site (same registered domain) are included.
§Returns
A [DashSet] of Link objects containing the URL and link type.
§Example
let links = response.links();
for link in links.iter() {
println!("Found {:?} link: {}", link.link_type, link.url);
}