Struct ParseContext 

pub struct ParseContext<'a, S>
where
    S: Spider + ?Sized,
{ /* private fields */ }

Core runtime types and traits used to define and run a crawl. Parse-time context passed into Spider::parse.

This bundles the current Response, shared spider state, and the async output sink into a single value so user-facing parse signatures stay small.

The context dereferences to Response, which means selector-heavy code can keep the natural cx.css(...) style without manually reaching through a nested response field.

Implementations

impl<'a, S> ParseContext<'a, S>

where S: Spider + ?Sized,

pub fn state(&self) -> &'a <S as Spider>::State

Returns the shared spider state for this parse call.

pub fn response(&self) -> &Response

Returns the current response explicitly.

pub fn response_mut(&mut self) -> &mut Response

Returns the current response as a mutable reference.

pub fn output(&self) -> &ParseOutput<<S as Spider>::Item>

Returns the underlying async parse output sink.

pub async fn add_item( &self, item: <S as Spider>::Item, ) -> Result<(), SpiderError>

Emits a scraped item into the runtime.

pub async fn add_items( &self, items: impl IntoIterator<Item = <S as Spider>::Item>, ) -> Result<(), SpiderError>

Emits multiple scraped items into the runtime.

pub async fn add_request(&self, request: Request) -> Result<(), SpiderError>

Emits a follow-up request into the runtime.

pub async fn add_requests( &self, requests: impl IntoIterator<Item = Request>, ) -> Result<(), SpiderError>

Emits multiple follow-up requests into the runtime.

pub fn into_parts( self, ) -> (Response, &'a <S as Spider>::State, ParseOutput<<S as Spider>::Item>)

Consumes the context and returns the inner response, state reference, and output sink.

pub fn into_response(self) -> Response

Consumes the context and returns the inner response.

Methods from Deref<Target = Response>

pub fn request_from_response(&self) -> Request

Reconstructs the original Request that led to this response.

This method creates a new Request with the same URL and metadata as the request that produced this response. Useful for retry scenarios or when you need to re-request the same resource.

Example

let original_request = response.request_from_response();

pub fn get_meta(&self, key: &str) -> Option<Value>

Returns a cloned metadata value by key.

pub fn meta_value<T>(&self, key: &str) -> Result<Option<T>, Error>

where T: DeserializeOwned,

Deserializes a metadata value into the requested type.

pub fn discovery_rule_name(&self) -> Option<String>

Returns the runtime discovery rule name attached to this response, if any.

pub fn matches_discovery_rule(&self, rule_name: &str) -> bool

Returns true when the response was reached through the named discovery rule.

pub fn insert_meta(&mut self, key: impl Into<String>, value: Value)

Inserts a metadata value, lazily allocating the map if needed.

pub fn clone_meta(&self) -> Option<Arc<DashMap<String, Value>>>

Returns a clone of the internal metadata map, if present.

pub fn json<T>(&self) -> Result<T, Error>

where T: DeserializeOwned,

Deserializes the response body as JSON.

Type Parameters

• T: The target type to deserialize into (must implement DeserializeOwned)

Errors

Returns a serde_json::Error if the body cannot be parsed as JSON or if it cannot be deserialized into type T.

Example

let data: Data = response.json()?;

pub fn css(&self, query: &str) -> Result<SelectorList, SpiderError>

Applies a builtin CSS selector to the response body using a Scrapy-like API.

Supports standard CSS selectors plus terminal extraction suffixes:

• ::text
• ::attr(name)

Example

let heading = response.css("h1::text")?.get().unwrap_or_default();
let next_href = response.css("a::attr(href)")?.get();

Errors

Returns SpiderError::Utf8Error when the body is not valid UTF-8 and SpiderError::HtmlParseError when the selector is invalid.

pub fn text(&self) -> Result<&str, Utf8Error>

Returns the response body as UTF-8 text.

pub fn page_metadata(&self) -> Result<PageMetadata, Utf8Error>

Extracts structured page metadata from HTML responses.

pub fn links_iter( &self, options: LinkExtractOptions, ) -> impl Iterator<Item = Link>

Returns a customizable iterator of links discovered in the response body.

Unlike Response::links, this method does not deduplicate results. Callers that need uniqueness can collect into a set or use Response::links.

Example

let links: Vec<_> = response
    .links_iter(LinkExtractOptions::default())
    .collect();
assert!(!links.is_empty());

pub fn links(&self) -> DashSet<Link>

Extracts all unique, same-site links from the response body.

This method discovers links from:

• HTML elements with href or src attributes (<a>, <link>, <script>, <img>, etc.)
• URLs found in text content (using link detection)

Only links pointing to the same site (same registered domain) are included.

Returns

A [DashSet] of Link objects containing the URL and link type.

Example

let links = response.links();
for link in links.iter() {
    println!("Found {:?} link: {}", link.link_type, link.url);
}

Trait Implementations

impl<S> Deref for ParseContext<'_, S>

where S: Spider + ?Sized,

type Target = Response

The resulting type after dereferencing.

fn deref(&self) -> &<ParseContext<'_, S> as Deref>::Target

Dereferences the value.

impl<S> DerefMut for ParseContext<'_, S>

where S: Spider + ?Sized,

fn deref_mut(&mut self) -> &mut <ParseContext<'_, S> as Deref>::Target

Mutably dereferences the value.