Crate spider_util 

spider-util

Shared types and helper modules used across the spider-* workspace.

This crate is where request and response models, error types, selector helpers, formatting helpers, metrics helpers, and other common utilities live.

Example

use spider_util::request::Request;
use url::Url;

let url = Url::parse("https://example.com").unwrap();
let request = Request::new(url);
let _ = request;

Re-exports

pub use formatters::ByteFormatter;

pub use formatters::DefaultByteFormatter;

pub use formatters::DefaultDurationFormatter;

pub use formatters::DefaultRateCalculator;

pub use formatters::DurationFormatter;

pub use formatters::RateCalculator;

pub use formatters::format_bytes;

pub use formatters::format_duration;

pub use formatters::format_request_time;

pub use http_client::HttpClient;

pub use constants::*;

Modules

bloom

Bloom filter used by the scheduler for cheap duplicate checks.

constants

Workspace-wide public constants.

error

Shared error types for the workspace.

formatters

Formatting helpers for metrics and runtime output.

http_client

Small HTTP client abstraction used by middleware.

item

Item traits and parse-time output helpers.

metrics

Metrics helpers shared by runtime reporting code.

request

Request types used by the crawler runtime.

response

Response types and response-side helpers.

selector

Cached CSS selector helpers.

util

Small utility helpers shared across the workspace.