Struct DiscoveryRule 

pub struct DiscoveryRule {
    pub name: String,
    pub allow_patterns: Vec<String>,
    pub deny_patterns: Vec<String>,
    pub allow_domains: Vec<String>,
    pub deny_domains: Vec<String>,
    pub allow_path_prefixes: Vec<String>,
    pub deny_path_prefixes: Vec<String>,
    pub link_extract_options: LinkExtractOptions,
}

Core runtime types and traits used to define and run a crawl. Rule-like configuration for runtime-managed discovery.

Fields

name: String

Stable rule name injected into request/response metadata when matched.

allow_patterns: Vec<String>

URL patterns that the source response must match.

deny_patterns: Vec<String>

URL patterns that exclude the source response.

allow_domains: Vec<String>

Domains or subdomains allowed for the source response.

deny_domains: Vec<String>

Domains or subdomains denied for the source response.

allow_path_prefixes: Vec<String>

Path prefixes allowed for the source response.

deny_path_prefixes: Vec<String>

Path prefixes denied for the source response.

link_extract_options: LinkExtractOptions

Link extraction behavior used when the rule matches.

Implementations

impl DiscoveryRule

pub fn new(name: impl Into<String>) -> DiscoveryRule

Creates a new discovery rule with the provided name.

pub fn with_link_extract_options( self, options: LinkExtractOptions, ) -> DiscoveryRule

Replaces the link extraction options used by this rule.

pub fn with_allow_patterns( self, patterns: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Restricts this rule to source response URLs that match at least one pattern.

pub fn with_deny_patterns( self, patterns: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Excludes this rule for source response URLs that match any pattern.

pub fn with_allow_domains( self, domains: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Restricts this rule to source response domains or subdomains.

pub fn with_deny_domains( self, domains: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Excludes this rule for source response domains or subdomains.

pub fn with_allow_path_prefixes( self, prefixes: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Restricts this rule to source response paths with one of the provided prefixes.

pub fn with_deny_path_prefixes( self, prefixes: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Excludes this rule for source response paths with one of the provided prefixes.

pub fn with_same_site_only(self, enabled: bool) -> DiscoveryRule

Sets whether only same-site links should be extracted for matching responses.

pub fn with_text_links(self, enabled: bool) -> DiscoveryRule

Sets whether text content should be scanned for plain-text URLs.

pub fn with_follow_allow_patterns( self, patterns: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Restricts discovered follow-up links to matching patterns.

pub fn with_follow_deny_patterns( self, patterns: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Excludes discovered follow-up links that match the given patterns.

pub fn with_follow_allow_domains( self, domains: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Restricts discovered follow-up links to the given domains or subdomains.

pub fn with_follow_deny_domains( self, domains: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Excludes discovered follow-up links for the given domains or subdomains.

pub fn with_follow_allow_path_prefixes( self, prefixes: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Restricts discovered follow-up links to the provided path prefixes.

pub fn with_follow_deny_path_prefixes( self, prefixes: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Excludes discovered follow-up links for the provided path prefixes.

pub fn with_allowed_tags( self, tags: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Restricts attribute extraction to specific HTML tags for matching responses.

pub fn with_allowed_attributes( self, attributes: impl IntoIterator<Item = impl Into<String>>, ) -> DiscoveryRule

Restricts attribute extraction to specific HTML attributes for matching responses.

pub fn with_allowed_link_types( self, link_types: impl IntoIterator<Item = LinkType>, ) -> DiscoveryRule

Restricts discovered follow-up links to the provided link types.

pub fn with_denied_link_types( self, link_types: impl IntoIterator<Item = LinkType>, ) -> DiscoveryRule

Excludes the provided link types from discovered follow-up links.

Trait Implementations

impl Clone for DiscoveryRule

fn clone(&self) -> DiscoveryRule

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DiscoveryRule

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl PartialEq for DiscoveryRule

fn eq(&self, other: &DiscoveryRule) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for DiscoveryRule

impl StructuralPartialEq for DiscoveryRule