Struct VisitedUrls 

pub struct VisitedUrls { /* private fields */ }

Core runtime types and traits used to define and run a crawl. A thread-safe URL tracker using DashMap.

This provides efficient concurrent access for tracking visited URLs without requiring explicit locks.

Example

use spider_core::state::VisitedUrls;

let visited = VisitedUrls::new();
visited.mark("https://example.com".to_string());
assert!(visited.is_visited("https://example.com"));

Implementations

impl VisitedUrls

pub fn new() -> VisitedUrls

Creates a new empty URL tracker.

pub fn with_capacity(capacity: usize) -> VisitedUrls

Creates a URL tracker with the specified capacity.

pub fn mark(&self, url: impl Into<String>)

Marks a URL as visited.

pub fn is_visited(&self, url: &str) -> bool

Checks if a URL has been visited.

pub fn remove(&self, url: &str)

Removes a URL from the visited set.

pub fn len(&self) -> usize

Returns the number of visited URLs.

pub fn is_empty(&self) -> bool

Returns true if no URLs have been visited.

pub fn clear(&self)

Clears all visited URLs.

Trait Implementations

impl Clone for VisitedUrls

fn clone(&self) -> VisitedUrls

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for VisitedUrls

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for VisitedUrls

fn default() -> VisitedUrls

Returns the “default value” for a type.