Struct StreamJsonPipeline 

pub struct StreamJsonPipeline<I>
where
    I: ScrapedItem,
{ /* private fields */ }

Streaming JSON output pipeline. A pipeline that streams items directly to a JSON file without accumulating them in memory.

Implementations

impl<I> StreamJsonPipeline<I>

where I: ScrapedItem,

pub fn new( file_path: impl AsRef<Path>, ) -> Result<StreamJsonPipeline<I>, PipelineError>

Creates a new StreamJsonPipeline with default batch size.

Errors

Returns an error when the output directory cannot be created.

pub fn with_batch_size( file_path: impl AsRef<Path>, batch_size: usize, ) -> Result<StreamJsonPipeline<I>, PipelineError>

Creates a new StreamJsonPipeline with a specified batch size.

Errors

Returns an error when the output directory cannot be created.

pub fn with_schema_export_config( self, config: SchemaExportConfig, ) -> StreamJsonPipeline<I>

Applies typed export mapping before values are written.

Trait Implementations

impl<I> Pipeline<I> for StreamJsonPipeline<I>

where I: ScrapedItem,

fn name(&self) -> &str

Returns the name of the pipeline.

fn process_item<'life0, 'async_trait>( &'life0 self, item: I, ) -> Pin<Box<dyn Future<Output = Result<Option<I>, PipelineError>> + Send + 'async_trait>>

where 'life0: 'async_trait, StreamJsonPipeline<I>: 'async_trait,

Processes a single scraped item.

fn close<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<(), PipelineError>> + Send + 'async_trait>>

where 'life0: 'async_trait, StreamJsonPipeline<I>: 'async_trait,

Called when the spider is closing.

fn get_state<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Option<Value>, PipelineError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Returns the current state of the pipeline as a JSON value.

fn restore_state<'life0, 'async_trait>( &'life0 self, _state: Value, ) -> Pin<Box<dyn Future<Output = Result<(), PipelineError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Restores the pipeline’s state from a JSON value.