Struct JsonPipeline 

pub struct JsonPipeline<I>
where
    I: ScrapedItem,
{ /* private fields */ }

JSON array output pipeline. A pipeline that writes all scraped items to a single JSON file as a JSON array. Items are collected in a blocking task and written to disk when the pipeline is closed.

Implementations

impl<I> JsonPipeline<I>

where I: ScrapedItem,

pub fn new( file_path: impl AsRef<Path>, ) -> Result<JsonPipeline<I>, PipelineError>

Creates a new JsonPipeline.

Errors

Returns an error when the output directory cannot be created.

pub fn with_schema_export_config( self, config: SchemaExportConfig, ) -> JsonPipeline<I>

Applies typed export mapping before values are written.

Trait Implementations

impl<I> Pipeline<I> for JsonPipeline<I>

where I: ScrapedItem,

fn name(&self) -> &str

Returns the name of the pipeline.

fn process_item<'life0, 'async_trait>( &'life0 self, item: I, ) -> Pin<Box<dyn Future<Output = Result<Option<I>, PipelineError>> + Send + 'async_trait>>

where 'life0: 'async_trait, JsonPipeline<I>: 'async_trait,

Processes a single scraped item.

fn close<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<(), PipelineError>> + Send + 'async_trait>>

where 'life0: 'async_trait, JsonPipeline<I>: 'async_trait,

Called when the spider is closing.

fn get_state<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Option<Value>, PipelineError>> + Send + 'async_trait>>

where 'life0: 'async_trait, JsonPipeline<I>: 'async_trait,

Returns the current state of the pipeline as a JSON value.

fn restore_state<'life0, 'async_trait>( &'life0 self, state: Value, ) -> Pin<Box<dyn Future<Output = Result<(), PipelineError>> + Send + 'async_trait>>

where 'life0: 'async_trait, JsonPipeline<I>: 'async_trait,

Restores the pipeline’s state from a JSON value.