Adding Pipeline Steps in HQ

There are four places you can add Pipeline steps, shown in the following diagram:

• Post-Scan - steps here are executed immediately after Scanning is complete
• Pre-Extraction - steps here are executed just prior to Extraction
• Post-Extraction - steps here are executed immediately after Extraction is complete
• Pre-Index -steps here are executed just prior to the information being written to the Index

See Creating Pipelines for details on how to create and save Pipelines.

Pipeline Steps

You can add one or more of the following steps to different points in the Pipeline sequence:

• S3 Thumbnail Upload
  Uploads thumbnails to Amazon S3Calculate MD5 ChecksumCalculates an MD5 checksum of the source file content
• Index Debug Information
  Indexes raw debug information about the indexing job
• Index Children
  Indexes child entries
• Index Links
  Indexes information about linked entities
• Copy Field
  This transformer copies the contents of an existing index field (source) into another field (target). The source field is not modified.
• Rename Field
  Renames one field (Source Field) to another (Destination Field). The source field is then removed from the index. 
• Remove Field
  Removes a field from the Index
• Transform Field Value
  Transforms a field value to a different value
• Convert from 2.x to 1.x Index
  Converts documents from Voyager 2.x to 1.x index schema
• Remove Temp Files
  Removes temporary files downloaded or generated during indexing
• Check Blob Store Files
  Checks for meta files in Microsoft Azure blob storage
• Set Geo Field
  Sets the geo field for the entry from spatial information
• Voyager Server Thumbnail Upload
  Uploads thumbnails generated by the agent to a Voyager Server instance
• Propagate Fields
  Propagates fields to children
• Add Format Tags
  Adds format tags based on a record's mime type
• Set Value for Field
  Sets a field to a specific value. Useful for grouping.
• Append Value to Field
  Appends a value to a field If the field is empty, behaves like Set Value for Field
• Extract Entities with NLP
  Uses Natural Language Processing to extract categorized entities from the text content
• Create Thumbnail with Base Map
  Create Thumbnail using the current Base Map
• GeoTag Standard
  Geotags the document content with the standard Gazetteer
• Geotag Custom
  Geotags the document content with a custom Gazetteer
• Set Extent from PRJ File
  Transforms spatial extent based on projection information found in a component prj file
• Add meta XML tags
  Adds meta XML tags