Package com.northconcepts.datapipeline.transform

Class Ngrams

java.lang.Object
com.northconcepts.datapipeline.transform.Transformer
com.northconcepts.datapipeline.transform.Ngrams
public class Ngrams extends Transformer
Extracts each run/sequence of N words from the source field and stores them in a target field as an array. See n-gram, Bigrams, and Trigrams for more details.

  • Constructor Details

    • Ngrams

      public Ngrams(String sourceFieldPath, String targetFieldPath, int ngramCount, Locale locale)
      Creates a transformer that stores each run of ngramCount words as one space-separated string, finding words with the given locale's rules (the default locale if null).

    • Ngrams

      public Ngrams(String sourceFieldPath, String targetFieldPath, int ngramCount)
      Creates a transformer that stores each run of ngramCount words as one space-separated string, finding words with the default locale's rules.

  • Method Details

    • transform

      public boolean transform(Record record) throws Throwable
      Description copied from class: Transformer
      Applies this transformer to a record on its way through a TransformingReader or TransformingWriter.

      Contract for subclasses (see also docs/authoring/Transformer.md):

      • Mutate record in place as needed.
      • Return true to keep the record in the pipeline.
      • Return false to drop the record (the host routes it to a discard writer if one is configured, otherwise it disappears).
      • Throw to fail (or route to a discard writer, depending on host configuration).
      • Do not call incrementRecordCount() on the host endpoint; the host handles it.
      Specified by:
      transform in class Transformer
      Throws:
      Throwable

    • toString

      public String toString()
      Overrides:
      toString in class Object