Package com.northconcepts.datapipeline.transform

Class RemoveDuplicateFields

java.lang.Object
com.northconcepts.datapipeline.transform.Transformer
com.northconcepts.datapipeline.transform.RemoveDuplicateFields
public class RemoveDuplicateFields extends Transformer
Removes or collapses multiple fields with the same name in each record. This transformation works with simple field names only and not field paths (FieldPath).

  • Constructor Details

    • RemoveDuplicateFields

      public RemoveDuplicateFields(RemoveDuplicateFields.IDuplicateFieldsPolicy duplicateFieldsPolicy, Set<String> fieldNames)
      Creates a transformer that applies the policy to fields sharing one of the given names, or any name if the set is null or empty.

    • RemoveDuplicateFields

      public RemoveDuplicateFields(RemoveDuplicateFields.IDuplicateFieldsPolicy duplicateFieldsPolicy, String... fieldNames)
      Creates a transformer that applies the policy to fields sharing one of the given names, or any name if none are given.

  • Method Details

    • getFields

      public Set<String> getFields()
      Returns the field names checked for duplicates; null or empty means all names.

    • transform

      public boolean transform(Record record) throws Throwable
      Description copied from class: Transformer
      Applies this transformer to a record on its way through a TransformingReader or TransformingWriter.

      Contract for subclasses (see also docs/authoring/Transformer.md):

      • Mutate record in place as needed.
      • Return true to keep the record in the pipeline.
      • Return false to drop the record (the host routes it to a discard writer if one is configured, otherwise it disappears).
      • Throw to fail (or route to a discard writer, depending on host configuration).
      • Do not call incrementRecordCount() on the host endpoint; the host handles it.
      Specified by:
      transform in class Transformer
      Throws:
      Throwable

    • toString

      public String toString()
      Overrides:
      toString in class Object