Package com.northconcepts.datapipeline.core

Class RemoveDuplicatesReader

java.lang.Object
com.northconcepts.datapipeline.core.DataObject
com.northconcepts.datapipeline.core.Endpoint
com.northconcepts.datapipeline.core.DataEndpoint
com.northconcepts.datapipeline.core.DataReader
com.northconcepts.datapipeline.core.ProxyReader
com.northconcepts.datapipeline.core.RemoveDuplicatesReader
public class RemoveDuplicatesReader extends ProxyReader
A proxy that removes duplicate records. Duplicates can be determined by checking all fields (RemoveDuplicatesReader(DataReader)) or subset of fields (RemoveDuplicatesReader(DataReader, FieldList)).

  • Constructor Details

    • RemoveDuplicatesReader

      public RemoveDuplicatesReader(DataReader targetDataReader, FieldList fields)
      Removes duplicate records based on the matching FieldList.
      Parameters:
      targetDataReader - source DataReader
      fields - the FieldList to be match

    • RemoveDuplicatesReader

      public RemoveDuplicatesReader(DataReader targetDataReader, String... fields)
      Remove duplicate records based on their field names.
      Parameters:
      targetDataReader - source DataReader
      fields - the field names to be checked for duplicates

    • RemoveDuplicatesReader

      public RemoveDuplicatesReader(DataReader targetDataReader)
      Remove duplicate records matching all fields.
      Parameters:
      targetDataReader - source DataReader

  • Method Details

    • getDiscardWriter

      public DataWriter getDiscardWriter()
      Returns the discard sink for discarded records or null if one was not assigned.

    • setDiscardWriter

      public RemoveDuplicatesReader setDiscardWriter(DataWriter writer)
      Assign a discard sink for duplicate records.
      Parameters:
      writer - the discard sink

    • getUniqueRecordCount

      public long getUniqueRecordCount()
      Returns the number of unique records that have been read.

    • getDuplicateRecordCount

      public long getDuplicateRecordCount()
      Returns the number of duplicate records that have been read.

    • open

      public void open() throws DataException
      Description copied from class: DataEndpoint
      Makes this endpoint ready for reading or writing.
      Overrides:
      open in class ProxyReader
      Throws:
      DataException

    • close

      public void close() throws DataException
      Description copied from class: DataEndpoint
      Indicates that this endpoint has finished reading or writing.
      Overrides:
      close in class ProxyReader
      Throws:
      DataException

    • interceptRecord

      protected Record interceptRecord(Record record) throws Throwable
      Description copied from class: ProxyReader
      Overridden by subclasses to transform, drop, or observe a record on its way through this proxy.

      Contract for subclasses (see also docs/authoring/ProxyReader.md):

      • Return the record (possibly modified) to pass it downstream.
      • Return null to drop the record; the framework reads the next one.
      • Throw to fail the pipeline; the framework wraps the throwable in a DataException with .setRecord(record).
      • The nested DataReader is opened and closed for you; do not read from it here.
      • Do not call DataEndpoint.incrementRecordCount() here.
      Overrides:
      interceptRecord in class ProxyReader
      Throws:
      Throwable

    • onUnique

      protected void onUnique(Record record)
      Called for each unique record.

    • onDuplicate

      protected void onDuplicate(Record record)
      Called for each duplicate record.

    • addExceptionProperties

      public DataException addExceptionProperties(DataException exception)
      Description copied from class: Endpoint
      Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.
      Overrides:
      addExceptionProperties in class ProxyReader