Package com.northconcepts.datapipeline.transform

Class TransformingReader

java.lang.Object
com.northconcepts.datapipeline.core.DataObject
com.northconcepts.datapipeline.core.Endpoint
com.northconcepts.datapipeline.core.DataEndpoint
com.northconcepts.datapipeline.core.DataReader
com.northconcepts.datapipeline.core.ProxyReader
com.northconcepts.datapipeline.transform.TransformingReader
public class TransformingReader extends ProxyReader
A proxy that applies transformations to records passing through.

  • Constructor Details

    • TransformingReader

      public TransformingReader(DataReader reader, DataWriter discardWriter, String discardReasonFieldName)
      Accepts the DataReader to transform and the DataWriter to send discarded records to (due to exception).
      Parameters:
      reader - the source reader to transform.
      discardWriter - the discard sink for bad records.
      discardReasonFieldName - the field in the record to place the description of the transformation that failed along with the exception message.

    • TransformingReader

      public TransformingReader(DataReader reader, DataWriter discardWriter)
      Accepts the DataReader to transform and the DataWriter to send discarded records to (due to exception).
      Parameters:
      reader - the source reader to transform.
      discardWriter - the discard sink for bad records.

    • TransformingReader

      public TransformingReader(DataReader reader)
      Accepts the DataReader to transform.
      Parameters:
      reader - the source reader to transform.

  • Method Details

    • getFilter

      @Deprecated public Filter getFilter()
      Deprecated.
      Use getCondition() instead.

    • setFilter

      @Deprecated public TransformingReader setFilter(Filter filter)
      Deprecated.
      Use setCondition(Filter) instead.

    • getCondition

      public Filter getCondition()
      Returns the condition under which this transformation applies or null if it always applies.

    • setCondition

      public TransformingReader setCondition(Filter condition)
      Sets the condition under which this transformation applies or null if it should always be applied.

    • setCondition

      public TransformingReader setCondition(String condition)
      Sets the condition under which this transformation applies or null if it should always be applied.

    • isExceptionOnFailure

      public boolean isExceptionOnFailure()
      Indicates if a transformer that throws or returns false stops processing with an exception (default is true unless a discard writer is set).

    • setExceptionOnFailure

      public TransformingReader setExceptionOnFailure(boolean exceptionOnFailure)
      Sets the flag to stop processing if an exception is thrown (default) or false if they should be ignored.

    • isRecordStackTraceInMessage

      public boolean isRecordStackTraceInMessage()
      Indicates if the warning added when a transformer returns false includes a stack trace (default is true).

    • setRecordStackTraceInMessage

      public TransformingReader setRecordStackTraceInMessage(boolean recordStackTraceInMessage)
      Indicates if the warning added when a transformer returns false includes a stack trace (default is true).

    • add

      public TransformingReader add(Transformer... transformer)
      Appends transformers; each record passes through them in the order added.

    • getCount

      public int getCount()
      Returns the number of transformers added.

    • get

      public Transformer get(int index)
      Returns the transformer at the given 0-based index.

    • getDiscardWriter

      public DataWriter getDiscardWriter()
      Returns the discard sink for bad records or null if one was not assigned.

    • getDiscardReasonFieldName

      public String getDiscardReasonFieldName()
      Returns the the field in the record to place the description of the transformation that failed along with the exception message.

    • open

      public void open() throws DataException
      Description copied from class: DataEndpoint
      Makes this endpoint ready for reading or writing.
      Overrides:
      open in class ProxyReader
      Throws:
      DataException

    • close

      public void close() throws DataException
      Description copied from class: DataEndpoint
      Indicates that this endpoint has finished reading or writing.
      Overrides:
      close in class ProxyReader
      Throws:
      DataException

    • interceptRecord

      protected Record interceptRecord(Record record) throws Throwable
      Description copied from class: ProxyReader
      Overridden by subclasses to transform, drop, or observe a record on its way through this proxy.

      Contract for subclasses (see also docs/authoring/ProxyReader.md):

      • Return the record (possibly modified) to pass it downstream.
      • Return null to drop the record; the framework reads the next one.
      • Throw to fail the pipeline; the framework wraps the throwable in a DataException with .setRecord(record).
      • The nested DataReader is opened and closed for you; do not read from it here.
      • Do not call DataEndpoint.incrementRecordCount() here.
      Overrides:
      interceptRecord in class ProxyReader
      Throws:
      Throwable

    • transformRecord

      protected boolean transformRecord(Record record, Transformer transformer, Messages messages)
      Applies the transformer to the record, passing any failure (an exception or a false result) to onFailure; returns true if the record should continue.

    • onFailure

      protected void onFailure(Record record, Transformer transformer, DataException exception, String message, Messages messages)
      Handles a failed transformation (exception is null if it returned false): sends the record to the discard writer if there is one, otherwise throws or, when exceptionOnFailure is off, adds warnings to messages.

    • addExceptionProperties

      public DataException addExceptionProperties(DataException exception)
      Description copied from class: Endpoint
      Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.
      Overrides:
      addExceptionProperties in class ProxyReader