Class DataMappingPipeline
java.lang.Object
com.northconcepts.datapipeline.foundations.core.Bean
com.northconcepts.datapipeline.foundations.core.FoundationObject
com.northconcepts.datapipeline.foundations.pipeline.PipelineObject
com.northconcepts.datapipeline.foundations.pipeline.AbstractPipeline
com.northconcepts.datapipeline.foundations.pipeline.DataMappingPipeline
- All Implemented Interfaces:
DataExceptionContributor,JsonSerializable,RecordSerializable,XmlSerializable,DataReaderFactory,DataWriterFactory,JavaCodeGenerator,Serializable
A pipeline that maps each input record to a new record using a
DataMapping; records failing the mapping go to the
discard writer, if one is set, instead of stopping the pipeline.- See Also:
-
Field Summary
Fields inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject
internalId, internalName, log, TIMESTAMP_FORMATFields inherited from interface com.northconcepts.datapipeline.core.RecordSerializable
SERIALIZED_CLASS_NAME, TYPEFields inherited from interface com.northconcepts.datapipeline.core.XmlSerializable
XML_SERIALIZED_CLASS_NAME -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprotected DataReaderapplyDataProcessingImpl(DataReader reader) Wraps the reader with this pipeline's own processing; called between the source and target entity transformers.fromJson(InputStream inputStream) Loads this object's state from the UTF-8 encoded JSON in the stream.Loads this object's state from the given JSON text.fromRecord(Record source) Loads this instance's state from a record and returnsthis(for fluid API call chaining).fromXml(InputStream inputStream) Parses the XML in the stream and loads this object's state from its root element.fromXmlElement(Element dataMappingPipelineElement) voidWrites the Java code for this pipeline's own processing, between the input and output code.booleanIndicates if records skipped because a mapping condition failed are also sent to the discard writer (defaults to false).setDataMapping(DataMapping dataMapping) setDescription(String description) setDiscardReasonFieldName(String discardReasonFieldName) Sets the name of the field added to discarded records to hold the failure message; only allowed with a discard writer.setDiscardWriter(DataWriter discardWriter) Sets the writer that receives records failing source or target entity validation, instead of stopping the pipeline.setInput(PipelineInput input) setInputAsDataReader(DataReader reader) Sets the input to aDataReaderPipelineInputthat always returns the given reader instance.Sets the input to aDataReaderPipelineInputthat creates its readers with the given factory.setMultithreaded(boolean multithreaded) Indicates if each processing stage runs in its own thread using anAsyncReader(defaults to true).setOutput(PipelineOutput output) setOutputAsDataWriter(DataWriter writer) Sets the output to aDataWriterPipelineOutputthat always returns the given writer instance.Sets the output to aDataWriterPipelineOutputthat creates its writers with the given factory.setSendSkippedToDiscardWriter(boolean sendSkippedToDiscardWriter) Indicates if records skipped because a mapping condition failed are also sent to the discard writer (defaults to false).setSourceEntity(EntityDef sourceEntity) Sets the schema entity used to convert and validate records read from the input, before any processing (null for none).setTargetEntity(EntityDef targetEntity) Sets the schema entity used to convert and validate records after processing, before they are written (null for none).toJson()Writes this object as JSON to the writer, then closes the writer.Writes this object as JSON to the writer, closing the writer afterwards ifcloseWriteris true.toRecord()Converts this object's state to a record thatRecordSerializable.fromRecord(Record)can load.Writes this object as indented XML, without a declaration, to the writer and then closes it.Writes this object as indented XML, without a declaration, to the writer, closing it afterwards ifcloseWriteris true.toXmlElement(Document document) Returns an element, created withdocument, describing this object; the default implementation throws aDataException.Methods inherited from class com.northconcepts.datapipeline.foundations.pipeline.AbstractPipeline
applyDataProcessing, applySourceEntityTransformer, applyTargetEntityTransformer, createDataReader, createDataWriter, createJob, generateJavaCode, generateJavaCodePostProcess, generateJavaCodePreProcess, getDateTimePatternDetector, getDescription, getDiscardReasonFieldName, getDiscardWriter, getInput, getJavaCode, getName, getNumberDetector, getOutput, getSourceEntity, getTargetEntity, isMultithreaded, run, runAsync, setDateTimePatternDetector, setNumberDetectorMethods inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject
addExceptionProperties, assertValid, assertValid, clone, exception, exception, exception, getInternalId, getInternalName, resetInternalIdMethods inherited from class java.lang.Object
equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, waitMethods inherited from interface com.northconcepts.datapipeline.core.DataExceptionContributor
addExceptionPropertiesMethods inherited from interface com.northconcepts.datapipeline.core.XmlSerializable
toXml, toXml, toXml
-
Constructor Details
-
DataMappingPipeline
public DataMappingPipeline()
-
-
Method Details
-
getDataMapping
-
setDataMapping
-
applyDataProcessingImpl
Description copied from class:AbstractPipelineWraps the reader with this pipeline's own processing; called between the source and target entity transformers.- Specified by:
applyDataProcessingImplin classAbstractPipeline
-
generateJavaCodeImpl
Description copied from class:AbstractPipelineWrites the Java code for this pipeline's own processing, between the input and output code.- Specified by:
generateJavaCodeImplin classAbstractPipeline
-
toRecord
Description copied from interface:RecordSerializableConverts this object's state to a record thatRecordSerializable.fromRecord(Record)can load.- Specified by:
toRecordin interfaceRecordSerializable- Overrides:
toRecordin classAbstractPipeline
-
fromRecord
Description copied from interface:RecordSerializableLoads this instance's state from a record and returnsthis(for fluid API call chaining). For fluid API call chaining, the overridden method should change the declared return type to its class.- Specified by:
fromRecordin interfaceRecordSerializable- Overrides:
fromRecordin classAbstractPipeline- Parameters:
source-- Returns:
- this instance.
-
fromJson
Description copied from interface:JsonSerializableLoads this object's state from the given JSON text.- Specified by:
fromJsonin interfaceJsonSerializable- Specified by:
fromJsonin interfaceRecordSerializable- Overrides:
fromJsonin classAbstractPipeline
-
fromJson
Description copied from interface:JsonSerializableLoads this object's state from the UTF-8 encoded JSON in the stream. -
toJson
-
toJson
Description copied from interface:JsonSerializableWrites this object as JSON to the writer, then closes the writer. -
toJson
Description copied from interface:JsonSerializableWrites this object as JSON to the writer, closing the writer afterwards ifcloseWriteris true. -
fromXml
Description copied from interface:XmlSerializableParses the XML in the stream and loads this object's state from its root element. -
fromXml
-
toXml
Description copied from interface:XmlSerializableWrites this object as indented XML, without a declaration, to the writer and then closes it. -
toXml
Description copied from interface:XmlSerializableWrites this object as indented XML, without a declaration, to the writer, closing it afterwards ifcloseWriteris true. -
setName
- Overrides:
setNamein classAbstractPipeline
-
setDescription
- Overrides:
setDescriptionin classAbstractPipeline
-
setInput
- Overrides:
setInputin classAbstractPipeline
-
setInputAsDataReaderFactory
Description copied from class:AbstractPipelineSets the input to aDataReaderPipelineInputthat creates its readers with the given factory.- Overrides:
setInputAsDataReaderFactoryin classAbstractPipeline
-
setInputAsDataReader
Description copied from class:AbstractPipelineSets the input to aDataReaderPipelineInputthat always returns the given reader instance.- Overrides:
setInputAsDataReaderin classAbstractPipeline
-
setOutput
- Overrides:
setOutputin classAbstractPipeline
-
setOutputAsDataWriterFactory
Description copied from class:AbstractPipelineSets the output to aDataWriterPipelineOutputthat creates its writers with the given factory.- Overrides:
setOutputAsDataWriterFactoryin classAbstractPipeline
-
setOutputAsDataWriter
Description copied from class:AbstractPipelineSets the output to aDataWriterPipelineOutputthat always returns the given writer instance.- Overrides:
setOutputAsDataWriterin classAbstractPipeline
-
setSourceEntity
Description copied from class:AbstractPipelineSets the schema entity used to convert and validate records read from the input, before any processing (null for none).- Overrides:
setSourceEntityin classAbstractPipeline
-
setTargetEntity
Description copied from class:AbstractPipelineSets the schema entity used to convert and validate records after processing, before they are written (null for none).- Overrides:
setTargetEntityin classAbstractPipeline
-
setMultithreaded
Description copied from class:AbstractPipelineIndicates if each processing stage runs in its own thread using anAsyncReader(defaults to true).- Overrides:
setMultithreadedin classAbstractPipeline
-
setDiscardWriter
Description copied from class:AbstractPipelineSets the writer that receives records failing source or target entity validation, instead of stopping the pipeline.- Overrides:
setDiscardWriterin classAbstractPipeline
-
setDiscardReasonFieldName
Description copied from class:AbstractPipelineSets the name of the field added to discarded records to hold the failure message; only allowed with a discard writer.- Overrides:
setDiscardReasonFieldNamein classAbstractPipeline
-
isSendSkippedToDiscardWriter
public boolean isSendSkippedToDiscardWriter()Indicates if records skipped because a mapping condition failed are also sent to the discard writer (defaults to false). -
setSendSkippedToDiscardWriter
Indicates if records skipped because a mapping condition failed are also sent to the discard writer (defaults to false). -
toXmlElement
Description copied from interface:XmlSerializableReturns an element, created withdocument, describing this object; the default implementation throws aDataException.- Specified by:
toXmlElementin interfaceXmlSerializable- Overrides:
toXmlElementin classAbstractPipeline
-
fromXmlElement
- Specified by:
fromXmlElementin interfaceXmlSerializable- Overrides:
fromXmlElementin classAbstractPipeline
-