com.northconcepts.datapipeline.transform

Class RemoveDuplicateFields

java.lang.Object

com.northconcepts.datapipeline.transform.Transformer

com.northconcepts.datapipeline.transform.RemoveDuplicateFields

public class RemoveDuplicateFields
extends Transformer

Removes or collapses multiple fields with the same name in each record. This transformation works with simple field names only and not field paths (FieldPath).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RemoveDuplicateFields.DuplicateFieldsPolicy
static interface	RemoveDuplicateFields.IDuplicateFieldsPolicy
static class	RemoveDuplicateFields.RenamePolicy An alternative to RemoveDuplicateFields.DuplicateFieldsPolicy.RENAME that uses a configurable separator between the original field name and the numeric suffix, preventing ambiguous names like 20101 when the field name itself ends in digits (e.g.

Constructor Summary

Constructors

Constructor and Description
RemoveDuplicateFields(RemoveDuplicateFields.IDuplicateFieldsPolicy duplicateFieldsPolicy, Set<String> fieldNames)
RemoveDuplicateFields(RemoveDuplicateFields.IDuplicateFieldsPolicy duplicateFieldsPolicy, String... fieldNames)

Method Summary

Modifier and Type	Method and Description
Set<String>	getFields()
String	toString()
boolean	transform(Record record) Applies this transformer to a record on its way through a TransformingReader or TransformingWriter.

Methods inherited from class com.northconcepts.datapipeline.transform.Transformer

addExceptionProperties, close, geEndpoint, getEndpoint, getReader, getWriter, of, open, setWriter

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

RemoveDuplicateFields

public RemoveDuplicateFields(RemoveDuplicateFields.IDuplicateFieldsPolicy duplicateFieldsPolicy,
                             Set<String> fieldNames)

RemoveDuplicateFields

public RemoveDuplicateFields(RemoveDuplicateFields.IDuplicateFieldsPolicy duplicateFieldsPolicy,
                             String... fieldNames)

Method Detail

getFields

public Set<String> getFields()

transform

public boolean transform(Record record)
                  throws Throwable

Description copied from class: Transformer

Applies this transformer to a record on its way through a TransformingReader or TransformingWriter.

Contract for subclasses (see also docs/authoring/Transformer.md):

• Mutate record in place as needed.
• Return true to keep the record in the pipeline.
• Return false to drop the record (the host routes it to a discard writer if one is configured, otherwise it disappears).
• Throw to fail (or route to a discard writer, depending on host configuration).
• Do not call incrementRecordCount() on the host endpoint; the host handles it.

Specified by:

transform in class Transformer

Throws:

Throwable

toString

public String toString()

Overrides:

toString in class Object