com.northconcepts.datapipeline.kafka.KafkaReader

public class KafkaReader extends IntegrationReader

Read Records from an Apache Kafka distributed messaging system.

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint
DataEndpoint.State
Field Summary

Fields inherited from class com.northconcepts.datapipeline.core.DataReader
fieldLineage, recordLineage

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint
lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEY

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint
BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject
id, log, name, TIMESTAMP_FORMAT
Constructor Summary

Constructors

Constructor

Description

KafkaReader(Properties properties, String topic, long pollTimeout)

Creates a reader for the topic that waits up to pollTimeout milliseconds for each poll.
Method Summary

Modifier and Type

Method

Description

DataException

addExceptionProperties(DataException exception)

Adds this endpoint's current state to a DataException.

void

close()

Indicates that this endpoint has finished reading or writing.

boolean

isKeepPolling()

Indicates if the reader keeps polling while no messages arrive instead of ending after an empty poll (default is true).

void

open()

Makes this endpoint ready for reading or writing.

protected Record

readImpl()

Overridden by subclasses to read the next record from this DataReader.

KafkaReader

setKeepPolling(boolean keepPolling)

Indicates if the reader keeps polling while no messages arrive instead of ending after an empty poll (default is true).

Methods inherited from class com.northconcepts.datapipeline.core.DataReader
addLineage, available, getBufferSize, getNestedEndpoint, getNestedReader, getReader, getRootEndpoint, getRootReader, isExhausted, isLineageSupported, isSaveLineage, peek, pop, push, read, setSaveLineage, skip

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint
decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, resetRecordCount, toString

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint
addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription

Methods inherited from class com.northconcepts.datapipeline.core.DataObject
exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object
clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details
- KafkaReader
  
  public KafkaReader(Properties properties, String topic, long pollTimeout)
  
  Creates a reader for the topic that waits up to pollTimeout milliseconds for each poll. Opening it puts auto.offset.reset=earliest and the key/value deserializers into the given properties.
Method Details
- isKeepPolling
  
  public boolean isKeepPolling()
  
  Indicates if the reader keeps polling while no messages arrive instead of ending after an empty poll (default is true).
- setKeepPolling
  
  public KafkaReader setKeepPolling(boolean keepPolling)
  
  Indicates if the reader keeps polling while no messages arrive instead of ending after an empty poll (default is true).
- open
  
  public void open() throws DataException
  
  Description copied from class: DataEndpoint
  
  Makes this endpoint ready for reading or writing.
  
  Overrides:
  
  open in class IntegrationReader
  
  Throws:
  
  DataException
- close
  
  public void close() throws DataException
  
  Description copied from class: DataEndpoint
  
  Indicates that this endpoint has finished reading or writing.
  
  Overrides:
  
  close in class DataEndpoint
  
  Throws:
  
  DataException
- readImpl
  
  protected Record readImpl() throws Throwable
  
  Description copied from class: DataReader
  Overridden by subclasses to read the next record from this DataReader. The default implementation of DataReader.read() now insures that this method will not be called again after it returns a null.
  If no record is available, null will be returned.
  Contract for subclasses (see also docs/authoring/DataReader.md):
  
  Return null exactly once to signal end-of-stream.
  
  Do not call this method directly — DataReader.read() is the template method that wraps exceptions, increments the record count, and tracks lineage.
  
  Do not call DataEndpoint.incrementRecordCount() here; DataReader.read() already does.
  
  Throw raw exceptions; DataReader.read() wraps them via exception(throwable).
  Specified by:
  
  readImpl in class DataReader
  
  Throws:
  
  Throwable
- addExceptionProperties
  
  public DataException addExceptionProperties(DataException exception)
  
  Description copied from class: Endpoint
  
  Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.
  
  Overrides:
  
  addExceptionProperties in class DataReader

Class KafkaReader

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint

Field Summary

Fields inherited from class com.northconcepts.datapipeline.core.DataReader

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

Constructor Summary

Method Summary

Methods inherited from class com.northconcepts.datapipeline.core.DataReader

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

Methods inherited from class java.lang.Object

Constructor Details

KafkaReader

Method Details

isKeepPolling

setKeepPolling

open

close

readImpl

addExceptionProperties