com.northconcepts.datapipeline.twitter.TwitterUserLookupReader

public class TwitterUserLookupReader extends TwitterProxyReader

Adds Twitter user details to records from a nested reader by looking up the user ID or screen name in a source field.

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint
DataEndpoint.State
Field Summary

Fields

Modifier and Type

Field

Description

static final int

MAX_BATCH_SIZE

Fields inherited from class com.northconcepts.datapipeline.twitter.TwitterProxyReader
EOF

Fields inherited from class com.northconcepts.datapipeline.core.DataReader
fieldLineage, recordLineage

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint
lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEY

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint
BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject
id, log, name, TIMESTAMP_FORMAT
Constructor Summary

Constructors

Constructor

Description

TwitterUserLookupReader(TwitterProvider provider, DataReader nestedDataReader, String sourceIdFieldName)

Creates a reader that looks up users by the ID or screen name at the sourceIdFieldName field path.

TwitterUserLookupReader(String consumerKey, String consumerSecret, DataReader nestedDataReader, String sourceIdFieldName)

Creates a reader that looks up users by the ID or screen name at the sourceIdFieldName field path.

TwitterUserLookupReader(String consumerKey, String consumerSecret, String accessToken, String accessTokenSecret, DataReader nestedDataReader, String sourceIdFieldName)

Creates a reader that looks up users by the ID or screen name at the sourceIdFieldName field path.
Method Summary

Modifier and Type

Method

Description

DataException

addExceptionProperties(DataException exception)

Adds this endpoint's current state to a DataException.

void

close()

Indicates that this endpoint has finished reading or writing.

int

getBatchSize()

Returns the number of records looked up per API request (default is 100).

DataWriter

getDiscardWriter()

Returns the discard sink for records where the lookup failed or null if one was not assigned.

boolean

isLookupById()

Indicates if users are looked up by ID (the default); the source field must then hold Long values.

boolean

isLookupByScreenName()

Indicates if users are looked up by the screen name in the source field instead of by ID (default is false).

boolean

isReturnOrphans()

Indicates if records failing the lookup should be returned — instead of being deleted (default is false).

protected void

onUserFound(Long userId, String screenName, Record record)

Called for each user found.

protected void

onUserNotFound(Long userId, String screenName, Record record)

Called for each user not found.

void

open()

Makes this endpoint ready for reading or writing.

protected Record

readImpl()

Overridden by subclasses to read the next record from this DataReader.

TwitterUserLookupReader

setBatchSize(int batchSize)

Sets the number of records looked up per API request, from 1 to MAX_BATCH_SIZE (default is 100).

TwitterUserLookupReader

setDiscardWriter(DataWriter writer)

Assign a discard sink for records where the lookup failed.

TwitterUserLookupReader

setLookupByScreenName(boolean lookupByScreenName)

Indicates if users are looked up by the screen name in the source field instead of by ID (default is false).

TwitterUserLookupReader

setReturnOrphans(boolean returnOrphans)

Indicates if records failing the lookup should be returned — instead of being deleted (default is false).

Methods inherited from class com.northconcepts.datapipeline.twitter.TwitterProxyReader
getApiLimitPolicy, getConverter, getEntityExpansionPolicy, getProvider, getRateLimit, getRateLimit, isRateLimitExceeded, onRateLimitExceeded, reloadRateLimit, setApiLimitPolicy, setConverter, setEntityExpansionPolicy, setRateLimit, setRateLimit, setRateLimitExceeded

Methods inherited from class com.northconcepts.datapipeline.core.ProxyReader
available, getNestedReader, interceptRecord, map, map, setNestedDataReader, setNestedDataReader

Methods inherited from class com.northconcepts.datapipeline.core.DataReader
addLineage, getBufferSize, getNestedEndpoint, getReader, getRootEndpoint, getRootReader, isExhausted, isLineageSupported, isSaveLineage, peek, pop, push, read, setSaveLineage, skip

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint
decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, resetRecordCount, toString

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint
addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription

Methods inherited from class com.northconcepts.datapipeline.core.DataObject
exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object
clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details
- MAX_BATCH_SIZE
  
  public static final int MAX_BATCH_SIZE
  See Also:
  
  Constant Field Values
Constructor Details
- TwitterUserLookupReader
  
  public TwitterUserLookupReader(String consumerKey, String consumerSecret, DataReader nestedDataReader, String sourceIdFieldName)
  
  Creates a reader that looks up users by the ID or screen name at the sourceIdFieldName field path.
- TwitterUserLookupReader
  
  public TwitterUserLookupReader(String consumerKey, String consumerSecret, String accessToken, String accessTokenSecret, DataReader nestedDataReader, String sourceIdFieldName)
  
  Creates a reader that looks up users by the ID or screen name at the sourceIdFieldName field path.
- TwitterUserLookupReader
  
  public TwitterUserLookupReader(TwitterProvider provider, DataReader nestedDataReader, String sourceIdFieldName)
  
  Creates a reader that looks up users by the ID or screen name at the sourceIdFieldName field path.
Method Details
- isLookupByScreenName
  
  public boolean isLookupByScreenName()
  
  Indicates if users are looked up by the screen name in the source field instead of by ID (default is false).
- isLookupById
  
  public boolean isLookupById()
  
  Indicates if users are looked up by ID (the default); the source field must then hold Long values.
- setLookupByScreenName
  
  public TwitterUserLookupReader setLookupByScreenName(boolean lookupByScreenName)
  
  Indicates if users are looked up by the screen name in the source field instead of by ID (default is false).
- getDiscardWriter
  
  public DataWriter getDiscardWriter()
  
  Returns the discard sink for records where the lookup failed or null if one was not assigned.
- setDiscardWriter
  
  public TwitterUserLookupReader setDiscardWriter(DataWriter writer)
  
  Assign a discard sink for records where the lookup failed.
  
  Parameters:
  
  writer - the discard sink
- isReturnOrphans
  
  public boolean isReturnOrphans()
  
  Indicates if records failing the lookup should be returned — instead of being deleted (default is false).
- setReturnOrphans
  
  public TwitterUserLookupReader setReturnOrphans(boolean returnOrphans)
  
  Indicates if records failing the lookup should be returned — instead of being deleted (default is false).
- getBatchSize
  
  public int getBatchSize()
  
  Returns the number of records looked up per API request (default is 100).
- setBatchSize
  
  public TwitterUserLookupReader setBatchSize(int batchSize)
  
  Sets the number of records looked up per API request, from 1 to MAX_BATCH_SIZE (default is 100).
- open
  
  public void open() throws DataException
  
  Description copied from class: DataEndpoint
  
  Makes this endpoint ready for reading or writing.
  
  Overrides:
  
  open in class TwitterProxyReader
  
  Throws:
  
  DataException
- close
  
  public void close() throws DataException
  
  Description copied from class: DataEndpoint
  
  Indicates that this endpoint has finished reading or writing.
  
  Overrides:
  
  close in class ProxyReader
  
  Throws:
  
  DataException
- readImpl
  
  protected Record readImpl() throws Throwable
  
  Description copied from class: DataReader
  Overridden by subclasses to read the next record from this DataReader. The default implementation of DataReader.read() now insures that this method will not be called again after it returns a null.
  If no record is available, null will be returned.
  Contract for subclasses (see also docs/authoring/DataReader.md):
  
  Return null exactly once to signal end-of-stream.
  
  Do not call this method directly — DataReader.read() is the template method that wraps exceptions, increments the record count, and tracks lineage.
  
  Do not call DataEndpoint.incrementRecordCount() here; DataReader.read() already does.
  
  Throw raw exceptions; DataReader.read() wraps them via exception(throwable).
  Overrides:
  
  readImpl in class ProxyReader
  
  Throws:
  
  Throwable
- onUserFound
  
  protected void onUserFound(Long userId, String screenName, Record record)
  
  Called for each user found.
  
  Parameters:
  
  userId -
- onUserNotFound
  
  protected void onUserNotFound(Long userId, String screenName, Record record)
  
  Called for each user not found.
- addExceptionProperties
  
  public DataException addExceptionProperties(DataException exception)
  
  Description copied from class: Endpoint
  
  Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.
  
  Overrides:
  
  addExceptionProperties in class TwitterProxyReader

Class TwitterUserLookupReader

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint

Field Summary

Fields inherited from class com.northconcepts.datapipeline.twitter.TwitterProxyReader

Fields inherited from class com.northconcepts.datapipeline.core.DataReader

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

Constructor Summary

Method Summary

Methods inherited from class com.northconcepts.datapipeline.twitter.TwitterProxyReader

Methods inherited from class com.northconcepts.datapipeline.core.ProxyReader

Methods inherited from class com.northconcepts.datapipeline.core.DataReader

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

Methods inherited from class java.lang.Object

Field Details

MAX_BATCH_SIZE

Constructor Details

TwitterUserLookupReader

TwitterUserLookupReader

TwitterUserLookupReader

Method Details

isLookupByScreenName

isLookupById

setLookupByScreenName

getDiscardWriter

setDiscardWriter

isReturnOrphans

setReturnOrphans

getBatchSize

setBatchSize

open

close

readImpl

onUserFound

onUserNotFound

addExceptionProperties