Class TwitterPageableReader<E,C>
java.lang.Object
com.northconcepts.datapipeline.core.DataObject
com.northconcepts.datapipeline.core.Endpoint
com.northconcepts.datapipeline.core.DataEndpoint
com.northconcepts.datapipeline.core.DataReader
com.northconcepts.datapipeline.internal.lang.IntegrationReader
com.northconcepts.datapipeline.twitter.TwitterReader
com.northconcepts.datapipeline.twitter.TwitterPageableReader<E,C>
- Direct Known Subclasses:
TwitterFollowerIDsReader,TwitterFollowerListReader,TwitterFollowingIDsReader,TwitterFollowingListReader,TwitterListMemberReader,TwitterListOwnershipReader
Base class for Twitter readers that page through cursor-based API results, returning one record per element.
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic enumPaging direction: follow each page's next cursor (FORWARD) or its previous cursor (BACKWARD).Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint
DataEndpoint.State -
Field Summary
FieldsFields inherited from class com.northconcepts.datapipeline.twitter.TwitterReader
EOFFields inherited from class com.northconcepts.datapipeline.core.DataReader
fieldLineage, recordLineageFields inherited from class com.northconcepts.datapipeline.core.DataEndpoint
lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEYFields inherited from class com.northconcepts.datapipeline.core.Endpoint
BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZEFields inherited from class com.northconcepts.datapipeline.core.DataObject
id, log, name, TIMESTAMP_FORMAT -
Constructor Summary
ConstructorsConstructorDescriptionTwitterPageableReader(String rateLimitResource, String rateLimitKey, TwitterProvider provider, int pageSize) TwitterPageableReader(String rateLimitResource, String rateLimitKey, String consumerKey, String consumerSecret, int pageSize) TwitterPageableReader(String rateLimitResource, String rateLimitKey, String consumerKey, String consumerSecret, String accessToken, String accessTokenSecret, int pageSize) -
Method Summary
Modifier and TypeMethodDescriptionaddExceptionProperties(DataException exception) Adds this endpoint's current state to aDataException.protected abstract voidaddPageToBuffer(C list, LinkedList<E> buffer) Appends the elements of a fetched page to the buffer of elements still to be returned.protected abstract voidcopyToRecord(E element, Record record, IEntityExpansionPolicy entityExpansionPolicy) Populates a new, empty record from one buffered element.Returns whether paging follows the next or the previous cursors (defaults toTwitterPageableReader.Direction.FORWARD).longgetPage()Returns the cursor of the next page to request.protected abstract CgetPageImpl(long page, int pageSize) Fetches the page at the given cursor, requestingpageSizeelements.intlongReturns the cursor of the first page requested (defaults toFIRST_PAGE).voidopen()Makes this endpoint ready for reading or writing.protected RecordreadImpl()Overridden by subclasses to read the next record from thisDataReader.setApiLimitPolicy(IApiLimitPolicy apiLimitPolicy) Sets the action taken when the API call limit is reached (defaults toApiLimitPolicy.FAIL).setDirection(TwitterPageableReader.Direction direction) Sets whether paging follows the next or the previous cursors (defaults toTwitterPageableReader.Direction.FORWARD).setStartingOffset(long startingOffset) Sets the cursor of the first page requested (defaults toFIRST_PAGE).Methods inherited from class com.northconcepts.datapipeline.twitter.TwitterReader
getApiLimitPolicy, getConverter, getEntityExpansionPolicy, getProvider, getRateLimit, getRateLimit, isRateLimitExceeded, onRateLimitExceeded, reloadRateLimit, setConverter, setEntityExpansionPolicy, setRateLimit, setRateLimit, setRateLimitExceededMethods inherited from class com.northconcepts.datapipeline.core.DataReader
addLineage, available, getBufferSize, getNestedEndpoint, getNestedReader, getReader, getRootEndpoint, getRootReader, isExhausted, isLineageSupported, isSaveLineage, peek, pop, push, read, setSaveLineage, skipMethods inherited from class com.northconcepts.datapipeline.core.DataEndpoint
close, decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, resetRecordCount, toStringMethods inherited from class com.northconcepts.datapipeline.core.Endpoint
addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription
-
Field Details
-
FIRST_PAGE
public static final int FIRST_PAGE- See Also:
-
-
Constructor Details
-
TwitterPageableReader
public TwitterPageableReader(String rateLimitResource, String rateLimitKey, TwitterProvider provider, int pageSize) -
TwitterPageableReader
-
TwitterPageableReader
-
-
Method Details
-
getPageSize
public int getPageSize() -
getStartingOffset
public long getStartingOffset()Returns the cursor of the first page requested (defaults toFIRST_PAGE). -
setStartingOffset
Sets the cursor of the first page requested (defaults toFIRST_PAGE). -
getPage
public long getPage()Returns the cursor of the next page to request. -
getDirection
Returns whether paging follows the next or the previous cursors (defaults toTwitterPageableReader.Direction.FORWARD). -
setDirection
Sets whether paging follows the next or the previous cursors (defaults toTwitterPageableReader.Direction.FORWARD). -
setApiLimitPolicy
Description copied from class:TwitterReaderSets the action taken when the API call limit is reached (defaults toApiLimitPolicy.FAIL).- Overrides:
setApiLimitPolicyin classTwitterReader
-
open
Description copied from class:DataEndpointMakes this endpoint ready for reading or writing.- Overrides:
openin classTwitterReader- Throws:
DataException
-
getPageImpl
Fetches the page at the given cursor, requestingpageSizeelements.- Throws:
Throwable
-
addPageToBuffer
Appends the elements of a fetched page to the buffer of elements still to be returned.- Throws:
Throwable
-
copyToRecord
protected abstract void copyToRecord(E element, Record record, IEntityExpansionPolicy entityExpansionPolicy) throws Throwable Populates a new, empty record from one buffered element.- Throws:
Throwable
-
readImpl
Description copied from class:DataReaderOverridden by subclasses to read the next record from thisDataReader. The default implementation ofDataReader.read()now insures that this method will not be called again after it returns anull.If no record is available,
nullwill be returned.Contract for subclasses (see also
docs/authoring/DataReader.md):- Return
nullexactly once to signal end-of-stream. - Do not call this method directly —
DataReader.read()is the template method that wraps exceptions, increments the record count, and tracks lineage. - Do not call
DataEndpoint.incrementRecordCount()here;DataReader.read()already does. - Throw raw exceptions;
DataReader.read()wraps them viaexception(throwable).
- Specified by:
readImplin classDataReader- Throws:
Throwable
- Return
-
addExceptionProperties
Description copied from class:EndpointAdds this endpoint's current state to aDataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.- Overrides:
addExceptionPropertiesin classTwitterReader
-