Package com.northconcepts.datapipeline.twitter

Class TwitterSearchReader

java.lang.Object
com.northconcepts.datapipeline.core.DataObject
com.northconcepts.datapipeline.core.Endpoint
com.northconcepts.datapipeline.core.DataEndpoint
com.northconcepts.datapipeline.core.DataReader
com.northconcepts.datapipeline.internal.lang.IntegrationReader
com.northconcepts.datapipeline.twitter.TwitterReader
com.northconcepts.datapipeline.twitter.TwitterSearchReader
public class TwitterSearchReader extends TwitterReader
Obtains records by searching Twitter for the most recent tweets matching a search criteria.

This reader uses the Twitter Search API and supports all Query Operators.

Example of record returned:

 Record {
    0:[Id]:LONG=[509084459472551936]:Long
    1:[Text]:STRING=[+ @Squarespace invalid input: '&' @Shutterstock MT @NewYork_CM: #TY again @Freshbooks invalid input: '&' @MailChimp for #CreativeMornings! https://t.co/aW7cdAMyYW...140]:String
    2:[Lang]:STRING=[en]:String
    3:[CreatedAt]:DATETIME=[Mon Sep 08 17:02:48 EDT 2014]:Date
    4:[FavoriteCount]:INT=[1]:Integer
    5:[RetweetCount]:INT=[0]:Integer
    6:[InReplyToScreenName]:STRING=[NewYork_CM]:String
    7:[UserScreenName]:STRING=[mitgc_cm]:String
    8:[UserDescription]:STRING=[#Socbiz strat + #Lean venture dev | @Plus_SocialGood Connector | #socent #impinv #sustdev | @StartingBloc #SocInn | NY exec prod...157]:String
    9:[UserCreatedAt]:DATETIME=[Mon Nov 28 21:09:22 EST 2011]:Date
    10:[UserTweets]:INT=[42485]:Integer
    11:[UserFavouritesCount]:INT=[13329]:Integer
    12:[UserFollowersCount]:INT=[6598]:Integer
    13:[UserFollowingCount]:INT=[6912]:Integer
    14:[UserLocation]:STRING=[New York, New York, USA]:String
    15:[UserLang]:STRING=[en]:String
    16:[UserTimeZone]:STRING=[London]:String
    17:[UserUtcOffset]:INT=[3600]:Integer
    18:[UserURL]:STRING=[http://t.co/D8ykYnFD3I]:String
    19:[GeoLocationLatitude]:DOUBLE=[40.7450718]:Double
    20:[GeoLocationLongitude]:DOUBLE=[-73.9964114]:Double
    21:[PlaceName]:STRING=[Manhattan]:String
    22:[PlaceFullName]:STRING=[Manhattan, NY]:String
    23:[PlaceType]:STRING=[city]:String
    24:[PlaceStreetAddress]:UNDEFINED=[null]
    25:[PlaceCountryCode]:STRING=[US]:String
    26:[PlaceCountry]:STRING=[United States]:String
    27:[PlaceBoundingBoxType]:STRING=[Polygon]:String
    28:[PlaceBoundingBoxCoordinates]:STRING=[[[40.683935,-74.026675],[40.683935,-73.910408],[40.877483,-73.910408],[40.877483,-74.026675]]]:String
    29:[UserMentionEntities]:STRING=[@Squarespace @Shutterstock @NewYork_CM @freshbooks @MailChimp @mitgc_cm]:String
    30:[HashtagEntities]:STRING=[#TY #CreativeMornings]:String
    31:[URLEntities]:STRING=[https://pbs.twimg.com/media/BwxyTHtIQAAKDUz.jpg]:String
    32:[MediaEntities]:UNDEFINED=[null]
}

  • Constructor Details

    • TwitterSearchReader

      public TwitterSearchReader(TwitterProvider provider, String searchQuery, int maxResults)
      Creates a reader for tweets matching searchQuery that stops after maxResults tweets.

    • TwitterSearchReader

      public TwitterSearchReader(String consumerKey, String consumerSecret, String searchQuery, int maxResults)
      Creates a reader for tweets matching searchQuery that stops after maxResults tweets.

    • TwitterSearchReader

      public TwitterSearchReader(String consumerKey, String consumerSecret, String accessToken, String accessTokenSecret, String searchQuery, int maxResults)
      Creates a reader for tweets matching searchQuery that stops after maxResults tweets.

  • Method Details

    • getMaxId

      public long getMaxId()
      Searches for tweets with IDs less than the specified ID (i.e. tweets created earlier) or -1 for all tweets (default-1).

    • setMaxId

      public TwitterSearchReader setMaxId(long maxId)
      Searches for tweets with IDs less than the specified ID (i.e. tweets created earlier) or -1 for all tweets (default-1).

    • getSinceId

      public long getSinceId()
      Searches for tweets with IDs greater than the specified ID (i.e. tweets created later) or -1 for all tweets (default-1).

    • setSinceId

      public TwitterSearchReader setSinceId(long sinceId)
      Searches for tweets with IDs greater than the specified ID (i.e. tweets created later) or -1 for all tweets (default-1).

    • open

      public void open() throws DataException
      Description copied from class: DataEndpoint
      Makes this endpoint ready for reading or writing.
      Overrides:
      open in class TwitterReader
      Throws:
      DataException

    • close

      public void close() throws DataException
      Description copied from class: DataEndpoint
      Indicates that this endpoint has finished reading or writing.
      Overrides:
      close in class DataEndpoint
      Throws:
      DataException

    • setApiLimitPolicy

      public TwitterSearchReader setApiLimitPolicy(IApiLimitPolicy apiLimitPolicy)
      Description copied from class: TwitterReader
      Sets the action taken when the API call limit is reached (defaults to ApiLimitPolicy.FAIL).
      Overrides:
      setApiLimitPolicy in class TwitterReader

    • readImpl

      protected Record readImpl() throws Throwable
      Description copied from class: DataReader
      Overridden by subclasses to read the next record from this DataReader. The default implementation of DataReader.read() now insures that this method will not be called again after it returns a null.

      If no record is available, null will be returned.

      Contract for subclasses (see also docs/authoring/DataReader.md):

      Specified by:
      readImpl in class DataReader
      Throws:
      Throwable

    • addExceptionProperties

      public DataException addExceptionProperties(DataException exception)
      Description copied from class: Endpoint
      Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.
      Overrides:
      addExceptionProperties in class TwitterReader