Assignment
Setup
• Copy the project CO7X17.miniproject1 under partI in the local copy of your private repository.
• Import the project into your workspace using Gradle Buildship 2.
Migration process setup
There are several clients for the Twitter API that allow us to query data on Twitter. We are going to make calls to the REST API directly via twurl copying data in bulk once.
The data migration process is achieved by executing from a shell terminal the command ./twitterMigrate.sh. The script will show a URL which should be pasted on to a browser and the PIN obtained should be pasted on the terminal. Once authorization is successful, friends and follower's tweets will be filtered and copied onto your MongoDB database on mLab (once you finish exercise 1).
Exercise 1: fetching data from Twitter
In class miniproject1.mongoDb.BulkTransfer, implement the code necessary to transfer data from the stored JSON files in your MongoDB instance on mLab. The database should contain two collections of documents:
• friends: containing information about friends and their tweets.
• followers: containing information about followers and their tweets.
Take into account the following requirements:
• Denormalization is going to be applied and a user may be stored in both collections and, therefore, their tweets will also be stored twice.
• Each user (friend or follower) must contain its tweets in a new field tweet.
• Store the following information about friends and followers
• id_str,
• name,
• description,
• favorites_count,
• followers_count,
• friends_count,
• language,
• location,
• screen_name,
• url,
• utc_offset
• Store the following information about tweets
• id_str,
• created_at,
• entities,
• favorite_count,
• favorited,
• user.id_str,
• in_reply_to_screen_name,
• in_reply_to_status_id,
• in_reply_to_user_id,
• language_code,
• retweet_count,
• retweeted,
• source,
• text
To fetch JSON documents from Twitter for testing your scripts without pushing them to your MongoDB instance, comment the line ./gradlewmigrateToMongoDb in the shell script twitterMigrate.sh using #, that is:
# ./gradlewmigrateToMongoDb
Exercise 2: followers report
In the controller IndexController, add code to the handler method followers in order to list all your follower that have tweets that mention martinfowler, i.e. screen_name=='martinfowler' for an entity user_mentions in a tweet published by your follower.
The controller must get the information from the data on your MongoDB instance on mLab, and not from the JSON files directly.
Exercise 3: friends report
In the controller IndexController, add code to the handler method friends in order to mine some intelligence about your Twitter account:
For each friend, add an object FriendDto into the list friends with the following information:
• name: your friend name
• description: your friend description
• noTweets: their number of tweets (up to 10)
• noRetweets: the overall number of retweets (aggregated retweet count for all tweets) for their selected tweets (those fetched from twurl or in the sample files)
• noActiveTweets: the number of tweets that have caused discussion (whose whose retweet count is greater than 100) for their selected tweets (those fetched from twurl or in the sample files)
The controller must get the information from the data on your MongoDB instance on mLab, and not from the JSON files directly.