{"action":"opened","number":7,"pull_request":{"url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/pulls/7","id":68524828,"html_url":"https://github.com/csula-students/beautiful-data-project-movieteam/pull/7","diff_url":"https://github.com/csula-students/beautiful-data-project-movieteam/pull/7.diff","patch_url":"https://github.com/csula-students/beautiful-data-project-movieteam/pull/7.patch","issue_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/issues/7","number":7,"state":"open","locked":false,"title":"Homework2pull3","user":{"login":"anto004","id":18249431,"avatar_url":"https://avatars.githubusercontent.com/u/18249431?v=3","gravatar_id":"","url":"https://api.github.com/users/anto004","html_url":"https://github.com/anto004","followers_url":"https://api.github.com/users/anto004/followers","following_url":"https://api.github.com/users/anto004/following{/other_user}","gists_url":"https://api.github.com/users/anto004/gists{/gist_id}","starred_url":"https://api.github.com/users/anto004/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anto004/subscriptions","organizations_url":"https://api.github.com/users/anto004/orgs","repos_url":"https://api.github.com/users/anto004/repos","events_url":"https://api.github.com/users/anto004/events{/privacy}","received_events_url":"https://api.github.com/users/anto004/received_events","type":"User","site_admin":false},"body":"1. What question(s) did you decide to work on as a team?\r\nCan we create a model using features of Twitter Tweets regarding movies to predict what user ratings for movies will be?\r\n\r\n2. What is your data source(s)?\r\nWe used Twitter Streaming API as well as .csv file data from http://grouplens.org/datasets/movielens/\r\n\r\n3. How long does it take for you to download data? Have you download complete data set?\r\nThe Csv file when compressed takes about 15 seconds to download. The unzipped file consisted of separate Csv files that we merged using pandas in python. The Twitter Streaming API does not have an end as it is live updates so we can run indefinitely.\r\n\r\n4. How large is your data (size wise and number of records wise)?\r\nThe CSV file is about 1.7 gigabytes with 22,000,000 records. Each of these records is a rating about a movie. Unfortunately we have not finished collecting the Twitter data so we don't know how big it is. We are searching for data based on a query of the form \"#movie_name\" where movie_name is extracted from the CSV file. There are many different movies. Some movie names will naturally yield many tweets whilst other will yield very little. I ran a test run for 1 hour, and I got about 3610 tweets, totaling to 1.23MB of data. If i run this test until the end of the quarter, we will end up with approximately 1.24GB of data. 24 hrs per day * 1.23 MB per hour = 29.52MB of data per day. 6 weeks * 7 days = 42 days. 42 days * 29.52 MB/day = 1240 MB = 1.24 GB\r\n\r\n5. Do you face any dirty data issue? If you do, how did you clean up your data?\r\nFor the CSV file: the largest problem in the csv file was that some titles were not in English. To handle this, the mungee() method within the CsvCollector class checks if the title of the Movie object is not blank or null and does not contain non-english characters using regex. Also, the title in the csv file has the year appended to the title in the format (“TITLE (YEAR)”) so the mungee() method also extracts the year and sets it to the movie object.\r\nFor the Twitter API: There were a lot of duplicate tweets. This happens when a particular user id had sent out many tweets with the same text content, but since each tweet had a unique id, the Twitter API returned each of them as separate tweets. In order to clean this up, I made a class called TwitterResponse that is constructed from the contents of each Tweet. Each TwitterResponse method then has a equals() method that tests for equality between two tweet's text values. I then used a Set to eliminate tweets with duplicate text values.\r\n\r\n6. How do you store the data you downloaded?\r\nWe store the data using MongoDB.\r\n\r\n<img width=\"1440\" src=\"https://cloud.githubusercontent.com/assets/14851822/14837075/387737e2-0bca-11e6-9348-3cca6c7941f7.jpg\">\r\n\r\n<img width=\"1440\" alt=\"screen shot 2016-05-01 at 9 23 07 pm\" src=\"https://cloud.githubusercontent.com/assets/15002174/14947530/c4661dae-0fe9-11e6-951b-fec27e255aa3.png\">","created_at":"2016-05-02T05:54:48Z","updated_at":"2016-05-02T05:54:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/pulls/7/commits","review_comments_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/pulls/7/comments","review_comment_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/pulls/comments{/number}","comments_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/issues/7/comments","statuses_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/statuses/bc88bde73e9bdf160b8f3ad5564cb9613f3d0fa1","head":{"label":"csula-students:homework2pull3","ref":"homework2pull3","sha":"bc88bde73e9bdf160b8f3ad5564cb9613f3d0fa1","user":{"login":"csula-students","id":18237553,"avatar_url":"https://avatars.githubusercontent.com/u/18237553?v=3","gravatar_id":"","url":"https://api.github.com/users/csula-students","html_url":"https://github.com/csula-students","followers_url":"https://api.github.com/users/csula-students/followers","following_url":"https://api.github.com/users/csula-students/following{/other_user}","gists_url":"https://api.github.com/users/csula-students/gists{/gist_id}","starred_url":"https://api.github.com/users/csula-students/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/csula-students/subscriptions","organizations_url":"https://api.github.com/users/csula-students/orgs","repos_url":"https://api.github.com/users/csula-students/repos","events_url":"https://api.github.com/users/csula-students/events{/privacy}","received_events_url":"https://api.github.com/users/csula-students/received_events","type":"Organization","site_admin":false},"repo":{"id":57000012,"name":"beautiful-data-project-movieteam","full_name":"csula-students/beautiful-data-project-movieteam","owner":{"login":"csula-students","id":18237553,"avatar_url":"https://avatars.githubusercontent.com/u/18237553?v=3","gravatar_id":"","url":"https://api.github.com/users/csula-students","html_url":"https://github.com/csula-students","followers_url":"https://api.github.com/users/csula-students/followers","following_url":"https://api.github.com/users/csula-students/following{/other_user}","gists_url":"https://api.github.com/users/csula-students/gists{/gist_id}","starred_url":"https://api.github.com/users/csula-students/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/csula-students/subscriptions","organizations_url":"https://api.github.com/users/csula-students/orgs","repos_url":"https://api.github.com/users/csula-students/repos","events_url":"https://api.github.com/users/csula-students/events{/privacy}","received_events_url":"https://api.github.com/users/csula-students/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/csula-students/beautiful-data-project-movieteam","description":"beautiful-data-project-movieteam created by GitHub Classroom","fork":false,"url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam","forks_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/forks","keys_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/keys{/key_id}","collaborators_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/teams","hooks_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/hooks","issue_events_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/issues/events{/number}","events_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/events","assignees_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/assignees{/user}","branches_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/branches{/branch}","tags_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/tags","blobs_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/git/refs{/sha}","trees_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/git/trees{/sha}","statuses_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/statuses/{sha}","languages_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/languages","stargazers_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/stargazers","contributors_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/contributors","subscribers_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/subscribers","subscription_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/subscription","commits_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/commits{/sha}","git_commits_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/git/commits{/sha}","comments_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/comments{/number}","issue_comment_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/issues/comments{/number}","contents_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/contents/{+path}","compare_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/compare/{base}...{head}","merges_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/merges","archive_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/downloads","issues_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/issues{/number}","pulls_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/pulls{/number}","milestones_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/milestones{/number}","notifications_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/labels{/name}","releases_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/releases{/id}","deployments_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/deployments","created_at":"2016-04-25T00:00:46Z","updated_at":"2016-04-25T01:27:46Z","pushed_at":"2016-05-02T05:54:49Z","git_url":"git://github.com/csula-students/beautiful-data-project-movieteam.git","ssh_url":"git@github.com:csula-students/beautiful-data-project-movieteam.git","clone_url":"https://github.com/csula-students/beautiful-data-project-movieteam.git","svn_url":"https://github.com/csula-students/beautiful-data-project-movieteam","homepage":null,"size":1014,"stargazers_count":1,"watchers_count":1,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":1,"default_branch":"master"}},"base":{"label":"csula-students:master","ref":"master","sha":"e1820174126091afd9648fa5b09bde4e96ae78c2","user":{"login":"csula-students","id":18237553,"avatar_url":"https://avatars.githubusercontent.com/u/18237553?v=3","gravatar_id":"","url":"https://api.github.com/users/csula-students","html_url":"https://github.com/csula-students","followers_url":"https://api.github.com/users/csula-students/followers","following_url":"https://api.github.com/users/csula-students/following{/other_user}","gists_url":"https://api.github.com/users/csula-students/gists{/gist_id}","starred_url":"https://api.github.com/users/csula-students/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/csula-students/subscriptions","organizations_url":"https://api.github.com/users/csula-students/orgs","repos_url":"https://api.github.com/users/csula-students/repos","events_url":"https://api.github.com/users/csula-students/events{/privacy}","received_events_url":"https://api.github.com/users/csula-students/received_events","type":"Organization","site_admin":false},"repo":{"id":57000012,"name":"beautiful-data-project-movieteam","full_name":"csula-students/beautiful-data-project-movieteam","owner":{"login":"csula-students","id":18237553,"avatar_url":"https://avatars.githubusercontent.com/u/18237553?v=3","gravatar_id":"","url":"https://api.github.com/users/csula-students","html_url":"https://github.com/csula-students","followers_url":"https://api.github.com/users/csula-students/followers","following_url":"https://api.github.com/users/csula-students/following{/other_user}","gists_url":"https://api.github.com/users/csula-students/gists{/gist_id}","starred_url":"https://api.github.com/users/csula-students/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/csula-students/subscriptions","organizations_url":"https://api.github.com/users/csula-students/orgs","repos_url":"https://api.github.com/users/csula-students/repos","events_url":"https://api.github.com/users/csula-students/events{/privacy}","received_events_url":"https://api.github.com/users/csula-students/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/csula-students/beautiful-data-project-movieteam","description":"beautiful-data-project-movieteam created by GitHub Classroom","fork":false,"url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam","forks_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/forks","keys_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/keys{/key_id}","collaborators_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/teams","hooks_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/hooks","issue_events_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/issues/events{/number}","events_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/events","assignees_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/assignees{/user}","branches_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/branches{/branch}","tags_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/tags","blobs_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/git/refs{/sha}","trees_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/git/trees{/sha}","statuses_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/statuses/{sha}","languages_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/languages","stargazers_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/stargazers","contributors_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/contributors","subscribers_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/subscribers","subscription_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/subscription","commits_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/commits{/sha}","git_commits_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/git/commits{/sha}","comments_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/comments{/number}","issue_comment_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/issues/comments{/number}","contents_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/contents/{+path}","compare_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/compare/{base}...{head}","merges_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/merges","archive_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/downloads","issues_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/issues{/number}","pulls_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/pulls{/number}","milestones_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/milestones{/number}","notifications_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/labels{/name}","releases_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/releases{/id}","deployments_url":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/deployments","created_at":"2016-04-25T00:00:46Z","updated_at":"2016-04-25T01:27:46Z","pushed_at":"2016-05-02T05:54:49Z","git_url":"git://github.com/csula-students/beautiful-data-project-movieteam.git","ssh_url":"git@github.com:csula-students/beautiful-data-project-movieteam.git","clone_url":"https://github.com/csula-students/beautiful-data-project-movieteam.git","svn_url":"https://github.com/csula-students/beautiful-data-project-movieteam","homepage":null,"size":1014,"stargazers_count":1,"watchers_count":1,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/pulls/7"},"html":{"href":"https://github.com/csula-students/beautiful-data-project-movieteam/pull/7"},"issue":{"href":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/issues/7"},"comments":{"href":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/issues/7/comments"},"review_comments":{"href":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/pulls/7/comments"},"review_comment":{"href":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/pulls/7/commits"},"statuses":{"href":"https://api.github.com/repos/csula-students/beautiful-data-project-movieteam/statuses/bc88bde73e9bdf160b8f3ad5564cb9613f3d0fa1"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":56,"additions":1668,"deletions":1922,"changed_files":65}}