{"action":"created","issue":{"url":"https://api.github.com/repos/moritzschaefer/the-search-engine/issues/1","repository_url":"https://api.github.com/repos/moritzschaefer/the-search-engine","labels_url":"https://api.github.com/repos/moritzschaefer/the-search-engine/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/moritzschaefer/the-search-engine/issues/1/comments","events_url":"https://api.github.com/repos/moritzschaefer/the-search-engine/issues/1/events","html_url":"https://github.com/moritzschaefer/the-search-engine/issues/1","id":143860914,"number":1,"title":"Crawler To-Do List","user":{"login":"brianignacio5","id":6033246,"avatar_url":"https://avatars.githubusercontent.com/u/6033246?v=3","gravatar_id":"","url":"https://api.github.com/users/brianignacio5","html_url":"https://github.com/brianignacio5","followers_url":"https://api.github.com/users/brianignacio5/followers","following_url":"https://api.github.com/users/brianignacio5/following{/other_user}","gists_url":"https://api.github.com/users/brianignacio5/gists{/gist_id}","starred_url":"https://api.github.com/users/brianignacio5/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brianignacio5/subscriptions","organizations_url":"https://api.github.com/users/brianignacio5/orgs","repos_url":"https://api.github.com/users/brianignacio5/repos","events_url":"https://api.github.com/users/brianignacio5/events{/privacy}","received_events_url":"https://api.github.com/users/brianignacio5/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-03-27T23:41:04Z","updated_at":"2016-03-29T05:06:54Z","closed_at":null,"body":"# Details of Web Crawler\r\n\r\nDefine the following functions which performs the crawler steps.\r\n*Database crawler:* Loop in charge to pass the url of authors list in http://dblp.uni-trier.de/pers?pos=# starting from 1 and incrementing 300\r\n\r\n## Functionality\r\n\r\n1. [x] Loop in charge to pass the url of authors list in http://dblp.uni-trier.de/pers?pos=# starting from 1 and incrementing 300.\r\n2. [x] Loop in charge to pass the url of journals list in http://dblp.uni-trier.de/db/journals/?pos=# starting from 1 and incrementing 100.\r\n3. [x] Loop in charge to pass the url of conferences list in http://dblp.uni-trier.de/db/conf/?pos=# starting from 1 and incrementing 100.\r\n4. [x] Design decision: A whole list of Authors URLs or a segmented one. ? Suppose a whole list, then (1) produce a whole list and pass this list to (3).\r\n5. [x] Receive list of URL and call crawl_site() to create the whole/partial raw directory.\r\n6. [ ] Test crawl_site on Journals and Conferences\r\n7. [ ] Test crawl_url on Journals and Conferences\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/moritzschaefer/the-search-engine/issues/comments/202712135","html_url":"https://github.com/moritzschaefer/the-search-engine/issues/1#issuecomment-202712135","issue_url":"https://api.github.com/repos/moritzschaefer/the-search-engine/issues/1","id":202712135,"user":{"login":"moritzschaefer","id":2714540,"avatar_url":"https://avatars.githubusercontent.com/u/2714540?v=3","gravatar_id":"","url":"https://api.github.com/users/moritzschaefer","html_url":"https://github.com/moritzschaefer","followers_url":"https://api.github.com/users/moritzschaefer/followers","following_url":"https://api.github.com/users/moritzschaefer/following{/other_user}","gists_url":"https://api.github.com/users/moritzschaefer/gists{/gist_id}","starred_url":"https://api.github.com/users/moritzschaefer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/moritzschaefer/subscriptions","organizations_url":"https://api.github.com/users/moritzschaefer/orgs","repos_url":"https://api.github.com/users/moritzschaefer/repos","events_url":"https://api.github.com/users/moritzschaefer/events{/privacy}","received_events_url":"https://api.github.com/users/moritzschaefer/received_events","type":"User","site_admin":false},"created_at":"2016-03-29T05:06:54Z","updated_at":"2016-03-29T05:06:54Z","body":"TODO for bian: the two tests :). Just duplicate the code for authors"}}