{"action":"created","issue":{"url":"https://api.github.com/repos/testomato/minicrawler/issues/2","labels_url":"https://api.github.com/repos/testomato/minicrawler/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/testomato/minicrawler/issues/2/comments","events_url":"https://api.github.com/repos/testomato/minicrawler/issues/2/events","html_url":"https://github.com/testomato/minicrawler/issues/2","id":123991667,"number":2,"title":"Crawl Example","user":{"login":"mitghi","id":4160350,"avatar_url":"https://avatars.githubusercontent.com/u/4160350?v=3","gravatar_id":"","url":"https://api.github.com/users/mitghi","html_url":"https://github.com/mitghi","followers_url":"https://api.github.com/users/mitghi/followers","following_url":"https://api.github.com/users/mitghi/following{/other_user}","gists_url":"https://api.github.com/users/mitghi/gists{/gist_id}","starred_url":"https://api.github.com/users/mitghi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mitghi/subscriptions","organizations_url":"https://api.github.com/users/mitghi/orgs","repos_url":"https://api.github.com/users/mitghi/repos","events_url":"https://api.github.com/users/mitghi/events{/privacy}","received_events_url":"https://api.github.com/users/mitghi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-12-27T18:42:41Z","updated_at":"2016-01-05T11:46:46Z","closed_at":null,"body":"@pracj3am\r\nHey, Thanks for putting this project on Github. It is very interesting and i am glad i found it.\r\n\r\nI have tried the example provided in README.md and also minicrawler executable in order to check the program but it seems that only first page is fetched, then program exits. \r\n\r\nIs that example correct ? It does not extracts urls to crawl. \r\n\r\nThanks"},"comment":{"url":"https://api.github.com/repos/testomato/minicrawler/issues/comments/168979359","html_url":"https://github.com/testomato/minicrawler/issues/2#issuecomment-168979359","issue_url":"https://api.github.com/repos/testomato/minicrawler/issues/2","id":168979359,"user":{"login":"pracj3am","id":191928,"avatar_url":"https://avatars.githubusercontent.com/u/191928?v=3","gravatar_id":"","url":"https://api.github.com/users/pracj3am","html_url":"https://github.com/pracj3am","followers_url":"https://api.github.com/users/pracj3am/followers","following_url":"https://api.github.com/users/pracj3am/following{/other_user}","gists_url":"https://api.github.com/users/pracj3am/gists{/gist_id}","starred_url":"https://api.github.com/users/pracj3am/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pracj3am/subscriptions","organizations_url":"https://api.github.com/users/pracj3am/orgs","repos_url":"https://api.github.com/users/pracj3am/repos","events_url":"https://api.github.com/users/pracj3am/events{/privacy}","received_events_url":"https://api.github.com/users/pracj3am/received_events","type":"User","site_admin":false},"created_at":"2016-01-05T11:46:46Z","updated_at":"2016-01-05T11:46:46Z","body":"I see what you mean now. Despite its name minicrawler does not extract links from fetched content, only downloads content from provided urls. So crawling a website you have to implement yourself.\r\n\r\nI would do it like this\r\n```c\r\nvoid onfinish(mcrawler_url *url, void *arg) {\r\n   mcrawler_url ***urls = (mcrawler_url ***)arg;\r\n   mcrawler_url **found_links;\r\n   find_links_in_content(found_links, url->buf, url->bufp);\r\n   add_found_links_to_urls(urls, found_links)\r\n}\r\n   \r\nvoid main() {\r\n...\r\n   mcrawler_url **urls, ***next;\r\n   mcrawler_url *start[] = {&start_url, NULL};\r\n   urls = start;\r\n   for (i = 0; i < depth; i++) {\r\n      mcrawler_go(urls, &settings, &onfinish, next);\r\n      urls = *next;\r\n   }\r\n}\r\n```"}}