{"action":"created","issue":{"url":"https://api.github.com/repos/shriphani/pegasus/issues/1","repository_url":"https://api.github.com/repos/shriphani/pegasus","labels_url":"https://api.github.com/repos/shriphani/pegasus/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/shriphani/pegasus/issues/1/comments","events_url":"https://api.github.com/repos/shriphani/pegasus/issues/1/events","html_url":"https://github.com/shriphani/pegasus/issues/1","id":128681111,"number":1,"title":"Support for crawlers other than clj-http?","user":{"login":"dhruvbhatia","id":1193022,"avatar_url":"https://avatars.githubusercontent.com/u/1193022?v=3","gravatar_id":"","url":"https://api.github.com/users/dhruvbhatia","html_url":"https://github.com/dhruvbhatia","followers_url":"https://api.github.com/users/dhruvbhatia/followers","following_url":"https://api.github.com/users/dhruvbhatia/following{/other_user}","gists_url":"https://api.github.com/users/dhruvbhatia/gists{/gist_id}","starred_url":"https://api.github.com/users/dhruvbhatia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhruvbhatia/subscriptions","organizations_url":"https://api.github.com/users/dhruvbhatia/orgs","repos_url":"https://api.github.com/users/dhruvbhatia/repos","events_url":"https://api.github.com/users/dhruvbhatia/events{/privacy}","received_events_url":"https://api.github.com/users/dhruvbhatia/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-01-26T00:18:23Z","updated_at":"2016-05-25T15:56:26Z","closed_at":null,"body":"@shriphani Awesome project!\r\n\r\nI noticed you're using `clj-http` to [pull page content](https://github.com/shriphani/pegasus/blob/master/src/pegasus/queue.clj#L18), but what are your thoughts on making this component configurable, so that one can use headless browsers?\r\n\r\nMore and more sites are inserting their data dynamically these days, so `clj-http` is fairly limited given it can't execute JS to \"see\" this content.\r\n\r\nI've had luck with using selenium's standalone-chrome/standalone-firefox and commanding it via [clj-webdriver](https://github.com/semperos/clj-webdriver).\r\n\r\nWould this be something you'd be interested in supporting in pegasus?\r\nIt'll obviously increase the complexity of your project, but I'm thinking the default case could still use `clj-http` while users that require a real browser / JS execution could hook in something headless with `clj-webdriver`.\r\n\r\nLet me know, I'm happy to try and put together a PR."},"comment":{"url":"https://api.github.com/repos/shriphani/pegasus/issues/comments/221621577","html_url":"https://github.com/shriphani/pegasus/issues/1#issuecomment-221621577","issue_url":"https://api.github.com/repos/shriphani/pegasus/issues/1","id":221621577,"user":{"login":"hollingsworthd","id":5241490,"avatar_url":"https://avatars.githubusercontent.com/u/5241490?v=3","gravatar_id":"","url":"https://api.github.com/users/hollingsworthd","html_url":"https://github.com/hollingsworthd","followers_url":"https://api.github.com/users/hollingsworthd/followers","following_url":"https://api.github.com/users/hollingsworthd/following{/other_user}","gists_url":"https://api.github.com/users/hollingsworthd/gists{/gist_id}","starred_url":"https://api.github.com/users/hollingsworthd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hollingsworthd/subscriptions","organizations_url":"https://api.github.com/users/hollingsworthd/orgs","repos_url":"https://api.github.com/users/hollingsworthd/repos","events_url":"https://api.github.com/users/hollingsworthd/events{/privacy}","received_events_url":"https://api.github.com/users/hollingsworthd/received_events","type":"User","site_admin":false},"created_at":"2016-05-25T15:56:26Z","updated_at":"2016-05-25T15:56:26Z","body":"@dhruvbhatia glad it works. Yes, it uses whatever webkit is in Java. They periodically update it. Last time it was updated I think was for Java 8u60: https://bugs.openjdk.java.net/browse/JDK-8093336\r\n\r\nAnother caveat is that some Linux distros separate Java and JavaFX into different packages so they both need to be installed (instructions in the project readme)."}}
{"actor":{"display_login":"hollingsworthd"}}