{"action":"created","issue":{"url":"https://api.github.com/repos/zrashwani/arachnid/issues/11","repository_url":"https://api.github.com/repos/zrashwani/arachnid","labels_url":"https://api.github.com/repos/zrashwani/arachnid/issues/11/labels{/name}","comments_url":"https://api.github.com/repos/zrashwani/arachnid/issues/11/comments","events_url":"https://api.github.com/repos/zrashwani/arachnid/issues/11/events","html_url":"https://github.com/zrashwani/arachnid/issues/11","id":121220324,"node_id":"MDU6SXNzdWUxMjEyMjAzMjQ=","number":11,"title":"Timeout configuration for Goutte client","user":{"login":"ollietreend","id":7735945,"node_id":"MDQ6VXNlcjc3MzU5NDU=","avatar_url":"https://avatars1.githubusercontent.com/u/7735945?v=4","gravatar_id":"","url":"https://api.github.com/users/ollietreend","html_url":"https://github.com/ollietreend","followers_url":"https://api.github.com/users/ollietreend/followers","following_url":"https://api.github.com/users/ollietreend/following{/other_user}","gists_url":"https://api.github.com/users/ollietreend/gists{/gist_id}","starred_url":"https://api.github.com/users/ollietreend/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ollietreend/subscriptions","organizations_url":"https://api.github.com/users/ollietreend/orgs","repos_url":"https://api.github.com/users/ollietreend/repos","events_url":"https://api.github.com/users/ollietreend/events{/privacy}","received_events_url":"https://api.github.com/users/ollietreend/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2015-12-09T11:55:17Z","updated_at":"2018-12-05T08:33:34Z","closed_at":"2016-12-27T22:16:28Z","author_association":"NONE","body":"It's not currently possible to configure a timeout for the Guzzle client which is used to make HTTP requests when spidering a site. Without a default, Guzzle defaults to 0 timeout – i.e. it'll wait indefinitely until a response is received. (Which arguably isn't a sensible default anyway.)\n\nI'm trying to spider a site which contains a link to a dead server. Requests to the URL never timeout, meaning the spider process gets stuck on this URL and never proceeds.\n\nThe timeout is configured when constructing a new Guzzle client, which is currently done in `Arachnid\\Crawler::getScrapClient()`:\n\n``` php\nprotected function getScrapClient()\n{\n    $client = new GoutteClient();\n    $client->followRedirects();\n\n    $guzzleClient = new \\GuzzleHttp\\Client(array(\n        'curl' => array(\n            CURLOPT_SSL_VERIFYHOST => false,\n            CURLOPT_SSL_VERIFYPEER => false,\n        ),\n    ));\n    $client->setClient($guzzleClient);\n\n    return $client;\n}\n```\n\nIt would be really helpful if a timeout was configured here. To do that, all we need to do is change the configuration array which is passed to the Guzzle client constructor method:\n\n``` php\n$guzzleClient = new \\GuzzleHttp\\Client(array(\n    'curl' => array(\n        CURLOPT_SSL_VERIFYHOST => false,\n        CURLOPT_SSL_VERIFYPEER => false,\n    ),\n    'timeout' => 30,\n    'connect_timeout' => 30,\n));\n```\n\nI think a sensible default would be a 30 second timeout, but it would be great to have that configurable. That could either be an additional parameter in the constructor method, or alternatively an object property which can be changed.\n\nIn fact – it might make sense to allow us to add **anything** to the Guzzle constructor configuration. Perhaps again by means of a class property or constructor parameter whereby we can pass in an array of configuration options. This could be useful when configuring other client options, for example HTTP authentication:\n\n``` php\n$crawler = new Crawler($url, 3, array(\n    'timeout' => 5,\n    'connect_timeout' => 5,\n    'auth' => array('username', 'password'),\n));\n```\n\nThoughts? I'd be happy to put together a PR for this, provided we can get some agreement on how this should be configured (class constructor, public property, static property, getter/setter, etc.)\n"},"comment":{"url":"https://api.github.com/repos/zrashwani/arachnid/issues/comments/444401790","html_url":"https://github.com/zrashwani/arachnid/issues/11#issuecomment-444401790","issue_url":"https://api.github.com/repos/zrashwani/arachnid/issues/11","id":444401790,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NDQwMTc5MA==","user":{"login":"zrashwani","id":5447528,"node_id":"MDQ6VXNlcjU0NDc1Mjg=","avatar_url":"https://avatars3.githubusercontent.com/u/5447528?v=4","gravatar_id":"","url":"https://api.github.com/users/zrashwani","html_url":"https://github.com/zrashwani","followers_url":"https://api.github.com/users/zrashwani/followers","following_url":"https://api.github.com/users/zrashwani/following{/other_user}","gists_url":"https://api.github.com/users/zrashwani/gists{/gist_id}","starred_url":"https://api.github.com/users/zrashwani/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zrashwani/subscriptions","organizations_url":"https://api.github.com/users/zrashwani/orgs","repos_url":"https://api.github.com/users/zrashwani/repos","events_url":"https://api.github.com/users/zrashwani/events{/privacy}","received_events_url":"https://api.github.com/users/zrashwani/received_events","type":"User","site_admin":false},"created_at":"2018-12-05T08:33:34Z","updated_at":"2018-12-05T08:33:34Z","author_association":"OWNER","body":"I have performed a major change on the package in order to allow two types of scrapper; one is the usual **Goutte** client and the other is headless browser mode - based on **Symfony Panther** that supports javascript rendering;\r\nthat's why settings CrawlerOptions is now different according to each adapter"}}
{"actor":{"display_login":"zrashwani"}}