{"action":"created","issue":{"url":"https://api.github.com/repos/jeroenooms/curl/issues/79","repository_url":"https://api.github.com/repos/jeroenooms/curl","labels_url":"https://api.github.com/repos/jeroenooms/curl/issues/79/labels{/name}","comments_url":"https://api.github.com/repos/jeroenooms/curl/issues/79/comments","events_url":"https://api.github.com/repos/jeroenooms/curl/issues/79/events","html_url":"https://github.com/jeroenooms/curl/issues/79","id":188639561,"number":79,"title":"Trouble with Connection: keep-alive","user":{"login":"nealrichardson","id":2975928,"avatar_url":"https://avatars.githubusercontent.com/u/2975928?v=3","gravatar_id":"","url":"https://api.github.com/users/nealrichardson","html_url":"https://github.com/nealrichardson","followers_url":"https://api.github.com/users/nealrichardson/followers","following_url":"https://api.github.com/users/nealrichardson/following{/other_user}","gists_url":"https://api.github.com/users/nealrichardson/gists{/gist_id}","starred_url":"https://api.github.com/users/nealrichardson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nealrichardson/subscriptions","organizations_url":"https://api.github.com/users/nealrichardson/orgs","repos_url":"https://api.github.com/users/nealrichardson/repos","events_url":"https://api.github.com/users/nealrichardson/events{/privacy}","received_events_url":"https://api.github.com/users/nealrichardson/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-11-10T22:46:39Z","updated_at":"2016-11-24T12:17:32Z","closed_at":"2016-11-24T12:17:32Z","body":"I've been trying to debug some slower-than-expected response times with a web service, and I think I've isolated it to a failure in the `curl` package's \"keep-alive\" behavior. As I understand it, cURL should respect a `Connection: keep-alive` response header by default. From the command line, that is what I see when I make two requests to the same host:\r\n\r\n```\r\n$ curl -v -w '\\ntime_redirect=%{time_redirect}\\ntime_namelookup=%{time_namelookup}\\ntime_connect=%{time_connect}\\ntime_pretransfer=%{time_pretransfer}\\ntime_starttransfer=%{time_starttransfer}\\ntime_total=%{time_total}\\n\\n' https://httpbin.org/get https://httpbin.org/get\r\n*   Trying 23.22.14.18...\r\n* Connected to httpbin.org (23.22.14.18) port 443 (#0)\r\n* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\r\n* Server certificate: *.httpbin.org\r\n* Server certificate: COMODO RSA Domain Validation Secure Server CA\r\n* Server certificate: COMODO RSA Certification Authority\r\n* Server certificate: AddTrust External CA Root\r\n> GET /get HTTP/1.1\r\n> Host: httpbin.org\r\n> User-Agent: curl/7.43.0\r\n> Accept: */*\r\n>\r\n< HTTP/1.1 200 OK\r\n< Server: nginx\r\n< Date: Thu, 10 Nov 2016 02:43:49 GMT\r\n< Content-Type: application/json\r\n< Content-Length: 185\r\n< Connection: keep-alive\r\n< Access-Control-Allow-Origin: *\r\n< Access-Control-Allow-Credentials: true\r\n<\r\n{\r\n  \"args\": {},\r\n  \"headers\": {\r\n    \"Accept\": \"*/*\",\r\n    \"Host\": \"httpbin.org\",\r\n    \"User-Agent\": \"curl/7.43.0\"\r\n  },\r\n  \"origin\": \"24.7.89.219\",\r\n  \"url\": \"https://httpbin.org/get\"\r\n}\r\n* Connection #0 to host httpbin.org left intact\r\n\r\ntime_redirect=0.000\r\ntime_namelookup=0.005\r\ntime_connect=0.090\r\ntime_pretransfer=0.433\r\ntime_starttransfer=0.517\r\ntime_total=0.517\r\n\r\n* Found bundle for host httpbin.org: 0x7fca2be01390\r\n* Re-using existing connection! (#0) with host httpbin.org\r\n* Connected to httpbin.org (23.22.14.18) port 443 (#0)\r\n> GET /get HTTP/1.1\r\n> Host: httpbin.org\r\n> User-Agent: curl/7.43.0\r\n> Accept: */*\r\n>\r\n< HTTP/1.1 200 OK\r\n< Server: nginx\r\n< Date: Thu, 10 Nov 2016 02:43:49 GMT\r\n< Content-Type: application/json\r\n< Content-Length: 185\r\n< Connection: keep-alive\r\n< Access-Control-Allow-Origin: *\r\n< Access-Control-Allow-Credentials: true\r\n<\r\n{\r\n  \"args\": {},\r\n  \"headers\": {\r\n    \"Accept\": \"*/*\",\r\n    \"Host\": \"httpbin.org\",\r\n    \"User-Agent\": \"curl/7.43.0\"\r\n  },\r\n  \"origin\": \"24.7.89.219\",\r\n  \"url\": \"https://httpbin.org/get\"\r\n}\r\n* Connection #0 to host httpbin.org left intact\r\n\r\ntime_redirect=0.000\r\ntime_namelookup=0.000\r\ntime_connect=0.000\r\ntime_pretransfer=0.000\r\ntime_starttransfer=0.086\r\ntime_total=0.086\r\n```\r\n\r\nNote that in the second request, there is the line `* Re-using existing connection! (#0) with host httpbin.org`, and the \"pretransfer\" time is zero. However, from R using `curl`, following the discussion in https://cran.r-project.org/web/packages/curl/vignettes/intro.html#re-using_handles about reusing handles in order to take advantage of keep-alive, the same behavior doesn't happen:\r\n\r\n```\r\n> h <- new_handle()\r\n> handle_setopt(h, verbose=TRUE)\r\n> req1 <- curl_fetch_memory(\"https://httpbin.org/get\", handle=h)\r\n*   Trying 23.22.14.18...\r\n* Connected to httpbin.org (23.22.14.18) port 443 (#0)\r\n* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\r\n* Server certificate: *.httpbin.org\r\n* Server certificate: COMODO RSA Domain Validation Secure Server CA\r\n* Server certificate: COMODO RSA Certification Authority\r\n* Server certificate: AddTrust External CA Root\r\n> GET /get HTTP/1.1\r\nHost: httpbin.org\r\nUser-Agent: r/curl/jeroen\r\nAccept: */*\r\nAccept-Encoding: gzip, deflate\r\n\r\n< HTTP/1.1 200 OK\r\n< Server: nginx\r\n< Date: Thu, 10 Nov 2016 02:42:15 GMT\r\n< Content-Type: application/json\r\n< Content-Length: 228\r\n< Connection: keep-alive\r\n< Access-Control-Allow-Origin: *\r\n< Access-Control-Allow-Credentials: true\r\n<\r\n* Connection #0 to host httpbin.org left intact\r\n```\r\n\r\nThe response has a `Connection: keep-alive` header, and it says that the connection is left intact, but a subsequent connection doesn't appear to reuse the existing connection:\r\n\r\n```\r\n> req2 <- curl_fetch_memory(\"https://httpbin.org/get\", handle=h)\r\n*   Trying 23.22.14.18...\r\n* Connected to httpbin.org (23.22.14.18) port 443 (#0)\r\n* SSL re-using session ID\r\n* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\r\n* Server certificate: *.httpbin.org\r\n* Server certificate: COMODO RSA Domain Validation Secure Server CA\r\n* Server certificate: COMODO RSA Certification Authority\r\n* Server certificate: AddTrust External CA Root\r\n> GET /get HTTP/1.1\r\nHost: httpbin.org\r\nUser-Agent: r/curl/jeroen\r\nAccept: */*\r\nAccept-Encoding: gzip, deflate\r\n\r\n< HTTP/1.1 200 OK\r\n< Server: nginx\r\n< Date: Thu, 10 Nov 2016 02:42:23 GMT\r\n< Content-Type: application/json\r\n< Content-Length: 228\r\n< Connection: keep-alive\r\n< Access-Control-Allow-Origin: *\r\n< Access-Control-Allow-Credentials: true\r\n<\r\n* Connection #0 to host httpbin.org left intact\r\n```\r\n\r\nMoreover, the timings are pretty much the same between the two requests: pretransfer time doesn't go to zero like it should if keep-alive were happening:\r\n\r\n```\r\n> req1$times\r\n     redirect    namelookup       connect   pretransfer starttransfer\r\n     0.000000      0.005980      0.090432      0.187124      0.384133\r\n        total\r\n     0.384201\r\n> req2$times\r\n     redirect    namelookup       connect   pretransfer starttransfer\r\n     0.000000      0.010981      0.095239      0.190441      0.387021\r\n        total\r\n     0.387104\r\n```\r\n\r\nI have seen keep-alive work with the `curl` package in the past, and I haven't done anything (intentionally) to alter the configuration such that calling libcurl from R wouldn't obey keep-alive. And interestingly, if I do the same exercise with `RCurl` instead, keep-alive works--the `* Re-using existing connection!` message appears, and the second request returns immediately. So there seems to be something different about how `curl` is wrapping libcurl that prevents keep-alive.\r\n\r\n```\r\n> library(RCurl)\r\nLoading required package: bitops\r\n> h <- getCurlHandle()              \r\n> curlSetOpt(curl=h, verbose=TRUE)\r\nAn object of class \"CURLHandle\"\r\nSlot \"ref\":\r\n<pointer: 0x7fe60b03d800>\r\n\r\n> x <- getURL(\"https://httpbin.org/get\", curl=h)\r\n*   Trying 54.175.219.8...\r\n* Connected to httpbin.org (54.175.219.8) port 443 (#0)\r\n* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\r\n* Server certificate: *.httpbin.org\r\n* Server certificate: COMODO RSA Domain Validation Secure Server CA\r\n* Server certificate: COMODO RSA Certification Authority\r\n* Server certificate: AddTrust External CA Root\r\n> GET /get HTTP/1.1\r\nHost: httpbin.org\r\nAccept: */*\r\n\r\n< HTTP/1.1 200 OK\r\n< Server: nginx\r\n< Date: Thu, 10 Nov 2016 06:05:01 GMT\r\n< Content-Type: application/json\r\n< Content-Length: 151\r\n< Connection: keep-alive\r\n< Access-Control-Allow-Origin: *\r\n< Access-Control-Allow-Credentials: true\r\n<\r\n* Connection #0 to host httpbin.org left intact\r\n> x2 <- getURL(\"https://httpbin.org/get\", curl=h)\r\n* Found bundle for host httpbin.org: 0x7fe6094dcc80\r\n* Re-using existing connection! (#0) with host httpbin.org\r\n* Connected to httpbin.org (54.175.219.8) port 443 (#0)\r\n> GET /get HTTP/1.1\r\nHost: httpbin.org\r\nAccept: */*\r\n\r\n< HTTP/1.1 200 OK\r\n< Server: nginx\r\n< Date: Thu, 10 Nov 2016 06:05:09 GMT\r\n< Content-Type: application/json\r\n< Content-Length: 151\r\n< Connection: keep-alive\r\n< Access-Control-Allow-Origin: *\r\n< Access-Control-Allow-Credentials: true\r\n<\r\n* Connection #0 to host httpbin.org left intact\r\n```\r\n\r\nSession/system info from the attempt using the `curl` package:\r\n\r\n```\r\n> sessionInfo()\r\nR Under development (unstable) (2016-08-09 r71070)\r\nPlatform: x86_64-apple-darwin13.4.0 (64-bit)\r\nRunning under: OS X El Capitan 10.11.6\r\n\r\nlocale:\r\n[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8\r\n\r\nattached base packages:\r\n[1] stats     graphics  grDevices utils     datasets  methods   base     \r\n\r\nother attached packages:\r\n[1] curl_2.2\r\n> curl_version()\r\n$version\r\n[1] \"7.43.0\"\r\n\r\n$ssl_version\r\n[1] \"SecureTransport\"\r\n\r\n$libz_version\r\n[1] \"1.2.5\"\r\n\r\n$libssh_version\r\n[1] NA\r\n\r\n$libidn_version\r\n[1] NA\r\n\r\n$host\r\n[1] \"x86_64-apple-darwin15.0\"\r\n\r\n$protocols\r\n [1] \"dict\"   \"file\"   \"ftp\"    \"ftps\"   \"gopher\" \"http\"   \"https\"  \"imap\"  \r\n [9] \"imaps\"  \"ldap\"   \"ldaps\"  \"pop3\"   \"pop3s\"  \"rtsp\"   \"smb\"    \"smbs\"  \r\n[17] \"smtp\"   \"smtps\"  \"telnet\" \"tftp\"  \r\n\r\n$ipv6\r\n[1] TRUE\r\n\r\n$http2\r\n[1] FALSE\r\n```\r\n\r\nAnd sessionInfo with `RCurl`:\r\n\r\n```\r\n> sessionInfo()\r\nR Under development (unstable) (2016-08-09 r71070)\r\nPlatform: x86_64-apple-darwin13.4.0 (64-bit)\r\nRunning under: OS X El Capitan 10.11.6\r\n\r\nlocale:\r\n[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8\r\n\r\nattached base packages:\r\n[1] stats     graphics  grDevices utils     datasets  methods   base     \r\n\r\nother attached packages:\r\n[1] RCurl_1.95-4.8 bitops_1.0-6  \r\n\r\nloaded via a namespace (and not attached):\r\n[1] tools_3.4.0\r\n```\r\n\r\nI'm happy to dig in and debug further, but I'm not sure how to proceed or where to look. My experimenting suggests that the issue is in how `curl` (re)uses handle objects, and it would be worth comparing how `RCurl` does that to look for why keep-alive works there. But maybe there's a configuration set wrong for me locally or something. "},"comment":{"url":"https://api.github.com/repos/jeroenooms/curl/issues/comments/262763447","html_url":"https://github.com/jeroenooms/curl/issues/79#issuecomment-262763447","issue_url":"https://api.github.com/repos/jeroenooms/curl/issues/79","id":262763447,"user":{"login":"jeroenooms","id":216319,"avatar_url":"https://avatars.githubusercontent.com/u/216319?v=3","gravatar_id":"","url":"https://api.github.com/users/jeroenooms","html_url":"https://github.com/jeroenooms","followers_url":"https://api.github.com/users/jeroenooms/followers","following_url":"https://api.github.com/users/jeroenooms/following{/other_user}","gists_url":"https://api.github.com/users/jeroenooms/gists{/gist_id}","starred_url":"https://api.github.com/users/jeroenooms/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeroenooms/subscriptions","organizations_url":"https://api.github.com/users/jeroenooms/orgs","repos_url":"https://api.github.com/users/jeroenooms/repos","events_url":"https://api.github.com/users/jeroenooms/events{/privacy}","received_events_url":"https://api.github.com/users/jeroenooms/received_events","type":"User","site_admin":false},"created_at":"2016-11-24T12:17:32Z","updated_at":"2016-11-24T12:17:32Z","body":"OK this is on CRAN now. Thanks for your detailed bug report, this was very helpful."}}
{"actor":{"display_login":"jeroenooms"}}