{"action":"created","issue":{"url":"https://api.github.com/repos/CartoDB/CartoDB-SQL-API/issues/496","repository_url":"https://api.github.com/repos/CartoDB/CartoDB-SQL-API","labels_url":"https://api.github.com/repos/CartoDB/CartoDB-SQL-API/issues/496/labels{/name}","comments_url":"https://api.github.com/repos/CartoDB/CartoDB-SQL-API/issues/496/comments","events_url":"https://api.github.com/repos/CartoDB/CartoDB-SQL-API/issues/496/events","html_url":"https://github.com/CartoDB/CartoDB-SQL-API/issues/496","id":328103274,"node_id":"MDU6SXNzdWUzMjgxMDMyNzQ=","number":496,"title":"COPYTO stream performance reseach","user":{"login":"oleurud","id":421145,"node_id":"MDQ6VXNlcjQyMTE0NQ==","avatar_url":"https://avatars3.githubusercontent.com/u/421145?v=4","gravatar_id":"","url":"https://api.github.com/users/oleurud","html_url":"https://github.com/oleurud","followers_url":"https://api.github.com/users/oleurud/followers","following_url":"https://api.github.com/users/oleurud/following{/other_user}","gists_url":"https://api.github.com/users/oleurud/gists{/gist_id}","starred_url":"https://api.github.com/users/oleurud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oleurud/subscriptions","organizations_url":"https://api.github.com/users/oleurud/orgs","repos_url":"https://api.github.com/users/oleurud/repos","events_url":"https://api.github.com/users/oleurud/events{/privacy}","received_events_url":"https://api.github.com/users/oleurud/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"oleurud","id":421145,"node_id":"MDQ6VXNlcjQyMTE0NQ==","avatar_url":"https://avatars3.githubusercontent.com/u/421145?v=4","gravatar_id":"","url":"https://api.github.com/users/oleurud","html_url":"https://github.com/oleurud","followers_url":"https://api.github.com/users/oleurud/followers","following_url":"https://api.github.com/users/oleurud/following{/other_user}","gists_url":"https://api.github.com/users/oleurud/gists{/gist_id}","starred_url":"https://api.github.com/users/oleurud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oleurud/subscriptions","organizations_url":"https://api.github.com/users/oleurud/orgs","repos_url":"https://api.github.com/users/oleurud/repos","events_url":"https://api.github.com/users/oleurud/events{/privacy}","received_events_url":"https://api.github.com/users/oleurud/received_events","type":"User","site_admin":false},"assignees":[{"login":"oleurud","id":421145,"node_id":"MDQ6VXNlcjQyMTE0NQ==","avatar_url":"https://avatars3.githubusercontent.com/u/421145?v=4","gravatar_id":"","url":"https://api.github.com/users/oleurud","html_url":"https://github.com/oleurud","followers_url":"https://api.github.com/users/oleurud/followers","following_url":"https://api.github.com/users/oleurud/following{/other_user}","gists_url":"https://api.github.com/users/oleurud/gists{/gist_id}","starred_url":"https://api.github.com/users/oleurud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oleurud/subscriptions","organizations_url":"https://api.github.com/users/oleurud/orgs","repos_url":"https://api.github.com/users/oleurud/repos","events_url":"https://api.github.com/users/oleurud/events{/privacy}","received_events_url":"https://api.github.com/users/oleurud/received_events","type":"User","site_admin":false}],"milestone":null,"comments":11,"created_at":"2018-05-31T11:36:56Z","updated_at":"2018-06-11T11:01:37Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"In the COPY feature, we are using the `pg-copy-streams` npm module. In the development tests, we have seen a big CPU usage in nodejs servers when the user makes a COPY TO request.\r\n\r\nAfter a first approach, we have found a theory: `pg-copy-streams` is pretty asymmetric in the way it manages chunks, fixed size in FROM, row-size in TO. So, in the COPY TO the size is dynamic (a row could have a Polygon with 4 corners or with 4.000 corners)\r\n\r\nWe need to investigate:\r\n- do we need work with a row-size?\r\n- could we send directly from postgres to output?"},"comment":{"url":"https://api.github.com/repos/CartoDB/CartoDB-SQL-API/issues/comments/396205130","html_url":"https://github.com/CartoDB/CartoDB-SQL-API/issues/496#issuecomment-396205130","issue_url":"https://api.github.com/repos/CartoDB/CartoDB-SQL-API/issues/496","id":396205130,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NjIwNTEzMA==","user":{"login":"rafatower","id":1914276,"node_id":"MDQ6VXNlcjE5MTQyNzY=","avatar_url":"https://avatars1.githubusercontent.com/u/1914276?v=4","gravatar_id":"","url":"https://api.github.com/users/rafatower","html_url":"https://github.com/rafatower","followers_url":"https://api.github.com/users/rafatower/followers","following_url":"https://api.github.com/users/rafatower/following{/other_user}","gists_url":"https://api.github.com/users/rafatower/gists{/gist_id}","starred_url":"https://api.github.com/users/rafatower/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rafatower/subscriptions","organizations_url":"https://api.github.com/users/rafatower/orgs","repos_url":"https://api.github.com/users/rafatower/repos","events_url":"https://api.github.com/users/rafatower/events{/privacy}","received_events_url":"https://api.github.com/users/rafatower/received_events","type":"User","site_admin":false},"created_at":"2018-06-11T11:01:37Z","updated_at":"2018-06-11T11:01:37Z","author_association":"MEMBER","body":"Repeating the tests above, after the fix:\r\n```\r\n$ time ./export-ukpostcodes.sh > uk_postcodes.out.csv\r\nreal    0m10.332s\r\nuser    0m0.129s\r\nsys     0m2.026s\r\n```\r\n\r\n```\r\n$ wrk http://development.localhost.lan:8080/ --latency -s simple_query.lua -d 10 -t 4                                                                                                                                                 \r\nRunning 10s test @ http://development.localhost.lan:8080/\r\n  4 threads and 10 connections\r\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\r\n    Latency     5.99ms    3.85ms  45.82ms   85.27%\r\n    Req/Sec   348.14    164.26     0.90k    83.75%\r\n  Latency Distribution\r\n     50%    4.94ms\r\n     75%    7.36ms\r\n     90%    9.74ms\r\n     99%   22.12ms\r\n  13875 requests in 10.01s, 4.35MB read\r\nRequests/sec:   1386.52\r\nTransfer/sec:    445.48K\r\n```\r\n\r\n(I reduced the time to 10 to make sure of the overlap of the two loads"}}
{"actor":{"display_login":"rafatower"}}