{"action":"created","issue":{"url":"https://api.github.com/repos/CartoDB/cartodb/issues/13874","repository_url":"https://api.github.com/repos/CartoDB/cartodb","labels_url":"https://api.github.com/repos/CartoDB/cartodb/issues/13874/labels{/name}","comments_url":"https://api.github.com/repos/CartoDB/cartodb/issues/13874/comments","events_url":"https://api.github.com/repos/CartoDB/cartodb/issues/13874/events","html_url":"https://github.com/CartoDB/cartodb/issues/13874","id":316689110,"number":13874,"title":"Limit concurrent imports properly","user":{"login":"jgoizueta","id":5909,"avatar_url":"https://avatars3.githubusercontent.com/u/5909?v=4","gravatar_id":"","url":"https://api.github.com/users/jgoizueta","html_url":"https://github.com/jgoizueta","followers_url":"https://api.github.com/users/jgoizueta/followers","following_url":"https://api.github.com/users/jgoizueta/following{/other_user}","gists_url":"https://api.github.com/users/jgoizueta/gists{/gist_id}","starred_url":"https://api.github.com/users/jgoizueta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jgoizueta/subscriptions","organizations_url":"https://api.github.com/users/jgoizueta/orgs","repos_url":"https://api.github.com/users/jgoizueta/repos","events_url":"https://api.github.com/users/jgoizueta/events{/privacy}","received_events_url":"https://api.github.com/users/jgoizueta/received_events","type":"User","site_admin":false},"labels":[{"id":159755870,"url":"https://api.github.com/repos/CartoDB/cartodb/labels/Backend","name":"Backend","color":"d4c5f9","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-23T07:14:04Z","updated_at":"2018-04-24T07:58:42Z","closed_at":null,"author_association":"MEMBER","body":"Our attempts to limit the number of concurrent imports per user are not [very effective](https://github.com/CartoDB/cartodb/blob/82a23d0cac74e5aeda9cb67961ceabdc402819f1/services/platform-limits/lib/importer/user_concurrent_imports_amount.rb#L12-L13\r\n).\r\n\r\nWe use a counter of concurrent imports in redis, but a TTL based on the db timeout is applied:\r\nhttps://github.com/CartoDB/cartodb/blob/82a23d0cac74e5aeda9cb67961ceabdc402819f1/services/platform-limits/lib/importer/user_concurrent_imports_amount.rb#L53-L54\r\n\r\nThe imports can easily take more than this TTL and in that case the counter is lost (so effectively reset to zero) and new imports accepted. We have a bad case recently with long imports being queued until they filled up the disk; in that case the user db timeout was 25 s and as a result the TTL was 1.2*25 = 30 s. The imports took 5-15 min each.\r\n\r\nI think we could either check the actual number of active imports for the user in the db or use redis-based rate limits like those we're using elsewhere.\r\n\r\ncc @javitonino "},"comment":{"url":"https://api.github.com/repos/CartoDB/cartodb/issues/comments/383839607","html_url":"https://github.com/CartoDB/cartodb/issues/13874#issuecomment-383839607","issue_url":"https://api.github.com/repos/CartoDB/cartodb/issues/13874","id":383839607,"user":{"login":"javitonino","id":7569673,"avatar_url":"https://avatars3.githubusercontent.com/u/7569673?v=4","gravatar_id":"","url":"https://api.github.com/users/javitonino","html_url":"https://github.com/javitonino","followers_url":"https://api.github.com/users/javitonino/followers","following_url":"https://api.github.com/users/javitonino/following{/other_user}","gists_url":"https://api.github.com/users/javitonino/gists{/gist_id}","starred_url":"https://api.github.com/users/javitonino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/javitonino/subscriptions","organizations_url":"https://api.github.com/users/javitonino/orgs","repos_url":"https://api.github.com/users/javitonino/repos","events_url":"https://api.github.com/users/javitonino/events{/privacy}","received_events_url":"https://api.github.com/users/javitonino/received_events","type":"User","site_admin":false},"created_at":"2018-04-24T07:58:41Z","updated_at":"2018-04-24T07:58:41Z","author_association":"MEMBER","body":"Can we put this in RT @ramiroaznar?"}}
{"actor":{"display_login":"javitonino"}}