{"action":"created","issue":{"url":"https://api.github.com/repos/fluo-io/fluo-recipes/issues/69","repository_url":"https://api.github.com/repos/fluo-io/fluo-recipes","labels_url":"https://api.github.com/repos/fluo-io/fluo-recipes/issues/69/labels{/name}","comments_url":"https://api.github.com/repos/fluo-io/fluo-recipes/issues/69/comments","events_url":"https://api.github.com/repos/fluo-io/fluo-recipes/issues/69/events","html_url":"https://github.com/fluo-io/fluo-recipes/issues/69","id":152819693,"number":69,"title":"Compact transient command has negative impact when processing falls behind","user":{"login":"keith-turner","id":1268739,"avatar_url":"https://avatars.githubusercontent.com/u/1268739?v=3","gravatar_id":"","url":"https://api.github.com/users/keith-turner","html_url":"https://github.com/keith-turner","followers_url":"https://api.github.com/users/keith-turner/followers","following_url":"https://api.github.com/users/keith-turner/following{/other_user}","gists_url":"https://api.github.com/users/keith-turner/gists{/gist_id}","starred_url":"https://api.github.com/users/keith-turner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keith-turner/subscriptions","organizations_url":"https://api.github.com/users/keith-turner/orgs","repos_url":"https://api.github.com/users/keith-turner/repos","events_url":"https://api.github.com/users/keith-turner/events{/privacy}","received_events_url":"https://api.github.com/users/keith-turner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-05-03T16:39:16Z","updated_at":"2016-05-03T17:09:56Z","closed_at":null,"body":"Noticied this on EC2 while running webindex.   I was doing a long run on 20 nodes at 1500 pages/sec.  After 12 hours processing of incoming pages started to fall behind.  This caused data to build up in CFM and export queues that was not processed.   I had the compact transient command running every 10 mins.  However because the CFM and export queues were not being processed, these compactions were taking a long a time and not getting rid of any data.  More importantly the compaction were stealing resources from the system when things were already in a bad state.\r\n\r\nMaybe the command could attempt to scan transient ranges to see how much data would be dropped.  Could possibly use new sampling feature in Accumulo 1.8.0."},"comment":{"url":"https://api.github.com/repos/fluo-io/fluo-recipes/issues/comments/216598359","html_url":"https://github.com/fluo-io/fluo-recipes/issues/69#issuecomment-216598359","issue_url":"https://api.github.com/repos/fluo-io/fluo-recipes/issues/69","id":216598359,"user":{"login":"keith-turner","id":1268739,"avatar_url":"https://avatars.githubusercontent.com/u/1268739?v=3","gravatar_id":"","url":"https://api.github.com/users/keith-turner","html_url":"https://github.com/keith-turner","followers_url":"https://api.github.com/users/keith-turner/followers","following_url":"https://api.github.com/users/keith-turner/following{/other_user}","gists_url":"https://api.github.com/users/keith-turner/gists{/gist_id}","starred_url":"https://api.github.com/users/keith-turner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keith-turner/subscriptions","organizations_url":"https://api.github.com/users/keith-turner/orgs","repos_url":"https://api.github.com/users/keith-turner/repos","events_url":"https://api.github.com/users/keith-turner/events{/privacy}","received_events_url":"https://api.github.com/users/keith-turner/received_events","type":"User","site_admin":false},"created_at":"2016-05-03T17:09:56Z","updated_at":"2016-05-03T17:09:56Z","body":"The CFM for computing inbound link counts had ~100 million backuped links to process."}}