{"action":"created","issue":{"url":"https://api.github.com/repos/rackerlabs/blueflood/issues/748","repository_url":"https://api.github.com/repos/rackerlabs/blueflood","labels_url":"https://api.github.com/repos/rackerlabs/blueflood/issues/748/labels{/name}","comments_url":"https://api.github.com/repos/rackerlabs/blueflood/issues/748/comments","events_url":"https://api.github.com/repos/rackerlabs/blueflood/issues/748/events","html_url":"https://github.com/rackerlabs/blueflood/issues/748","id":185525045,"number":748,"title":"Blueflood rollups issue","user":{"login":"ViswanathSRIRAM","id":21371966,"avatar_url":"https://avatars.githubusercontent.com/u/21371966?v=3","gravatar_id":"","url":"https://api.github.com/users/ViswanathSRIRAM","html_url":"https://github.com/ViswanathSRIRAM","followers_url":"https://api.github.com/users/ViswanathSRIRAM/followers","following_url":"https://api.github.com/users/ViswanathSRIRAM/following{/other_user}","gists_url":"https://api.github.com/users/ViswanathSRIRAM/gists{/gist_id}","starred_url":"https://api.github.com/users/ViswanathSRIRAM/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ViswanathSRIRAM/subscriptions","organizations_url":"https://api.github.com/users/ViswanathSRIRAM/orgs","repos_url":"https://api.github.com/users/ViswanathSRIRAM/repos","events_url":"https://api.github.com/users/ViswanathSRIRAM/events{/privacy}","received_events_url":"https://api.github.com/users/ViswanathSRIRAM/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-10-26T22:16:01Z","updated_at":"2016-11-01T20:07:41Z","closed_at":null,"body":"Hi Team,\n\nWe observed that we are missing roll up data in our test environment. Further digging led to below conclusion. Want to share this to validate my findings. \n\nFew things about our environment: We ingest data every 10 sec and the default TTL is set to 2 days.\n\nThe metrics were missing in metrics_locator so the corresponding rollups were not  happening. \n\nMy observation of this behavior: \n\n_**AstyanaxWriter.java:**_\n    _private static final Cache<String, Boolean> insertedLocators = CacheBuilder.newBuilder().expireAfterAccess(10,\n            TimeUnit.MINUTES).concurrencyLevel(16).build();_\n\nAccording to the above code, the cache built by blueflood ingest  is expiring in 10 minutes from the time of access. Since we write the metric for every 10 secs the cache renews every 10secs for those metrics and never expires. \n\nMetrics_locator CF has TTL of 7 days set due to the following code when writing to this CF.\n_public interface TenantTtlProvider {\n    public static final int LOCATOR_TTL = 604800;  \n..\n}_\n\nFor rollup metrics_locator will play a keyrole.\nAfter 7 days from the time of inserting the metric to metrics_locator the locator data expires but the cache never expires(since we are accessing the metrics for 10 secs).\nSo the subsequent rollups were missing from the 8th day.\n\nUntil we further restart the application the data wouldn't come in to locator again in our case.\n\nWe fixed this by changing the **expireAfterAccess** to **expireAfterWrite** with TTL of 2 days and monitored for few days. We see this is working for our case now.\n\nWe are validating now the same with the below files.\n_blueflood-core/src/main/java/com/rackspacecloud/blueflood/utils/Util.java \nblueflood-core/src/main/java/com/rackspacecloud/blueflood/io/AbstractMetricsRW.java_ \n\nPlease share your feedback if you see any concerns in the above approach.\n"},"comment":{"url":"https://api.github.com/repos/rackerlabs/blueflood/issues/comments/257680028","html_url":"https://github.com/rackerlabs/blueflood/issues/748#issuecomment-257680028","issue_url":"https://api.github.com/repos/rackerlabs/blueflood/issues/748","id":257680028,"user":{"login":"ChandraAddala","id":1525859,"avatar_url":"https://avatars.githubusercontent.com/u/1525859?v=3","gravatar_id":"","url":"https://api.github.com/users/ChandraAddala","html_url":"https://github.com/ChandraAddala","followers_url":"https://api.github.com/users/ChandraAddala/followers","following_url":"https://api.github.com/users/ChandraAddala/following{/other_user}","gists_url":"https://api.github.com/users/ChandraAddala/gists{/gist_id}","starred_url":"https://api.github.com/users/ChandraAddala/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ChandraAddala/subscriptions","organizations_url":"https://api.github.com/users/ChandraAddala/orgs","repos_url":"https://api.github.com/users/ChandraAddala/repos","events_url":"https://api.github.com/users/ChandraAddala/events{/privacy}","received_events_url":"https://api.github.com/users/ChandraAddala/received_events","type":"User","site_admin":false},"created_at":"2016-11-01T20:07:41Z","updated_at":"2016-11-01T20:07:41Z","body":"@ViswanathSRIRAM Thanks for the detailed write up. I was able to look into the issue. I submitted the below PR to take care of the issue you had identified.\r\n\r\nhttps://github.com/rackerlabs/blueflood/pull/750\r\n\r\nAs you can notice from the above PR, I am using a combination of `expireAfterAccess` and `expireAfterWrite` which I think would address all the scenarios. Pls let me know what you think. "}}
{"actor":{"display_login":"ChandraAddala"}}