{"action":"created","issue":{"url":"https://api.github.com/repos/objectify/objectify/issues/328","repository_url":"https://api.github.com/repos/objectify/objectify","labels_url":"https://api.github.com/repos/objectify/objectify/issues/328/labels{/name}","comments_url":"https://api.github.com/repos/objectify/objectify/issues/328/comments","events_url":"https://api.github.com/repos/objectify/objectify/issues/328/events","html_url":"https://github.com/objectify/objectify/issues/328","id":238857066,"node_id":"MDU6SXNzdWUyMzg4NTcwNjY=","number":328,"title":"Provide a hook so that the application can arbitrarily handle memcache failures","user":{"login":"nbali","id":581927,"node_id":"MDQ6VXNlcjU4MTkyNw==","avatar_url":"https://avatars1.githubusercontent.com/u/581927?v=4","gravatar_id":"","url":"https://api.github.com/users/nbali","html_url":"https://github.com/nbali","followers_url":"https://api.github.com/users/nbali/followers","following_url":"https://api.github.com/users/nbali/following{/other_user}","gists_url":"https://api.github.com/users/nbali/gists{/gist_id}","starred_url":"https://api.github.com/users/nbali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nbali/subscriptions","organizations_url":"https://api.github.com/users/nbali/orgs","repos_url":"https://api.github.com/users/nbali/repos","events_url":"https://api.github.com/users/nbali/events{/privacy}","received_events_url":"https://api.github.com/users/nbali/received_events","type":"User","site_admin":false},"labels":[{"id":6905055,"node_id":"MDU6TGFiZWw2OTA1MDU1","url":"https://api.github.com/repos/objectify/objectify/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-06-27T13:47:49Z","updated_at":"2018-06-21T19:19:51Z","closed_at":null,"author_association":"NONE","body":"First of all let's clarify that we have dedicated memcache with the following stats:\r\nDedicated (20 GB, 200k ops/sec)\r\nHit ratio | 91.21% | 3,417,497,685 hit / 329,223,846 miss\r\nItems in cache | 15,315,614\r\nOldest item age | 3 days 13 hr\r\nTotal cache size | 19.27 GB\r\n\r\nThe issue we encountered is that the memcache contained an outdated state of the entity. To be precise it contains that it's missing, meanwhile it isn't. (Most likely both case happens, it's just this was more serious for us, so we noticed it). Compared to the whole amount of data the occurrence isn't high, but due it being critical business data it's effect is rather serious.\r\n\r\nAnyway after some investigation we suspect it was caused by the lack of memcache update when the entity was first created.\r\n\r\n```java\r\n\tat com.googlecode.objectify.cache.MemcacheServiceRetryProxy.invoke(MemcacheServiceRetryProxy.java:68)\r\n\tat com.sun.proxy.$Proxy10.putAll(Unknown Source)\r\n\tat com.googlecode.objectify.cache.KeyMemcacheService.putAll(KeyMemcacheService.java:91)\r\n\tat com.googlecode.objectify.cache.EntityMemcache.empty(EntityMemcache.java:312)\r\n\tat com.googlecode.objectify.cache.CachingAsyncDatastoreService$5.trigger(CachingAsyncDatastoreService.java:445)\r\n\tat com.googlecode.objectify.cache.TriggerFuture.isDone(TriggerFuture.java:87)\r\n\tat com.googlecode.objectify.cache.TriggerFuture.get(TriggerFuture.java:102)\r\n\tat com.googlecode.objectify.impl.ResultAdapter.now(ResultAdapter.java:34)\r\n\tat com.googlecode.objectify.util.ResultWrapper.translate(ResultWrapper.java:22)\r\n\tat com.googlecode.objectify.util.ResultWrapper.translate(ResultWrapper.java:10)\r\n\tat com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)\r\n\tat com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)\r\n```\r\n\r\nI get the problems with handling memcache thrown exceptions. We can't retry it in ```MemcacheServiceRetryProxy``` till the end of time given that if the memcache is down it might never succeed (during that request I mean). Also throwing random exceptions out-of-the-blue might break application code.\r\n\r\nMy problem is that there is no way to get notified about any failed memcache operation. \"Normal\" memcache operations log it as SEVERE, retried operations as WARNING-WARNING-WARNING-SEVERE, but both eventually returns with null. For get* actions that might be fine, but for anything that updates the data is bad IMO.\r\n\r\n- What I missing here, why would it be bad to use MemcacheServiceRetryProxy for every memcache operation?\r\n- Why isn't there any exponential backoff between the retries?\r\n- Would it be accepted if I would implement a new configuration possibility that would accept an object implementing an interface that would be called upon memcache failures? For example calling it with something (string/enum) that identifies operation we tried to do as one parameter, memcache namespace as another, and the list of keys that we couldn't clear for example as a third parameter, etc?\r\n- Or do you see any better fix/workaround?"},"comment":{"url":"https://api.github.com/repos/objectify/objectify/issues/comments/399214266","html_url":"https://github.com/objectify/objectify/issues/328#issuecomment-399214266","issue_url":"https://api.github.com/repos/objectify/objectify/issues/328","id":399214266,"node_id":"MDEyOklzc3VlQ29tbWVudDM5OTIxNDI2Ng==","user":{"login":"nbali","id":581927,"node_id":"MDQ6VXNlcjU4MTkyNw==","avatar_url":"https://avatars1.githubusercontent.com/u/581927?v=4","gravatar_id":"","url":"https://api.github.com/users/nbali","html_url":"https://github.com/nbali","followers_url":"https://api.github.com/users/nbali/followers","following_url":"https://api.github.com/users/nbali/following{/other_user}","gists_url":"https://api.github.com/users/nbali/gists{/gist_id}","starred_url":"https://api.github.com/users/nbali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nbali/subscriptions","organizations_url":"https://api.github.com/users/nbali/orgs","repos_url":"https://api.github.com/users/nbali/repos","events_url":"https://api.github.com/users/nbali/events{/privacy}","received_events_url":"https://api.github.com/users/nbali/received_events","type":"User","site_admin":false},"created_at":"2018-06-21T19:19:51Z","updated_at":"2018-06-21T19:19:51Z","author_association":"NONE","body":"Btw maybe I don't remember well, but I think they have changed the thrown exception. If anyone have the original stack trace/remembers more accurately than I do, please feel free to correct me.\r\n\r\n```\r\njava.lang.reflect.InvocationTargetException\r\n\tat sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:498)\r\n\tat com.googlecode.objectify.cache.MemcacheServiceRetryProxy.invoke(MemcacheServiceRetryProxy.java:68)\r\n\tat com.sun.proxy.$Proxy34.putAll(Unknown Source)\r\n\tat com.googlecode.objectify.cache.KeyMemcacheService.putAll(KeyMemcacheService.java:91)\r\n\tat com.googlecode.objectify.cache.EntityMemcache.empty(EntityMemcache.java:323)\r\n\tat com.googlecode.objectify.cache.CachingTransaction$1.trigger(CachingTransaction.java:71)\r\n\tat com.googlecode.objectify.cache.TriggerFuture.isDone(TriggerFuture.java:87)\r\n\tat com.googlecode.objectify.cache.TriggerFuture.get(TriggerFuture.java:102)\r\n\tat com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:93)\r\n\tat com.googlecode.objectify.util.FutureHelper.quietGet(FutureHelper.java:29)\r\n\tat com.googlecode.objectify.impl.TransactionImpl.commit(TransactionImpl.java:72)\r\n\tat com.googlecode.objectify.impl.TransactorNo.transactOnce(TransactorNo.java:123)\r\n\tat com.googlecode.objectify.impl.TransactorNo.transactNew(TransactorNo.java:97)\r\n\tat com.googlecode.objectify.impl.TransactorNo.transact(TransactorNo.java:85)\r\n\tat com.googlecode.objectify.impl.ObjectifyImpl.transact(ObjectifyImpl.java:192)\r\n\tat com.googlecode.objectify.impl.ObjectifyImpl.transact(ObjectifyImpl.java:197)\r\nCaused by: com.google.appengine.api.memcache.MemcacheServiceException: Memcache putAll: Unknown exception setting 1 keys\r\n\tat com.google.appengine.api.memcache.MemcacheServiceApiHelper$RpcResponseHandler.handleApiProxyException(MemcacheServiceApiHelper.java:80)\r\n\tat com.google.appengine.api.memcache.AsyncMemcacheServiceImpl$RpcResponseHandlerForPut.handleApiProxyException(AsyncMemcacheServiceImpl.java:370)\r\n\tat com.google.appengine.api.memcache.MemcacheServiceApiHelper$1.absorbParentException(MemcacheServiceApiHelper.java:129)\r\n\tat com.google.appengine.api.utils.FutureWrapper.handleParentException(FutureWrapper.java:56)\r\n\tat com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:95)\r\n\tat com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:93)\r\n\tat com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:28)\r\n\tat com.google.appengine.api.memcache.MemcacheServiceImpl.putAll(MemcacheServiceImpl.java:117)\r\n\t... 66 more\r\nCaused by: com.google.apphosting.api.ApiProxy$CapabilityDisabledException: The API call memcache.Set() is temporarily unavailable: \r\n\tat com.google.apphosting.utils.runtime.ApiProxyUtils.getRpcError(ApiProxyUtils.java:142)\r\n\tat com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.setRpcError(ApiProxyImpl.java:619)\r\n\tat com.google.apphosting.runtime.ApiProxyImpl$AsyncApiFuture.failure(ApiProxyImpl.java:605)\r\n\tat com.google.apphosting.runtime.http.HttpApiHostClient.receivedResponse(HttpApiHostClient.java:266)\r\n\tat com.google.apphosting.runtime.http.JettyHttpApiHostClient$Listener.onComplete(JettyHttpApiHostClient.java:158)\r\n\t... 1 more \r\n```\r\n\r\nAnyway my point being is that this ```CapabilityDisabledException``` is something that should be a clear candidate for this callback hook."}}
{"actor":{"display_login":"nbali"}}