{"action":"created","issue":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/7785","repository_url":"https://api.github.com/repos/hazelcast/hazelcast","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/7785/labels{/name}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/7785/comments","events_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/7785/events","html_url":"https://github.com/hazelcast/hazelcast/issues/7785","id":141862085,"number":7785,"title":"MapStore: Twice the configured write delay in 3.6?","user":{"login":"lukasblu","id":3501420,"avatar_url":"https://avatars.githubusercontent.com/u/3501420?v=3","gravatar_id":"","url":"https://api.github.com/users/lukasblu","html_url":"https://github.com/lukasblu","followers_url":"https://api.github.com/users/lukasblu/followers","following_url":"https://api.github.com/users/lukasblu/following{/other_user}","gists_url":"https://api.github.com/users/lukasblu/gists{/gist_id}","starred_url":"https://api.github.com/users/lukasblu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukasblu/subscriptions","organizations_url":"https://api.github.com/users/lukasblu/orgs","repos_url":"https://api.github.com/users/lukasblu/repos","events_url":"https://api.github.com/users/lukasblu/events{/privacy}","received_events_url":"https://api.github.com/users/lukasblu/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Team:%20Core","name":"Team: Core","color":"109def"},{"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Type:%20Defect","name":"Type: Defect","color":"eb6420"}],"state":"closed","locked":false,"assignee":{"login":"ahmetmircik","id":2960360,"avatar_url":"https://avatars.githubusercontent.com/u/2960360?v=3","gravatar_id":"","url":"https://api.github.com/users/ahmetmircik","html_url":"https://github.com/ahmetmircik","followers_url":"https://api.github.com/users/ahmetmircik/followers","following_url":"https://api.github.com/users/ahmetmircik/following{/other_user}","gists_url":"https://api.github.com/users/ahmetmircik/gists{/gist_id}","starred_url":"https://api.github.com/users/ahmetmircik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ahmetmircik/subscriptions","organizations_url":"https://api.github.com/users/ahmetmircik/orgs","repos_url":"https://api.github.com/users/ahmetmircik/repos","events_url":"https://api.github.com/users/ahmetmircik/events{/privacy}","received_events_url":"https://api.github.com/users/ahmetmircik/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/74","html_url":"https://github.com/hazelcast/hazelcast/milestones/3.6.3","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/74/labels","id":1665511,"number":74,"title":"3.6.3","description":null,"creator":{"login":"sancar","id":570462,"avatar_url":"https://avatars.githubusercontent.com/u/570462?v=3","gravatar_id":"","url":"https://api.github.com/users/sancar","html_url":"https://github.com/sancar","followers_url":"https://api.github.com/users/sancar/followers","following_url":"https://api.github.com/users/sancar/following{/other_user}","gists_url":"https://api.github.com/users/sancar/gists{/gist_id}","starred_url":"https://api.github.com/users/sancar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sancar/subscriptions","organizations_url":"https://api.github.com/users/sancar/orgs","repos_url":"https://api.github.com/users/sancar/repos","events_url":"https://api.github.com/users/sancar/events{/privacy}","received_events_url":"https://api.github.com/users/sancar/received_events","type":"User","site_admin":false},"open_issues":25,"closed_issues":47,"state":"open","created_at":"2016-03-25T09:59:04Z","updated_at":"2016-05-18T13:56:03Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2016-03-18T13:12:11Z","updated_at":"2016-05-18T13:56:03Z","closed_at":"2016-05-18T13:56:03Z","body":"Hi,\r\n\r\nwhile testing 3.6.2-SNAPSHOT, I discovered that the MapStore persists map entries with roughly twice the write delay than what is configured. In 3.5.x this was not the case.\r\n\r\nPlease see the attached testcase which shows the problem.\r\n\r\nThanks and best,\r\nLukas\r\n\r\n```java\r\npackage com.nm.test.hazelcast.mapstore;\r\n\r\nimport com.hazelcast.config.Config;\r\nimport com.hazelcast.config.MapConfig;\r\nimport com.hazelcast.config.MapStoreConfig;\r\nimport com.hazelcast.config.XmlConfigBuilder;\r\nimport com.hazelcast.core.Hazelcast;\r\nimport com.hazelcast.core.HazelcastInstance;\r\nimport com.hazelcast.core.IMap;\r\nimport com.nm.test.hazelcast.TestHazelcast;\r\nimport com.nm.test.hazelcast.utils.RecordingMapStore;\r\nimport com.nm.test.hazelcast.utils.Sleep;\r\nimport org.apache.log4j.BasicConfigurator;\r\nimport org.apache.log4j.Logger;\r\nimport java.util.concurrent.atomic.AtomicInteger;\r\nimport junit.framework.TestCase;\r\n\r\n/**\r\n * A test to ensure the write-behind interval is more or less respected.\r\n */\r\npublic class TestMapStore28 extends TestCase {\r\n\r\n\tprivate static final Logger logger = Logger.getLogger(TestMapStore28.class);\r\n\r\n\tprivate static final String mapName = \"testMap\" + TestMapStore28.class.getSimpleName();\r\n\r\n\t@Override\r\n\tprotected void setUp() throws Exception {\r\n\r\n\t\t// configure logging\r\n\t\tif (!TestHazelcast.loggingInitialized) {\r\n\t\t\tTestHazelcast.loggingInitialized = true;\r\n\t\t\tBasicConfigurator.configure();\r\n\t\t}\r\n\t}\r\n\r\n\tpublic void test1s() throws Exception {\r\n\t\texecute(1);\r\n\t}\r\n\r\n\tpublic void test2s() throws Exception {\r\n\t\texecute(2);\r\n\t}\r\n\r\n\tpublic void test3s() throws Exception {\r\n\t\texecute(3);\r\n\t}\r\n\r\n\tpublic void test4s() throws Exception {\r\n\t\texecute(4);\r\n\t}\r\n\r\n\tpublic void test5s() throws Exception {\r\n\t\texecute(5);\r\n\t}\r\n\r\n\tpublic void test10s() throws Exception {\r\n\t\texecute(10);\r\n\t}\r\n\r\n\tprivate void execute(int writeDelaySeconds) throws Exception {\r\n\r\n\t\t// create hazelcast config\r\n\t\tConfig config = new XmlConfigBuilder().build();\r\n\t\tconfig.setProperty(\"hazelcast.logging.type\", \"log4j\");\r\n\t\tconfig.setProperty(\"hazelcast.version.check.enabled\", \"false\");\r\n\t\tconfig.setProperty(\"hazelcast.phone.home.enabled\", \"false\");\r\n\r\n\t\t// disable multicast for faster startup\r\n\t\tconfig.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);\r\n\r\n\t\t// create map store\r\n\t\tRecordingMapStore store = new RecordingMapStore(false, false);\r\n\r\n\t\t// configure map store\r\n\t\tMapStoreConfig mapStoreConfig = new MapStoreConfig();\r\n\t\tmapStoreConfig.setEnabled(true);\r\n\t\tmapStoreConfig.setWriteDelaySeconds(writeDelaySeconds);\r\n\t\tmapStoreConfig.setClassName(null);\r\n\t\tmapStoreConfig.setImplementation(store);\r\n\t\tMapConfig mapConfig = config.getMapConfig(mapName);\r\n\t\tmapConfig.setMapStoreConfig(mapStoreConfig);\r\n\r\n\t\t// start hazelcast instance\r\n\t\tHazelcastInstance hcInstance = Hazelcast.newHazelcastInstance(config);\r\n\r\n\t\t// --------------------------------------------------------------- {5s}\r\n\r\n\t\t// execute put\r\n\t\tlogger.info(\"Before put().\");\r\n\t\tfinal String key = \"keyA\";\r\n\t\tIMap<String, String> map = hcInstance.getMap(mapName);\r\n\t\tmap.put(key, \"valueA\");\r\n\t\tlogger.info(\"After put()\");\r\n\r\n\t\t// loop until entry was written\r\n\t\tlogger.info(\"Configured write delay is = \" + writeDelaySeconds);\r\n\t\tint index = 0;\r\n\t\twhile (true) {\r\n\r\n\t\t\t// get store count\r\n\t\t\tAtomicInteger storeCountAtomic = store.getStoreCounts().get(key);\r\n\t\t\tint storeCount = storeCountAtomic == null ? 0 : storeCountAtomic.get();\r\n\t\t\tlogger.info(\"Store count after \" + index + \"s = \" + storeCount);\r\n\r\n\t\t\t// decide if continue waiting, good, or fail\r\n\t\t\tif (storeCount == 0) {\r\n\t\t\t\t// for write delay  1s the entry was written after ~ 1.5s\r\n\t\t\t\t// for write delay  2s the entry was written after ~ 3.5s\r\n\t\t\t\t// for write delay  3s the entry was written after ~ 5.5s\r\n\t\t\t\t// for write delay  4s the entry was written after ~ 7.5s\r\n\t\t\t\t// for write delay  5s the entry was written after ~ 9.5s\r\n\t\t\t\t// for write delay 10s the entry was written after ~19.5s\r\n\t\t\t\tif (index > writeDelaySeconds + 2) {\r\n\t\t\t\t\tfail(\"Entry was not written for \" + index + \" seconds.\");\r\n\t\t\t\t}\r\n\t\t\t} else if (storeCount == 1) {\r\n\t\t\t\t// good\r\n\t\t\t\tbreak;\r\n\t\t\t} else {\r\n\t\t\t\tfail(\"Exactly one write expected, but \" + storeCount + \" writes were executed!\");\r\n\t\t\t}\r\n\r\n\t\t\t// sleep and loop\r\n\t\t\tSleep.sleep(1000, false);\r\n\t\t\tindex++;\r\n\t\t}\r\n\r\n\t\t// shutdown hazelcast instance\r\n\t\thcInstance.getLifecycleService().terminate();\r\n\t}\r\n\r\n}\r\n\r\npackage com.nm.test.hazelcast.utils;\r\n\r\nimport com.hazelcast.core.MapStore;\r\nimport org.apache.log4j.Logger;\r\nimport java.util.*;\r\nimport java.util.concurrent.ConcurrentHashMap;\r\nimport java.util.concurrent.atomic.AtomicInteger;\r\n\r\n/**\r\n * A map store which counts the number of load/store/delete operations.\r\n */\r\npublic class RecordingMapStore implements MapStore<String, String> {\r\n\r\n\tprivate static final Logger logger = Logger.getLogger(RecordingMapStore.class);\r\n\r\n\tprivate final boolean warnOnUpdate;\r\n\r\n\tprivate final boolean infoOnLoad;\r\n\r\n\tprivate ConcurrentHashMap<String, String> store = new ConcurrentHashMap<String, String>();\r\n\r\n\t/**\r\n\t * Load counts per key.\r\n\t * <p>\r\n\t * A map from key to its load count.\r\n\t */\r\n\tprivate ConcurrentHashMap<String, AtomicInteger> loadCounts = new ConcurrentHashMap<String, AtomicInteger>();\r\n\r\n\t/**\r\n\t * Store counts per key.\r\n\t * <p>\r\n\t * A map from key to its store count.\r\n\t */\r\n\tprivate ConcurrentHashMap<String, AtomicInteger> storeCounts = new ConcurrentHashMap<String, AtomicInteger>();\r\n\r\n\t/**\r\n\t * Delete counts per key.\r\n\t * <p>\r\n\t * A map from key to its delete count.\r\n\t */\r\n\tprivate ConcurrentHashMap<String, AtomicInteger> deleteCounts = new ConcurrentHashMap<String, AtomicInteger>();\r\n\r\n\t/**\r\n\t * Default constructor.\r\n\t * <p>\r\n\t * This will log a INFO message upon every operation.\r\n\t */\r\n\tpublic RecordingMapStore() {\r\n\t\tthis.warnOnUpdate = false;\r\n\t\tthis.infoOnLoad = true;\r\n\t}\r\n\r\n\t/**\r\n\t * Constructor.\r\n\t * \r\n\t * @param warnOnUpdate if set to true, a WARN message is printed if a single store() call\r\n\t *            resulted in updating a existing value.\r\n\t * @param infoOnLoad if set to true, a INFO message is printed on load().\r\n\t */\r\n\tpublic RecordingMapStore(boolean warnOnUpdate, boolean infoOnLoad) {\r\n\t\tthis.warnOnUpdate = warnOnUpdate;\r\n\t\tthis.infoOnLoad = infoOnLoad;\r\n\t}\r\n\r\n\tpublic ConcurrentHashMap<String, String> getStore() {\r\n\t\treturn store;\r\n\t}\r\n\r\n\tpublic ConcurrentHashMap<String, AtomicInteger> getLoadCounts() {\r\n\t\treturn loadCounts;\r\n\t}\r\n\r\n\tpublic int getTotalLoadCounts() {\r\n\t\treturn getTotalCount(loadCounts);\r\n\t}\r\n\r\n\tpublic ConcurrentHashMap<String, AtomicInteger> getStoreCounts() {\r\n\t\treturn storeCounts;\r\n\t}\r\n\r\n\tpublic int getTotalStoreCounts() {\r\n\t\treturn getTotalCount(storeCounts);\r\n\t}\r\n\r\n\tpublic ConcurrentHashMap<String, AtomicInteger> getDeleteCounts() {\r\n\t\treturn deleteCounts;\r\n\t}\r\n\r\n\tpublic int getTotalDeleteCounts() {\r\n\t\treturn getTotalCount(deleteCounts);\r\n\t}\r\n\r\n\t@Override\r\n\tpublic String load(String key) {\r\n\t\tif (infoOnLoad) {\r\n\t\t\tlogger.info(\"load(\" + key + \") called.\");\r\n\t\t}\r\n\t\tString result = store.get(key);\r\n\t\tincrementCount(loadCounts, key);\r\n\t\treturn result;\r\n\t}\r\n\r\n\t@Override\r\n\tpublic Map<String, String> loadAll(Collection<String> keys) {\r\n\t\tList<String> keysList = new ArrayList<String>(keys);\r\n\t\tCollections.sort(keysList);\r\n\t\tlogger.info(\"loadAll(\" + keysList + \") called.\");\r\n\t\tMap<String, String> result = new HashMap<String, String>();\r\n\t\tfor (String key : keys) {\r\n\t\t\tString value = store.get(key);\r\n\t\t\tincrementCount(loadCounts, key);\r\n\t\t\tif (value != null) {\r\n\t\t\t\tresult.put(key, value);\r\n\t\t\t}\r\n\t\t}\r\n\t\treturn result;\r\n\t}\r\n\r\n\t@Override\r\n\tpublic Set<String> loadAllKeys() {\r\n\t\tlogger.info(\"loadAllKeys() called.\");\r\n\t\tSet<String> result = new HashSet<String>(store.keySet());\r\n\t\tlogger.info(\"loadAllKeys result = \" + result);\r\n\t\treturn result;\r\n\t}\r\n\r\n\t@Override\r\n\tpublic void store(String key, String value) {\r\n\t\tlogger.info(\"store(\" + key + \") called.\");\r\n\t\tString valuePrev = store.put(key, value);\r\n\t\tif (warnOnUpdate && valuePrev != null) {\r\n\t\t\tlogger.warn(\"- Unexpected Update (operations reordered?): \" + key);\r\n\t\t}\r\n\t\tincrementCount(storeCounts, key);\r\n\t}\r\n\r\n\t@Override\r\n\tpublic void storeAll(Map<String, String> map) {\r\n\t\tTreeSet<String> setSorted = new TreeSet<String>(map.keySet());\r\n\t\tlogger.info(\"storeAll(\" + setSorted + \") called.\");\r\n\t\tstore.putAll(map);\r\n\t\tfor (String key : map.keySet()) {\r\n\t\t\tincrementCount(storeCounts, key);\r\n\t\t}\r\n\t}\r\n\r\n\t@Override\r\n\tpublic void delete(String key) {\r\n\t\tlogger.info(\"delete(\" + key + \") called.\");\r\n\t\tString valuePrev = store.remove(key);\r\n\t\tincrementCount(deleteCounts, key);\r\n\t\tif (valuePrev == null) {\r\n\t\t\tlogger.warn(\"- Unnecessary delete (operations reordered?): \" + key);\r\n\t\t}\r\n\t}\r\n\r\n\t@Override\r\n\tpublic void deleteAll(Collection<String> keys) {\r\n\t\tList<String> keysList = new ArrayList<String>(keys);\r\n\t\tCollections.sort(keysList);\r\n\t\tlogger.info(\"deleteAll(\" + keysList + \") called.\");\r\n\t\tfor (String key : keys) {\r\n\t\t\tString valuePrev = store.remove(key);\r\n\t\t\tincrementCount(deleteCounts, key);\r\n\t\t\tif (valuePrev == null) {\r\n\t\t\t\tlogger.warn(\"- Unnecessary delete (operations reordered?): \" + key);\r\n\t\t\t}\r\n\t\t}\r\n\t}\r\n\r\n\t// -------------------------------------------------------- private methods\r\n\r\n\t/**\r\n\t * Increment count for given key in given map.\r\n\t * \r\n\t * @param counts map to count.\r\n\t * @param key key to increment count for.\r\n\t */\r\n\tprivate void incrementCount(ConcurrentHashMap<String, AtomicInteger> counts, String key) {\r\n\t\tAtomicInteger count = counts.get(key);\r\n\t\tif (count == null) {\r\n\t\t\tcount = new AtomicInteger(0);\r\n\t\t\tAtomicInteger prev = counts.putIfAbsent(key, count);\r\n\t\t\tif (prev != null) {\r\n\t\t\t\tcount = prev;\r\n\t\t\t}\r\n\t\t}\r\n\t\tcount.incrementAndGet();\r\n\t}\r\n\r\n\t/**\r\n\t * Get sum of all individual key counts from given map.\r\n\t * \r\n\t * @param counts the map to build the sum on.\r\n\t * @return total increment count on given map.\r\n\t */\r\n\tprivate int getTotalCount(ConcurrentHashMap<String, AtomicInteger> counts) {\r\n\t\tint result = 0;\r\n\t\tfor (AtomicInteger value : counts.values()) {\r\n\t\t\tresult += value.get();\r\n\t\t}\r\n\t\treturn result;\r\n\t}\r\n\r\n}\r\n\r\npackage com.nm.test.hazelcast.utils;\r\n\r\nimport org.apache.log4j.Logger;\r\n\r\npublic class Sleep {\r\n\r\n\tprivate static final Logger logger = Logger.getLogger(Sleep.class);\r\n\r\n\t/**\r\n\t * Tries to sleep for the desired number of milliseconds, throwing a\r\n\t * RuntimeException in case the thread is interrupted.\r\n\t * \r\n\t * @param ms Number of milliseconds to sleep\r\n\t * @param log true to log, false otherwise.\r\n\t */\r\n\tpublic static void sleep(long ms, boolean log) {\r\n\t\tif (ms <= 0) {\r\n\t\t\treturn;\r\n\t\t}\r\n\t\ttry {\r\n\t\t\tThread.sleep(ms);\r\n\t\t\tif (log) {\r\n\t\t\t\tlogger.info(\"Slept \" + (ms / 1000) + \"s.\");\r\n\t\t\t}\r\n\t\t} catch (InterruptedException e) {\r\n\t\t\tthrow new RuntimeException(e);\r\n\t\t}\r\n\t}\r\n\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/comments/220033512","html_url":"https://github.com/hazelcast/hazelcast/issues/7785#issuecomment-220033512","issue_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/7785","id":220033512,"user":{"login":"ahmetmircik","id":2960360,"avatar_url":"https://avatars.githubusercontent.com/u/2960360?v=3","gravatar_id":"","url":"https://api.github.com/users/ahmetmircik","html_url":"https://github.com/ahmetmircik","followers_url":"https://api.github.com/users/ahmetmircik/followers","following_url":"https://api.github.com/users/ahmetmircik/following{/other_user}","gists_url":"https://api.github.com/users/ahmetmircik/gists{/gist_id}","starred_url":"https://api.github.com/users/ahmetmircik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ahmetmircik/subscriptions","organizations_url":"https://api.github.com/users/ahmetmircik/orgs","repos_url":"https://api.github.com/users/ahmetmircik/repos","events_url":"https://api.github.com/users/ahmetmircik/events{/privacy}","received_events_url":"https://api.github.com/users/ahmetmircik/received_events","type":"User","site_admin":false},"created_at":"2016-05-18T13:56:03Z","updated_at":"2016-05-18T13:56:03Z","body":"https://github.com/hazelcast/hazelcast/pull/8190 should solve this issue."}}