{"action":"created","issue":{"url":"https://api.github.com/repos/CorfuDB/CorfuDB/issues/141","repository_url":"https://api.github.com/repos/CorfuDB/CorfuDB","labels_url":"https://api.github.com/repos/CorfuDB/CorfuDB/issues/141/labels{/name}","comments_url":"https://api.github.com/repos/CorfuDB/CorfuDB/issues/141/comments","events_url":"https://api.github.com/repos/CorfuDB/CorfuDB/issues/141/events","html_url":"https://github.com/CorfuDB/CorfuDB/issues/141","id":158204786,"number":141,"title":"System crash after adding ~7M entries","user":{"login":"ohadshacham","id":7221711,"avatar_url":"https://avatars.githubusercontent.com/u/7221711?v=3","gravatar_id":"","url":"https://api.github.com/users/ohadshacham","html_url":"https://github.com/ohadshacham","followers_url":"https://api.github.com/users/ohadshacham/followers","following_url":"https://api.github.com/users/ohadshacham/following{/other_user}","gists_url":"https://api.github.com/users/ohadshacham/gists{/gist_id}","starred_url":"https://api.github.com/users/ohadshacham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ohadshacham/subscriptions","organizations_url":"https://api.github.com/users/ohadshacham/orgs","repos_url":"https://api.github.com/users/ohadshacham/repos","events_url":"https://api.github.com/users/ohadshacham/events{/privacy}","received_events_url":"https://api.github.com/users/ohadshacham/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/CorfuDB/CorfuDB/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"no2chem","id":6496984,"avatar_url":"https://avatars.githubusercontent.com/u/6496984?v=3","gravatar_id":"","url":"https://api.github.com/users/no2chem","html_url":"https://github.com/no2chem","followers_url":"https://api.github.com/users/no2chem/followers","following_url":"https://api.github.com/users/no2chem/following{/other_user}","gists_url":"https://api.github.com/users/no2chem/gists{/gist_id}","starred_url":"https://api.github.com/users/no2chem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/no2chem/subscriptions","organizations_url":"https://api.github.com/users/no2chem/orgs","repos_url":"https://api.github.com/users/no2chem/repos","events_url":"https://api.github.com/users/no2chem/events{/privacy}","received_events_url":"https://api.github.com/users/no2chem/received_events","type":"User","site_admin":false},"milestone":null,"comments":9,"created_at":"2016-06-02T18:11:14Z","updated_at":"2016-06-12T03:35:21Z","closed_at":null,"body":"Hi,\r\n\r\nAfter adding ~7M entries, the client crashes with the following exception.\r\nCould you please help?\r\n\r\nThx,\r\nOhad\r\n\r\njava.lang.NullPointerException\r\n\tat org.corfudb.runtime.view.Layout.getLogUnitClient(Layout.java:218)\r\n\tat org.corfudb.runtime.view.ChainReplicationView.write(ChainReplicationView.java:71)\r\n\tat org.corfudb.runtime.view.AddressSpaceView.lambda$write$28(AddressSpaceView.java:142)\r\n\tat org.corfudb.runtime.view.AddressSpaceView$$Lambda$190/1388907830.apply(Unknown Source)\r\n\tat org.corfudb.runtime.view.AbstractView.layoutHelper(AbstractView.java:74)\r\n\tat org.corfudb.runtime.view.AddressSpaceView.write(AddressSpaceView.java:140)\r\n\tat org.corfudb.runtime.view.StreamsView.acquireAndWrite(StreamsView.java:115)\r\n\tat org.corfudb.runtime.view.StreamsView.write(StreamsView.java:81)\r\n\tat com.yahoo.ycsb.db.CorfuClient.insert(CorfuClient.java:181)\r\n\tat com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)\r\n\tat com.yahoo.ycsb.workloads.CoreWorkload.doTransactionInsert(CoreWorkload.java:793)\r\n\tat com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:620)\r\n\tat com.yahoo.ycsb.ClientThread.run(Client.java:329)\r\njava.lang.NullPointerExceptionjava.lang.NullPointerException\r\n\r\n\tat org.corfudb.runtime.view.Layout.getLogUnitClient(Layout.java:218)\r\n\tat org.corfudb.runtime.view.Layout.getLogUnitClient(Layout.java:218)\tat org.corfudb.runtime.view.ChainReplicationView.write(ChainReplicationView.java:71)\r\n\r\n\tat org.corfudb.runtime.view.ChainReplicationView.write(ChainReplicationView.java:71)\r\n\tat org.corfudb.runtime.view.AddressSpaceView.lambda$write$28(AddressSpaceView.java:142)\tat org.corfudb.runtime.view.AddressSpaceView.lambda$write$28(AddressSpaceView.java:142)\r\n\r\n\tat org.corfudb.runtime.view.AddressSpaceView$$Lambda$190/1388907830.apply(Unknown Source)\tat org.corfudb.runtime.view.AddressSpaceView$$Lambda$190/1388907830.apply(Unknown Source)\r\n\r\n\tat org.corfudb.runtime.view.AbstractView.layoutHelper(AbstractView.java:74)\r\n\tat org.corfudb.runtime.view.AbstractView.layoutHelper(AbstractView.java:74)\tat org.corfudb.runtime.view.AddressSpaceView.write(AddressSpaceView.java:140)\r\n\r\n\tat org.corfudb.runtime.view.AddressSpaceView.write(AddressSpaceView.java:140)\tat org.corfudb.runtime.view.StreamsView.acquireAndWrite(StreamsView.java:115)\r\n\r\n\tat org.corfudb.runtime.view.StreamsView.write(StreamsView.java:81)\tat org.corfudb.runtime.view.StreamsView.acquireAndWrite(StreamsView.java:115)\r\n\r\n\tat com.yahoo.ycsb.db.CorfuClient.insert(CorfuClient.java:181)\tat org.corfudb.runtime.view.StreamsView.write(StreamsView.java:81)\r\n\r\n\tat com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)\tat com.yahoo.ycsb.db.CorfuClient.insert(CorfuClient.java:181)\r\n\r\n\tat com.yahoo.ycsb.workloads.CoreWorkload.doTransactionInsert(CoreWorkload.java:793)\r\n\tat com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)\tat com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:620)\r\n\r\n\tat com.yahoo.ycsb.ClientThread.run(Client.java:329)\tat com.yahoo.ycsb.workloads.CoreWorkload.doTransactionInsert(CoreWorkload.java:793)\r\n\r\n\tat com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:620)\r\n\tat com.yahoo.ycsb.ClientThread.run(Client.java:329)\r\njava.lang.NullPointerException\r\n\tat org.corfudb.runtime.view.Layout.getLogUnitClient(Layout.java:218)\r\n\tat org.corfudb.runtime.view.ChainReplicationView.write(ChainReplicationView.java:71)\r\n\tat org.corfudb.runtime.view.AddressSpaceView.lambda$write$28(AddressSpaceView.java:142)\r\n\tat org.corfudb.runtime.view.AddressSpaceView$$Lambda$190/1388907830.apply(Unknown Source)\r\n\tat org.corfudb.runtime.view.AbstractView.layoutHelper(AbstractView.java:74)\r\n\tat org.corfudb.runtime.view.AddressSpaceView.write(AddressSpaceView.java:140)\r\n\tat org.corfudb.runtime.view.StreamsView.acquireAndWrite(StreamsView.java:115)\r\n\tat org.corfudb.runtime.view.StreamsView.write(StreamsView.java:81)\r\n\tat com.yahoo.ycsb.db.CorfuClient.insert(CorfuClient.java:181)\r\n\tat com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)\r\n\tat com.yahoo.ycsb.workloads.CoreWorkload.doTransactionInsert(CoreWorkload.java:793)\r\n\tat com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:620)\r\n\tat com.yahoo.ycsb.ClientThread.run(Client.java:329)\r\njava.lang.NullPointerException\r\n\tat org.corfudb.runtime.view.Layout.getSequencer(Layout.java:120)\r\n\tat org.corfudb.runtime.view.SequencerView.lambda$nextToken$24(SequencerView.java:27)\r\n\tat org.corfudb.runtime.view.SequencerView$$Lambda$155/2004931730.apply(Unknown Source)\r\n\tat org.corfudb.runtime.view.AbstractView.layoutHelper(AbstractView.java:74)\r\n\tat org.corfudb.runtime.view.SequencerView.nextToken(SequencerView.java:27)\r\n\tat org.corfudb.runtime.view.StreamsView.acquireAndWrite(StreamsView.java:101)\r\n\tat org.corfudb.runtime.view.StreamsView.write(StreamsView.java:81)\r\n\tat com.yahoo.ycsb.db.CorfuClient.insert(CorfuClient.java:181)\r\n\tat com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:208)\r\n\tat com.yahoo.ycsb.workloads.CoreWorkload.doTransactionInsert(CoreWorkload.java:793)\r\n"},"comment":{"url":"https://api.github.com/repos/CorfuDB/CorfuDB/issues/comments/225408119","html_url":"https://github.com/CorfuDB/CorfuDB/issues/141#issuecomment-225408119","issue_url":"https://api.github.com/repos/CorfuDB/CorfuDB/issues/141","id":225408119,"user":{"login":"Maithem","id":7818642,"avatar_url":"https://avatars.githubusercontent.com/u/7818642?v=3","gravatar_id":"","url":"https://api.github.com/users/Maithem","html_url":"https://github.com/Maithem","followers_url":"https://api.github.com/users/Maithem/followers","following_url":"https://api.github.com/users/Maithem/following{/other_user}","gists_url":"https://api.github.com/users/Maithem/gists{/gist_id}","starred_url":"https://api.github.com/users/Maithem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Maithem/subscriptions","organizations_url":"https://api.github.com/users/Maithem/orgs","repos_url":"https://api.github.com/users/Maithem/repos","events_url":"https://api.github.com/users/Maithem/events{/privacy}","received_events_url":"https://api.github.com/users/Maithem/received_events","type":"User","site_admin":false},"created_at":"2016-06-12T03:35:21Z","updated_at":"2016-06-12T03:35:21Z","body":"@ohadshacham np\r\n\r\n@no2chem I think we have some concurrency issues in the corfu run-time object, it is not thread safe.\r\nDifferent threads are accessing shared object members that are not declared as volatile, so sometimes garbage values are being accessed (i.e. null).\r\n\r\nI have a fix, but can't write a good test for it. I can't find a good way of writing a reliable test that can detect this or similar concurrency issues. Anyways, below is my output from running\r\n\r\n/bin/ycsb load corfu -P workloads/workloada &> out.log &\r\n\r\n[OVERALL], RunTime(ms), 3969078.0\r\n[OVERALL], Throughput(ops/sec), 2519.476815522396\r\n[CLEANUP], Operations, 1.0\r\n[CLEANUP], AverageLatency(us), 2.0\r\n[CLEANUP], MinLatency(us), 2.0\r\n[CLEANUP], MaxLatency(us), 2.0\r\n[CLEANUP], 95thPercentileLatency(us), 2.0\r\n[CLEANUP], 99thPercentileLatency(us), 2.0\r\n[INSERT], Operations, 1.0E7\r\n[INSERT], AverageLatency(us), 394.6120441\r\n[INSERT], MinLatency(us), 281.0\r\n[INSERT], MaxLatency(us), 1.0018815E7\r\n[INSERT], 95thPercentileLatency(us), 524.0\r\n[INSERT], 99thPercentileLatency(us), 693.0\r\n[INSERT], Return=OK, 10000000\r\n"}}
{"actor":{"display_login":"Maithem"}}