{"action":"created","issue":{"url":"https://api.github.com/repos/orientechnologies/orientdb/issues/5821","repository_url":"https://api.github.com/repos/orientechnologies/orientdb","labels_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/5821/labels{/name}","comments_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/5821/comments","events_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/5821/events","html_url":"https://github.com/orientechnologies/orientdb/issues/5821","id":140308245,"number":5821,"title":"Server deadlock with multiple concurrent binary clients","user":{"login":"gerdhub","id":17478826,"avatar_url":"https://avatars.githubusercontent.com/u/17478826?v=3","gravatar_id":"","url":"https://api.github.com/users/gerdhub","html_url":"https://github.com/gerdhub","followers_url":"https://api.github.com/users/gerdhub/followers","following_url":"https://api.github.com/users/gerdhub/following{/other_user}","gists_url":"https://api.github.com/users/gerdhub/gists{/gist_id}","starred_url":"https://api.github.com/users/gerdhub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gerdhub/subscriptions","organizations_url":"https://api.github.com/users/gerdhub/orgs","repos_url":"https://api.github.com/users/gerdhub/repos","events_url":"https://api.github.com/users/gerdhub/events{/privacy}","received_events_url":"https://api.github.com/users/gerdhub/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/orientechnologies/orientdb/labels/bug","name":"bug","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"laa","id":105066,"avatar_url":"https://avatars.githubusercontent.com/u/105066?v=3","gravatar_id":"","url":"https://api.github.com/users/laa","html_url":"https://github.com/laa","followers_url":"https://api.github.com/users/laa/followers","following_url":"https://api.github.com/users/laa/following{/other_user}","gists_url":"https://api.github.com/users/laa/gists{/gist_id}","starred_url":"https://api.github.com/users/laa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laa/subscriptions","organizations_url":"https://api.github.com/users/laa/orgs","repos_url":"https://api.github.com/users/laa/repos","events_url":"https://api.github.com/users/laa/events{/privacy}","received_events_url":"https://api.github.com/users/laa/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/orientechnologies/orientdb/milestones/64","html_url":"https://github.com/orientechnologies/orientdb/milestones/2.1.x%20(next%20hotfix)","labels_url":"https://api.github.com/repos/orientechnologies/orientdb/milestones/64/labels","id":1282180,"number":64,"title":"2.1.x (next hotfix)","description":"","creator":{"login":"lvca","id":312606,"avatar_url":"https://avatars.githubusercontent.com/u/312606?v=3","gravatar_id":"","url":"https://api.github.com/users/lvca","html_url":"https://github.com/lvca","followers_url":"https://api.github.com/users/lvca/followers","following_url":"https://api.github.com/users/lvca/following{/other_user}","gists_url":"https://api.github.com/users/lvca/gists{/gist_id}","starred_url":"https://api.github.com/users/lvca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvca/subscriptions","organizations_url":"https://api.github.com/users/lvca/orgs","repos_url":"https://api.github.com/users/lvca/repos","events_url":"https://api.github.com/users/lvca/events{/privacy}","received_events_url":"https://api.github.com/users/lvca/received_events","type":"User","site_admin":false},"open_issues":160,"closed_issues":2,"state":"open","created_at":"2015-08-31T15:18:49Z","updated_at":"2016-03-30T13:31:40Z","due_on":null,"closed_at":null},"comments":23,"created_at":"2016-03-11T22:30:14Z","updated_at":"2016-03-30T17:17:53Z","closed_at":null,"body":"Using version 2.1.12, I have multiple concurrent threads, each using its own graph instance, reading, creating and updating records in the DB. After about 20.000 cycles, all clients hang. I debugged the server and noticed that_ all _binary clients threads (at the server) where stuck at:\r\n\r\n```\r\n\"OrientDB <- BinaryClient (/10.1.100.22:61097)\"\r\n\tat com.orientechnologies.common.concur.lock.OReadersWriterSpinLock.acquireWriteLock(OReadersWriterSpinLock.java:132)\r\n\tat com.orientechnologies.common.concur.lock.OModificationLock.prohibitModifications(OModificationLock.java:96)\r\n\tat com.orientechnologies.common.concur.lock.OModificationLock.prohibitModifications(OModificationLock.java:84)\r\n\tat com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.synch(OAbstractPaginatedStorage.java:1276)\r\n\tat com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkLowDiskSpaceAndFullCheckpointRequests(OAbstractPaginatedStorage.java:2894)\r\n\tat com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1150)\r\n\tat com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:606)\r\n\tat com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:156)\r\n\tat com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2679)\r\n\tat com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2648)\r\n\tat com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.commit(OrientTransactionalGraph.java:161)\r\n\tat com.orientechnologies.orient.graph.sql.OGraphCommandExecutorSQLFactory.runInConfiguredTxMode(OGraphCommandExecutorSQLFactory.java:262)\r\n\tat com.orientechnologies.orient.graph.sql.OCommandExecutorSQLCreateEdge.execute(OCommandExecutorSQLCreateEdge.java:153)\r\n\tat com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:90)\r\n\tat com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:1530)\r\n\tat com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:1511)\r\n\tat com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:67)\r\n\tat com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1323)\r\n\tat com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:400)\r\n\tat com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:223)\r\n\tat com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)\r\n\r\n```\r\nI tried to reduce the transaction sizes to an absolute minimum without luck. In this situation I even cannot shutdown the server cleanly but have to do a kill -9 and restart to re-enable binary clients to connect to the server.\r\n\r\nThis might relate to #4613"},"comment":{"url":"https://api.github.com/repos/orientechnologies/orientdb/issues/comments/203535573","html_url":"https://github.com/orientechnologies/orientdb/issues/5821#issuecomment-203535573","issue_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/5821","id":203535573,"user":{"login":"taburet","id":1912884,"avatar_url":"https://avatars.githubusercontent.com/u/1912884?v=3","gravatar_id":"","url":"https://api.github.com/users/taburet","html_url":"https://github.com/taburet","followers_url":"https://api.github.com/users/taburet/followers","following_url":"https://api.github.com/users/taburet/following{/other_user}","gists_url":"https://api.github.com/users/taburet/gists{/gist_id}","starred_url":"https://api.github.com/users/taburet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/taburet/subscriptions","organizations_url":"https://api.github.com/users/taburet/orgs","repos_url":"https://api.github.com/users/taburet/repos","events_url":"https://api.github.com/users/taburet/events{/privacy}","received_events_url":"https://api.github.com/users/taburet/received_events","type":"User","site_admin":false},"created_at":"2016-03-30T17:17:53Z","updated_at":"2016-03-30T17:17:53Z","body":"@gerdhub Yep, that's is the *thread* dump, but I'm talking about the *heap* dump :) All I can tell from the thread dumps is that for some reason one of the spin locks has some potentially invalid state so the busy waiting loop spins forever. It will be helpful to see the actual state of the spin lock from a heap dump, that may help to diagnose the problem."}}