{"action":"created","issue":{"url":"https://api.github.com/repos/eu-emi/canl-java/issues/80","repository_url":"https://api.github.com/repos/eu-emi/canl-java","labels_url":"https://api.github.com/repos/eu-emi/canl-java/issues/80/labels{/name}","comments_url":"https://api.github.com/repos/eu-emi/canl-java/issues/80/comments","events_url":"https://api.github.com/repos/eu-emi/canl-java/issues/80/events","html_url":"https://github.com/eu-emi/canl-java/issues/80","id":122099479,"number":80,"title":"Lock contention on OpensslCertChainValidator#validate","user":{"login":"gbehrmann","id":1125766,"avatar_url":"https://avatars.githubusercontent.com/u/1125766?v=3","gravatar_id":"","url":"https://api.github.com/users/gbehrmann","html_url":"https://github.com/gbehrmann","followers_url":"https://api.github.com/users/gbehrmann/followers","following_url":"https://api.github.com/users/gbehrmann/following{/other_user}","gists_url":"https://api.github.com/users/gbehrmann/gists{/gist_id}","starred_url":"https://api.github.com/users/gbehrmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gbehrmann/subscriptions","organizations_url":"https://api.github.com/users/gbehrmann/orgs","repos_url":"https://api.github.com/users/gbehrmann/repos","events_url":"https://api.github.com/users/gbehrmann/events{/privacy}","received_events_url":"https://api.github.com/users/gbehrmann/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/eu-emi/canl-java/labels/Severity:%20Feature","name":"Severity: Feature","color":"84b6eb"}],"state":"closed","locked":false,"assignee":{"login":"golbi","id":1583731,"avatar_url":"https://avatars.githubusercontent.com/u/1583731?v=3","gravatar_id":"","url":"https://api.github.com/users/golbi","html_url":"https://github.com/golbi","followers_url":"https://api.github.com/users/golbi/followers","following_url":"https://api.github.com/users/golbi/following{/other_user}","gists_url":"https://api.github.com/users/golbi/gists{/gist_id}","starred_url":"https://api.github.com/users/golbi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/golbi/subscriptions","organizations_url":"https://api.github.com/users/golbi/orgs","repos_url":"https://api.github.com/users/golbi/repos","events_url":"https://api.github.com/users/golbi/events{/privacy}","received_events_url":"https://api.github.com/users/golbi/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/eu-emi/canl-java/milestones/15","html_url":"https://github.com/eu-emi/canl-java/milestones/2.2.0","labels_url":"https://api.github.com/repos/eu-emi/canl-java/milestones/15/labels","id":1162421,"number":15,"title":"2.2.0","description":"","creator":{"login":"golbi","id":1583731,"avatar_url":"https://avatars.githubusercontent.com/u/1583731?v=3","gravatar_id":"","url":"https://api.github.com/users/golbi","html_url":"https://github.com/golbi","followers_url":"https://api.github.com/users/golbi/followers","following_url":"https://api.github.com/users/golbi/following{/other_user}","gists_url":"https://api.github.com/users/golbi/gists{/gist_id}","starred_url":"https://api.github.com/users/golbi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/golbi/subscriptions","organizations_url":"https://api.github.com/users/golbi/orgs","repos_url":"https://api.github.com/users/golbi/repos","events_url":"https://api.github.com/users/golbi/events{/privacy}","received_events_url":"https://api.github.com/users/golbi/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":4,"state":"open","created_at":"2015-06-12T17:06:30Z","updated_at":"2016-01-04T16:17:53Z","due_on":null,"closed_at":null},"comments":7,"created_at":"2015-12-14T18:17:02Z","updated_at":"2016-02-03T14:20:45Z","closed_at":"2016-01-04T16:17:53Z","body":"We recently ported dCache to CANL and the first deployments are happening now. After NDGF upgraded we now see lock contention on OpensslCertChainValidator#validate:\r\n\r\n    \"jetty-srm-8778\" #8778 prio=5 os_prio=0 tid=0x00007fbe9c20b800 nid=0x4249 waiting for monitor entry [0x00007fbe0e3a1000]\r\n    java.lang.Thread.State: BLOCKED (on object monitor)\r\n        at eu.emi.security.authn.x509.impl.OpensslCertChainValidator.validate(OpensslCertChainValidator.java:219)\r\n        - waiting to lock <0x00000006c115d388> (a eu.emi.security.authn.x509.impl.OpensslCertChainValidator)\r\n        at eu.emi.security.authn.x509.helpers.ssl.SSLTrustManager.checkIfTrusted(SSLTrustManager.java:66)\r\n        at eu.emi.security.authn.x509.helpers.ssl.SSLTrustManager.checkClientTrusted(SSLTrustManager.java:51)\r\n        at sun.security.ssl.AbstractTrustManagerWrapper.checkClientTrusted(SSLContextImpl.java:929)\r\n        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1869)\r\n        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:230)\r\n        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)\r\n        at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)\r\n        at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)\r\n        - locked <0x00000007ba473230> (a sun.security.ssl.SSLEngineImpl)\r\n        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:613)\r\n        - locked <0x00000007ba474fd0> (a org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint)\r\n        at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:315)\r\n        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:223)\r\n        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)\r\n        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\r\n        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:192)\r\n        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)\r\n        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\r\n        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)\r\n        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)\r\n        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)\r\n        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)\r\n        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n\r\n\r\nWe follow the advice of the manual: _\"Try to use as few instances as possible, typically one shared instance is enough. Of course all validators are thread safe.\"_. Thus we only have one OpensslCertChainValidator per service and this becomes a contention point. We don't want to resort to one validator per connection as the overhead of reloading certificates all the time is bad too.\r\n\r\nIs `BCCertPathValidator` not thread safe or why does OpensslCertChainValidator need to use such coarse grained locking? This looks to be a major show stopper for us."},"comment":{"url":"https://api.github.com/repos/eu-emi/canl-java/issues/comments/179262014","html_url":"https://github.com/eu-emi/canl-java/issues/80#issuecomment-179262014","issue_url":"https://api.github.com/repos/eu-emi/canl-java/issues/80","id":179262014,"user":{"login":"golbi","id":1583731,"avatar_url":"https://avatars.githubusercontent.com/u/1583731?v=3","gravatar_id":"","url":"https://api.github.com/users/golbi","html_url":"https://github.com/golbi","followers_url":"https://api.github.com/users/golbi/followers","following_url":"https://api.github.com/users/golbi/following{/other_user}","gists_url":"https://api.github.com/users/golbi/gists{/gist_id}","starred_url":"https://api.github.com/users/golbi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/golbi/subscriptions","organizations_url":"https://api.github.com/users/golbi/orgs","repos_url":"https://api.github.com/users/golbi/repos","events_url":"https://api.github.com/users/golbi/events{/privacy}","received_events_url":"https://api.github.com/users/golbi/received_events","type":"User","site_admin":false},"created_at":"2016-02-03T14:20:45Z","updated_at":"2016-02-03T14:20:45Z","body":"Hi,\r\n\r\nNo, pre 2 canl is in pure maintenance mode. However if you want to backport some patches - go on."}}