{"action":"closed","issue":{"url":"https://api.github.com/repos/openssl/openssl/issues/5526","repository_url":"https://api.github.com/repos/openssl/openssl","labels_url":"https://api.github.com/repos/openssl/openssl/issues/5526/labels{/name}","comments_url":"https://api.github.com/repos/openssl/openssl/issues/5526/comments","events_url":"https://api.github.com/repos/openssl/openssl/issues/5526/events","html_url":"https://github.com/openssl/openssl/issues/5526","id":302547542,"number":5526,"title":"Persistent & random crypto failures on ARM64 Ubuntu 16.04","user":{"login":"rvagg","id":495647,"avatar_url":"https://avatars2.githubusercontent.com/u/495647?v=4","gravatar_id":"","url":"https://api.github.com/users/rvagg","html_url":"https://github.com/rvagg","followers_url":"https://api.github.com/users/rvagg/followers","following_url":"https://api.github.com/users/rvagg/following{/other_user}","gists_url":"https://api.github.com/users/rvagg/gists{/gist_id}","starred_url":"https://api.github.com/users/rvagg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rvagg/subscriptions","organizations_url":"https://api.github.com/users/rvagg/orgs","repos_url":"https://api.github.com/users/rvagg/repos","events_url":"https://api.github.com/users/rvagg/events{/privacy}","received_events_url":"https://api.github.com/users/rvagg/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/openssl/openssl/milestones/9","html_url":"https://github.com/openssl/openssl/milestone/9","labels_url":"https://api.github.com/repos/openssl/openssl/milestones/9/labels","id":2145708,"number":9,"title":"1.1.1","description":"","creator":{"login":"levitte","id":698918,"avatar_url":"https://avatars1.githubusercontent.com/u/698918?v=4","gravatar_id":"","url":"https://api.github.com/users/levitte","html_url":"https://github.com/levitte","followers_url":"https://api.github.com/users/levitte/followers","following_url":"https://api.github.com/users/levitte/following{/other_user}","gists_url":"https://api.github.com/users/levitte/gists{/gist_id}","starred_url":"https://api.github.com/users/levitte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/levitte/subscriptions","organizations_url":"https://api.github.com/users/levitte/orgs","repos_url":"https://api.github.com/users/levitte/repos","events_url":"https://api.github.com/users/levitte/events{/privacy}","received_events_url":"https://api.github.com/users/levitte/received_events","type":"User","site_admin":false},"open_issues":73,"closed_issues":307,"state":"open","created_at":"2016-11-19T09:18:52Z","updated_at":"2018-05-06T20:50:17Z","due_on":null,"closed_at":null},"comments":7,"created_at":"2018-03-06T03:06:54Z","updated_at":"2018-05-06T20:50:17Z","closed_at":"2018-05-06T20:50:17Z","author_association":"NONE","body":"We have a strange problem over at Node.js with our test infrastructure. I'm not sure exactly where the best place to go with this is so I'm going to try here first and perhaps you have an idea of where we might look next.\r\n\r\nAs part of our test infrastructure we use Cavium ThunderX (as available from packet.net) machines to help test our ARM64 support. We get occasional failures in crypto operations (either some direct crypto work or via tls) with errors that we don't see anywhere else but on these machines. These are the kinds of output we see along with failures, keeping in mind that we're working against a statically linked 1.0.2n out of our source tree:\r\n\r\n```\r\n    Error: 281472750350336:error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac:../deps/openssl/openssl/ssl/s3_pkt.c:535:\r\n```\r\n\r\n```\r\n    Error: 281473003442176:error:140940F6:SSL routines:ssl3_read_bytes:unknown alert type:../deps/openssl/openssl/ssl/s3_pkt.c:1508:\r\n```\r\n\r\n```\r\n    Error: write EPROTO 281473430319104:error:1409441B:SSL routines:ssl3_read_bytes:tlsv1 alert decrypt error:../deps/openssl/openssl/ssl/s3_pkt.c:1500:SSL alert number 51\r\n    281473430319104:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:659:\r\n```\r\n\r\nAnd some higher-level failures that suggest deeper problems:\r\n\r\nwhen trying to do a pbkdf2 test:\r\n\r\n```\r\n    AssertionError [ERR_ASSERTION]: '�æ�_]\\u0013G\\u000e;F§�Ù\\u0013e+¸\\u000fc�' strictEqual 'ÅäxÕ��ÈAªS\\r¶�\\\\L��(� '\r\n```\r\n\r\nAnd an apparent failure when testing for memory leakage after performing a simple DH keypair creation, it's hard to be sure but the failure case does suggest there was an unexpected increase in retained memory somewhere in stack after cleanup.\r\n\r\nThe first error seems to show up most often.\r\n\r\nThere's a number of reasons why this is weird:\r\n\r\n* As far as I can tell these are relatively new, perhaps since we started testing on this particular hardware\r\n* The errors don't happen with the same frequency on different machines with the same configuration--we have two Ubuntu 16.04 machines in the mix here and one of them seems to throw these up much more often than the other, but they still happen on both so I don't think it's as likely to just be one dodgy machine.\r\n* We have CentOS 7 running on two more of these machines and I can't find a record of the errors there at all.\r\n\r\nPerhaps most disturbingly, this isn't limited to our direct OpenSSL usage. We first recognised this with repeated SSL errors thrown up by Jenkins while trying to maintain a connection on one of these machines. We saw a whole lot of this kind of thing:\r\n\r\n```\r\njavax.crypto.BadPaddingException: bad record MAC\r\n\tat sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:238)\r\n\tat sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)\r\nCaused: javax.net.ssl.SSLException: bad record MAC\r\n\tat sun.security.ssl.Alerts.getSSLException(Alerts.java:208)\r\n\tat sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)\r\n\tat sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)\r\n\tat sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)\r\n\tat sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)\r\n\tat javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)\r\n```\r\n\r\nWhich has nothing to do with Node.js, this is communication between a Java (8 iirc) JNLP client and an nginx reverse proxy to the Jenkins server. But it appears to be a similar error to our most common failure. As far as I know the Java crypto stack doesn't touch OpenSSL does it?\r\n\r\n/cc @vielmetti"}}
{"actor":{"display_login":"richsalz"}}