title: PKCS#11 Smartcard getting error: javax.net.ssl.SSLHandshakeException due to java.security.InvalidKeyException: The RSA asymmetric cipher
body: <h2>Situation</h2>
  
  <p>I am working on a project where we must get Smartcard to work in Linux. The card is by manufacturer Izenpe. I get an <code>javax.net.ssl.SSLHandshakeException</code> every time when I am doing SSL handshake with the server.</p>
  
  <p>I load the Izenpe driver with these code:</p>
  
  <pre><code>private final static String config_path3 = "name=IZENPE\nlibrary=/usr/lib/libbit4ipki.so";
  
  ...
  
  String config = config_path3;
  Provider provider = new SunPKCS11(new ByteArrayInputStream(config.getBytes()));
  Security.removeProvider("IAIK");
  Security.insertProviderAt(provider, 1);
  try {
      keystore = KeyStore.getInstance("PKCS11", provider);
  } catch (KeyStoreException e) {
      throw e;
  }
  KeyStore.Builder builder = KeyStore.Builder.newInstance("PKCS11", provider, 
          new KeyStore.CallbackHandlerProtection(new UtilTarjetas(). new callback()));
  keystore = builder.getKeyStore();
  </code></pre>
  
  <p>It loads the certificates to show in a table, and I can proceed to the part to obtain the <code>OutputStream</code> from the response of server, where it fails when we do SSL handshake (signing the challenge). Full stacktraces below:</p>
  
  <pre><code>javax.net.ssl.SSLHandshakeException: Error signing certificate verify
      at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
      at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1904)
      at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:279)
      at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1054)
      at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:341)
      at sun.security.ssl.Handshaker.processLoop(Handshaker.java:901)
      at sun.security.ssl.Handshaker.process_record(Handshaker.java:837)
      at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1023)
      at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
      at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)
      at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1343)
      at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
      at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
      at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1092)
      at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
      at gestores.comunicacion.gisscide.HTTPS_TGSS.doHTTP_Get(HTTPS_TGSS.java:470)
      at gestores.comunicacion.Gestor_Comunicaciones$RecibeMensajesSSL.descargaMsgHTTPS(Gestor_Comunicaciones.java:1284)
      at gestores.comunicacion.Gestor_Comunicaciones$RecibeMensajesSSL.run(Gestor_Comunicaciones.java:1852)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.security.InvalidKeyException: The RSA asymmetric cipher only operates with RSA keys; unsupported key found (sun.security.pkcs11.P11Key$P11PrivateKey)
      at com.entrust.toolkit.security.provider.RSA.a(Unknown Source)
      at com.entrust.toolkit.security.provider.RSA.engineGetKeySize(Unknown Source)
      at javax.crypto.Cipher.passCryptoPermCheck(Cipher.java:1052)
      at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1010)
      at javax.crypto.Cipher.init(Cipher.java:1209)
      at java.security.Signature$CipherAdapter.engineInitSign(Signature.java:1254)
      at java.security.Signature$Delegate.init(Signature.java:1128)
      at java.security.Signature$Delegate.chooseProvider(Signature.java:1085)
      at java.security.Signature$Delegate.engineInitSign(Signature.java:1158)
      at java.security.Signature.initSign(Signature.java:529)
      at sun.security.ssl.RSASignature.engineInitSign(RSASignature.java:125)
      at java.security.Signature$Delegate.engineInitSign(Signature.java:1156)
      at java.security.Signature.initSign(Signature.java:529)
      at sun.security.ssl.HandshakeMessage$CertificateVerify.&lt;init&gt;(HandshakeMessage.java:1556)
      at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1049)
      ... 15 more
  </code></pre>
  
  <p>It seems that when getting the unextractable private key in the card for signing the challenge in SSL handshake, it fail because PKCS#11 private key is not supported/recognized as valid. </p>
  
  <p>By adding this line into my code I get many lines of logs about SSL.</p>
  
  <pre><code>Properties systemProps = System.getProperties();
  systemProps.put("javax.net.debug", "all");          
  System.setProperties(systemProps);
  </code></pre>
  
  <p>At the end I can see this:</p>
  
  <pre><code>SESSION KEYGEN:
  PreMaster Secret:
  0000: A5 F7 9C 89 A4 B5 B4 66   D4 DC CC 40 45 C8 41 07  .......f...@E.A.
  0010: 0E F1 E9 5C 99 36 C8 84   06 B0 6B 95              ...\.6....k.
  CONNECTION KEYGEN:
  Client Nonce:
  0000: 58 07 74 02 CD 8B 65 BB   E0 03 8D 53 95 C4 87 8C  X.t...e....S....
  0010: EE 95 B2 92 C1 DF E8 CA   B0 7D E4 AD 16 B7 31 D2  ..............1.
  Server Nonce:
  0000: 4A 41 AA 83 DD 1D 9C DE   84 4A 56 40 A3 32 F7 53  JA.......JV@.2.S
  0010: 18 48 32 BD 7A E2 3A 1D   19 AD 67 6E DD E1 3B 20  .H2.z.:...gn..; 
  Master Secret:
  0000: 72 58 C0 3D 90 67 2E 3B   B2 AE D4 54 15 AB 18 AA  rX.=.g.;...T....
  0010: 95 73 91 9A DA AE EF 3D   77 A1 CD 7A 68 8B 37 56  .s.....=w..zh.7V
  0020: 0F 05 64 EB DD 93 AF 6C   C4 C2 8A 75 A7 C2 CA 06  ..d....l...u....
  Client MAC write Secret:
  0000: 30 2C D3 A0 4C 2D 3F 67   ED B9 64 B8 3B 81 47 0E  0,..L-?g..d.;.G.
  0010: D1 7B 75 A9                                        ..u.
  Server MAC write Secret:
  0000: 4B 22 25 8E 81 D1 55 6D   B9 40 0F 2A A2 26 49 F5  K"%...Um.@.*.&amp;I.
  0010: 66 6A 91 AE                                        fj..
  Client write key:
  0000: 72 BD C0 56 3C 1A E5 61   90 2C A6 AE AA FE B9 71  r..V&lt;..a.,.....q
  Server write key:
  0000: C6 CE F4 C6 CE A4 E3 55   F6 2D 29 D6 2E 4C CA 7A  .......U.-)..L.z
  Client write IV:
  0000: 7D DB C8 17 F4 18 72 33   A1 DC 03 D6 2F 87 65 F1  ......r3..../.e.
  Server write IV:
  0000: 58 B1 4E BE 1A 90 0C B3   0D AE 5C 5B CD 36 74 4D  X.N.......\[.6tM
  %% Invalidated:  [Session-3, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA]
  Thread-7, SEND TLSv1 ALERT:  fatal, description = handshake_failure
  Thread-7, WRITE: TLSv1 Alert, length = 2
  [Raw write]: length = 7
  0000: 15 03 01 00 02 02 28                               ......(
  Thread-7, called closeSocket()
  Thread-7, handling exception: javax.net.ssl.SSLHandshakeException: Error signing certificate verify
  Finalizer, called close()
  Finalizer, called closeInternal(true)
  </code></pre>
  
  <p>I don't know if it's relevant, but when testing my server site in www.ssllabs.com, I found TLSv1.2 is supported but TLSv1.1 is not. And the algorithm which is picked here in the log is not listed in the preferred list of the server:</p>
  
  <p><a href="https://i.stack.imgur.com/q3iby.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/q3iby.png" alt="enter image description here"></a></p>
  
  <p>So, what can be the reason?</p>
  
  <ul>
  <li><p>I have changed the <code>$JAVA_HOME/jre/lib/security/java.security</code>, disabling the <code>TLSv1.2</code> and <code>SSLv3</code> in the option <code>jdk.tls.disabledAlgorithms=SSLv3, TLSv1.1</code>, leaving only <code>TLSv1.1</code> because it's the server's choice, to no avail.</p></li>
  <li><p>The driver I used to load the Provider is from Izenpe website. From what I got in the Console of Eclipse, it's working.</p></li>
  </ul>
  
  <h2>My thought about it:</h2>
  
  <ul>
  <li><p>What fails maybe the Provider <code>SunPKCS11</code>. I say this because the original code works in Windows when we use <code>SunMSCAPI</code>. Some internal mechanism of <code>SunMSCAPI</code> did the work for us to sign the challenge. But now as we must adapt the project to be multi-platformed, in Linux we have no choice but <code>SunPKCS11</code>. The JAR <code>sunpkcs11.jar</code> of Java 6 and Java 8 both failed. (In Java 7 it's not present.)</p></li>
  <li><p>The original intention is to make (almost) all smart cards available for authentication in my APP, but at the moment we only want Izenpe and DNIe(Spanish electronic ID card, supported by OpenSC) to work. </p></li>
  </ul>
  
  <hr>