title: usage of RSA_NO_PADDING when decrypting data with RSA private key using openssl libcrypto
body: <p>I generated RSA private key and public key as below,</p>
  
  <pre><code>openssl genpkey -algorithm RSA -out key.pem -pkeyopt rsa_keygen_bits:2048
  openssl rsa -pubout -in pri.key -out pub.key
  </code></pre>
  
  <p>And encrypted text file as below,</p>
  
  <pre><code>openssl pkeyutl -encrypt -pubin -inkey ~/pub.key -in ~/1.txt -out ~/1e.txt
  </code></pre>
  
  <p>Then I wrote below program to decrypt the encryted file. However, it seemed that decrypt didn't work as  expected.</p>
  
  <pre><code>#include &lt;openssl/evp.h&gt;
  #include &lt;openssl/rsa.h&gt;
  #include &lt;openssl/pem.h&gt;
  #include &lt;openssl/err.h&gt;
  #include &lt;openssl/conf.h&gt;
  #include &lt;iostream&gt;
  
  using namespace std;
  
  void
  cleanup()
  {
      EVP_cleanup();
      CRYPTO_cleanup_all_ex_data();
      ERR_free_strings();
  }
  
  int
  main(int argc, char** argv)
  {
      ERR_load_crypto_strings();
      OpenSSL_add_all_algorithms();
      OPENSSL_config(nullptr);
  
      cout&lt;&lt;"Initialize crypto library done"&lt;&lt;endl;
  
      EVP_PKEY * key = EVP_PKEY_new();
      if (key == nullptr) {
          cout&lt;&lt;"Failed to contruct new key"&lt;&lt;endl;
          return 1;
      }
      FILE * fpri = nullptr;
      fpri = fopen("/home/stack/pri.key", "r");
      if (fpri == nullptr) {
          cout&lt;&lt;"Failed to load private key"&lt;&lt;endl;
          return 1;
      }
      key = PEM_read_PrivateKey(fpri, &amp;key, nullptr, nullptr);
      if (key == nullptr) {
          std::cout&lt;&lt;"Read private key failed"&lt;&lt;endl;
          return 1;
      }
      cout&lt;&lt;"load private key successfully"&lt;&lt;endl;
      EVP_PKEY_CTX *ctx = nullptr;
      ctx = EVP_PKEY_CTX_new(key, nullptr);
      EVP_PKEY_decrypt_init(ctx);
      EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_NO_PADDING);
  
      size_t outlen = 0, inlen = 0;
      unsigned char * out = nullptr, * in = nullptr;
  
      char buf[1024];
      FILE * fe = nullptr;
      fe = fopen("/home/stack/1e.txt", "r");
      size_t len = fread(buf, 1, sizeof(buf),  fe);
      cout&lt;&lt;"data input length is "&lt;&lt;len&lt;&lt;endl;
      EVP_PKEY_decrypt(ctx, NULL, &amp;outlen, in, inlen);
      cout&lt;&lt;"outlen is "&lt;&lt;outlen&lt;&lt;endl;
  
      out = (unsigned char*)OPENSSL_malloc(outlen);
      EVP_PKEY_decrypt(ctx, out, &amp;outlen, in, inlen);
      cout&lt;&lt;"decrypted data "&lt;&lt;out&lt;&lt;endl;
      cleanup();
  
      return 0;
  
  }
  </code></pre>
  
  <p>When executing the code, the result is as below,</p>
  
  <pre><code>[stack@agent ~]$ ./test
  Initialize crypto library done
  load private key successfully
  data input length is 256
  outlen is 256
  decrypted data
  </code></pre>
  
  <p>It seemed the decrypted data length was not correct and not printable.</p>
  
  <p>When I commented out the instruction "EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_NO_PADDING);", it worked well.</p>
  
  <p>I also tried with RSA_PKCS1_OAEP_PADDING, it didn't work either. If RSA PADDING is not set, it worked.</p>
  
  <p>My question is as below,</p>
  
  <ol>
  <li><p>Which padding is used in following command?</p>
  
  <pre><code>openssl pkeyutl -encrypt -pubin -inkey ~/pub.key -in ~/1.txt -out ~/1e.txt
  </code></pre></li>
  <li><p>Is padding necessary for RSA encrypt/decrypt? If so, how could I apply the padding mechanism?</p></li>
  </ol>