title: Why does system() fail with error code 127?
body: <p>On a Linux system I am trying to call a program at runtime with the <code>system()</code> call.
  The system call exits with an return code not equal zero.</p>
  
  <p>Calling <code>WEXITSTATUS</code> on the error code gives "127".</p>
  
  <p>According to the man page of system this code indicates that <code>/bin/sh</code> could not be called:</p>
  
  <p>In case <code>/bin/sh</code> could not be executed,
  the exit status will be that of a command that does <code>exit(127)</code>.</p>
  
  <p>I checked: <code>/bin/sh</code> is a link to <code>bash</code>. <code>bash</code> is there. I can execute it from the shell.</p>
  
  <p>Now, how can I find out why <code>/bin/sh</code> could not be called ? 
  Any kernel history or something?</p>
  
  <p>Edit:</p>
  
  <p>After the very helpful tip (see below) i <code>strace -f -p &lt;PID&gt;</code> the process. This is what I get during the <code>system</code> call:</p>
  
  <pre><code>Process 16080 detached
  [pid 11779] &lt;... select resumed&gt; )      = ? ERESTARTNOHAND (To be restarted)
  [pid 11774] &lt;... wait4 resumed&gt; [{WIFEXITED(s) &amp;&amp; WEXITSTATUS(s) == 127}], 0, NULL) = 16080
  [pid 11779] --- SIGCHLD (Child exited) @ 0 (0) ---
  [pid 11779] rt_sigaction(SIGCHLD, {0x2ae0ff898ae2, [CHLD], SA_RESTORER|SA_RESTART, 0x32dd2302d0},  &lt;unfinished ...&gt;
  [pid 11774] rt_sigaction(SIGINT, {0x2ae1042070f0, [], SA_RESTORER|SA_SIGINFO, 0x32dd2302d0},  &lt;unfinished ...&gt;
  [pid 11779] &lt;... rt_sigaction resumed&gt; {0x2ae0ff898ae2, [CHLD], SA_RESTORER|SA_RESTART, 0x32dd2302d0}, 8) = 0
  [pid 11779] sendto(5, "a", 1, 0, NULL, 0 &lt;unfinished ...&gt;
  [pid 11774] &lt;... rt_sigaction resumed&gt; NULL, 8) = 0
  [pid 11779] &lt;... sendto resumed&gt; )      = 1
  [pid 11779] rt_sigreturn(0x2 &lt;unfinished ...&gt;
  [pid 11774] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x32dd2302d0},  &lt;unfinished ...&gt;
  [pid 11779] &lt;... rt_sigreturn resumed&gt; ) = -1 EINTR (Interrupted system call)
  [pid 11779] select(16, [9 15], [], NULL, NULL &lt;unfinished ...&gt;
  [pid 11774] &lt;... rt_sigaction resumed&gt; NULL, 8) = 0
  [pid 11774] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
  [pid 11774] write(1, "Problems calling nvcc jitter: ex"..., 49) = 49
  [pid 11774] rt_sigaction(SIGINT, {0x1, [], SA_RESTORER, 0x32dd2302d0}, {0x2ae1042070f0, [], SA_RESTORER|SA_SIGINFO, 0x32dd2302d0}, 8) = 0
  [pid 11774] rt_sigaction(SIGQUIT, {0x1, [], SA_RESTORER, 0x32dd2302d0}, {SIG_DFL, [], SA_RESTORER, 0x32dd2302d0}, 8) = 0
  [pid 11774] rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
  [pid 11774] clone(Process 16081 attached (waiting for parent)
  Process 16081 resumed (parent 11774 ready)
  child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fff0177ab68) = 16081
  [pid 16081] rt_sigaction(SIGINT, {0x2ae1042070f0, [], SA_RESTORER|SA_SIGINFO, 0x32dd2302d0},  &lt;unfinished ...&gt;
  [pid 11774] wait4(16081, Process 11774 suspended
   &lt;unfinished ...&gt;
  [pid 16081] &lt;... rt_sigaction resumed&gt; NULL, 8) = 0
  [pid 16081] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x32dd2302d0}, NULL, 8) = 0
  [pid 16081] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
  [pid 16081] execve("/bin/sh", ["sh", "-c", 0xdda1d98], [/* 58 vars */]) = -1 EFAULT (Bad address)
  [pid 16081] exit_group(127)             = ?
  Process 11774 resumed
  </code></pre>
  
  <p>When it comes to the call to <code>/bin/sh</code> it says bad address. Why that ?</p>
  
  <p>Edit:</p>
  
  <p>Here the whole part that involves the failing <code>system</code> (here already the safe copy to a buffer is in place):</p>
  
  <pre><code>  std::ostringstream jit_command;
  
    jit_command &lt;&lt; string(CUDA_DIR) &lt;&lt; "/bin/nvcc -v --ptxas-options=-v ";
    jit_command &lt;&lt; "-arch=" &lt;&lt; string(GPUARCH);
    jit_command &lt;&lt; " -m64 --compiler-options -fPIC,-shared -link ";
    jit_command &lt;&lt; fname_src &lt;&lt; " -I$LIB_PATH/include -o " &lt;&lt; fname_dest;
  
    string gen = jit_command.str();
    cout &lt;&lt; gen &lt;&lt; endl;
  
    char* cmd = new(nothrow) char[gen.size()+1];
    if (!cmd) ___error_exit("no memory for jitter command");
    strcpy(cmd,gen.c_str());
  
    int ret;
  
    if (ret=system(cmd)) {
  
      cout &lt;&lt; "Problems calling nvcc jitter: ";
  
      if (WIFEXITED(ret)) {
        printf("exited, status=%d\n", WEXITSTATUS(ret));
      } else if (WIFSIGNALED(ret)) {
        printf("killed by signal %d\n", WTERMSIG(ret));
      } else if (WIFSTOPPED(ret)) {
        printf("stopped by signal %d\n", WSTOPSIG(ret));
      } else if (WIFCONTINUED(ret)) {
        printf("continued\n");
      } else {
        printf("not recognized\n");
      }
  
      cout &lt;&lt; "Checking shell.. ";
      if(system(NULL))
        cout &lt;&lt; "ok!\n";
      else
        cout &lt;&lt; "nope!\n";
  
      __error_exit("Nvcc error\n");
  
    }
    delete[] cmd;
    return true;
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>/usr/local/cuda/bin/nvcc -v --ptxas-options=-v -arch=sm_20 -m64 --compiler-options -fPIC,-shared -link bench_cudp_Oku2fm.cu -I$LIB_PATH/include -o bench_cudp_Oku2fm.o
  Problems calling nvcc jitter: exited, status=127
  Checking shell.. ok!
  </code></pre>
  
  <p>Edit (first version of the code):</p>
  
  <pre><code>string gen = jit_command.str();
  cout &lt;&lt; gen &lt;&lt; endl;
  int ret;
  if (ret=system(gen.c_str())) {
    ....
  </code></pre>
  
  <p>The complexity of the string creation is not the problem here. As <code>strace</code> shows a "bad address" is the problem. Its a legal string. A "bad address" should not occur.</p>
  
  <p>As far as i know the <code>std::string::c_str()</code> returns a <code>const char *</code> that might point to a scratch space of libc++ where a read only copy of the string might be kept. </p>
  
  <p>Unfortunately the error is not really reproduceable. The call to <code>system</code> succeeds several times before it fails.</p>
  
  <p>I don't want to be hasty but it smells like a bug in either in the kernel, libc or the hardware.</p>
  
  <p>Edit:</p>
  
  <p>I produced a more verbose <code>strace</code> output (<code>strace -f -v -s 2048 -e trace=process -p $!</code>) of the failing <code>execve</code> system call:</p>
  
  <p>First a succeeding call:</p>
  
  <pre><code>[pid  2506] execve("/bin/sh", ["sh", "-c", "/usr/local/cuda/bin/nvcc -v --ptxas-options=-v -arch=sm_20 -m64 --compiler-options -fPIC,-shared -link /home/user/toolchain/kernels-empty/bench_cudp_U11PSy.cu -I$LIB_PATH/include -o /home/user/toolchain/kernels-empty/bench_cudp_U11PSy.o"], ["MODULE_VERSION_STACK=3.2.8", ... ]) = 0
  </code></pre>
  
  <p>Now the failing one:</p>
  
  <pre><code>[pid 17398] execve("/bin/sh", ["sh", "-c", 0x14595af0], &lt;list of vars&gt;) = -1 EFAULT (Bad address)
  </code></pre>
  
  <p>Here <code>&lt;list of vars&gt;</code> is identical. It seems its not the list of environment variables that cause the bad address.
  As Chris Dodd mentioned the 3rd argument to execve is the raw pointer 0x14595af0, which strace thinks (and the kernel agrees) is invalid. <code>strace</code> does not recognize it as a string (so it prints the hex value and not the string).</p>
  
  <p>Edit:</p>
  
  <p>I inserted print out of the pointer value <code>cmd</code> to see what's the value of this pointer in the parent process:</p>
  
  <pre><code>  string gen = jit_command.str();
    cout &lt;&lt; gen &lt;&lt; endl;
    char* cmd = new(nothrow) char[gen.size()+1];
    if (!cmd) __error_exit("no memory for jitter command");
    strcpy(cmd,gen.c_str());
    cout &lt;&lt; "cmd = " &lt;&lt; (void*)cmd &lt;&lt; endl;
    int ret;
    if (ret=system(cmd)) {
      cout &lt;&lt; "failed cmd = " &lt;&lt; (void*)cmd &lt;&lt; endl;
      cout &lt;&lt; "Problems calling nvcc jitter: ";
  </code></pre>
  
  <p>Output (for the failing call):</p>
  
  <pre><code>cmd = 0x14595af0
  failed cmd = 0x14595af0
  Problems calling nvcc jitter: exited, status=127
  Checking shell.. ok!
  </code></pre>
  
  <p>Its the same pointer value as the 3rd argument from <code>strace</code>. (I updated the <code>strace</code> output above).</p>
  
  <p>Regards the 32bit looking of the <code>cmd</code> pointer: I checked the value of the <code>cmd</code> pointer for a succeeding call. Can't see any difference in structure. That's one of the values of <code>cmd</code> when then <code>system</code> call succeeds:</p>
  
  <pre><code>cmd = 0x145d4f20
  </code></pre>
  
  <p>So, before the <code>system</code> call the pointer is valid. As the <code>strace</code> output from above suggests the child process (after calling <code>fork</code>) receives the correct pointer value. But, for some reason, the pointer value is marked invalid in the child process.</p>
  
  <p>Right now we think its either:</p>
  
  <ul>
  <li>libc/kernel bug</li>
  <li>hardware problem</li>
  </ul>
  
  <p>Edit:</p>
  
  <p>Meanwhile let me post a workaround. Its so silly to be forced to implement something like that... but it works. So the following code block gets executed in case the <code>system</code> call fails. It allocates new command strings and retries until it succeeds (well not indefinitely).</p>
  
  <pre><code>    list&lt;char*&gt; listPtr;
      int maxtry=1000;
      do{
        char* tmp = new(nothrow) char[gen.size()+1];
        if (!tmp) __error_exit("no memory for jitter command");
        strcpy(tmp,gen.c_str());
        listPtr.push_back( tmp );
      } while ((ret=system(listPtr.back())) &amp;&amp; (--maxtry&gt;0));
  
      while(listPtr.size()) {
        delete[] listPtr.back();
        listPtr.pop_back();
      }
  </code></pre>
  
  <p>Edit:</p>
  
  <p>I just saw that this workaround in one particular run did not work. It went the whole way, 1000 attempts, all with newly allocated <code>cmd</code> command strings. All 1000 failed.
  Not only this. I tried on a different Linux host (same Linux/software configuration tho).</p>
  
  <p>Taking this into account one would maybe exclude a hardware problem. (Must be on 2 physically different hosts then). Remains a kernel bug ??</p>
  
  <p>Edit:</p>
  
  <p>torek, i will try and install a modified <code>system</code> call. Give me some time for that.</p>