title: How to prevent a process from spawning more childs
body: <p>I am working on an online judge for conducting ACM-ICPC like competitions on my college LAN.
  For that I require that the judge may be safe enough to prevent malicious programs from executing themselves on my server.
  (An example of such a program would be )</p>
  
  <pre><code>int main(){
      while(1) fork();
  }
  </code></pre>
  
  <p>Lets call the executable of this program <strong>testcode</strong>.</p>
  
  <p>This program would cause my server running the judge to freeze.
  Obviously i dont want that to happen.So to prevent that i tried using ptrace.I came up with the following code:(Lets call the executable of this code <strong>monitor</strong>)</p>
  
  <pre><code>#include &lt;sys/ptrace.h&gt;
  #include &lt;sys/types.h&gt;
  #include &lt;sys/wait.h&gt;
  #include &lt;unistd.h&gt;
  #include &lt;string.h&gt;
  #include &lt;errno.h&gt;
  #include &lt;sys/user.h&gt;
  #include &lt;sys/syscall.h&gt;
  #include &lt;sys/reg.h&gt;
  #include&lt;stdio.h&gt;
  #include &lt;sys/ptrace.h&gt;
  #include &lt;sys/types.h&gt;
  #include &lt;sys/wait.h&gt;
  #include &lt;unistd.h&gt;
  #include &lt;string.h&gt;
  #include &lt;errno.h&gt;
  #include &lt;sys/user.h&gt;
  #include &lt;sys/syscall.h&gt;
  #include &lt;sys/reg.h&gt;
  #include&lt;stdio.h&gt;
  #include&lt;signal.h&gt;
  #include&lt;sys/prctl.h&gt;
  #include&lt;stdlib.h&gt;
  #define NOBANNEDSYS 40
  int bannedsys[50]={2,14,12,15,26,37,38,39,39,40,41,42,46,47,48,49,50,60,61,63,72,83,88,120,102,182,183,190};
  
  int main(int argc,char **argv) {
                  int insyscall=0;
                  if(argc!=2) {
                      fprintf(stderr,"Usage: %s &lt;prog name&gt; ",argv[0]);
                      exit(-1);
                      }
  
    int status = 0;
    int syscall_n = 0;
    int entering = 1;
    int amp;
    struct user_regs_struct regs;
    int pid = fork();
  
    if ( !pid ) {
    prctl(PR_SET_PDEATHSIG, SIGKILL);
      ptrace( PTRACE_TRACEME, 0, 0, 0 );
      execlp( argv[1],argv[1], 0 );
      }
  
    else {
  
      //ptrace( PTRACE_SINGLESTEP ,pid, 0, 0 );
      // ptrace( PTRACE_SYSCALL, pid, 0, 0 ); 
      while (1) {
  
          wait( &amp;amp); 
              if ( WIFEXITED( amp ) ) break;
  
                  //ptrace( PTRACE_SINGLESTEP ,pid, 0, 0 ); 
  
      if(insyscall==0){
          ptrace( PTRACE_GETREGS, pid, 0,&amp;regs ); 
        int i=0;
        for(i=0;i&lt;NOBANNEDSYS;i++) if(regs.orig_eax==bannedsys[i]) {  kill(pid,SIGKILL);
                                  printf("%d killed due to illegal system call\n",pid);
  
                                      abort();
                                      }
  
          insyscall=1;
                 }
         else insyscall=0;
        // ptrace(PTRACE_CONT,pid,0,0);
       // wait(&amp;amp);
  
      ptrace( PTRACE_SYSCALL, pid, 0, 0 );
       // puts("Here");
  //ptrace(PTRACE_CONT, pid, 0, 0);
  
      }
  
    }
  
    return 0;
  }
  </code></pre>
  
  <p>This code works quite well in blocking system calls that may cause problem.But when the code to be monitored contains  fork calls in a loop like the <strong>testcode</strong> the machine freezes because of thrashing.The reason i could make out is while the original process is killed by the monitor code its child survives and continues to carry the fork bomb.
  How to fix the monitor code so that it can be deployed successfully?</p>
  
  <p>PS:Portability is not a concern.I am looking for a linux specific answer.</p>
  
  <p>EDIT:
  I used setrlimit for setting the maximum number of child processes to 0 before calling exec.This till now appears to be a a good solution.Though it would be nice to hear from the community if there are still loopholes in the monitor code.</p>