title: Self-modifying code on macOS v10.15 (Catalina) / x64
body: <p>As part as porting a Forth compiler, I'm trying to create a binary that allows for self-modifying code. The gory details are at <a href="https://github.com/klapauciusisgreat/jonesforth-MacOS-x64" rel="nofollow noreferrer">https://github.com/klapauciusisgreat/jonesforth-MacOS-x64</a></p>
  <p>Ideally, I create a bunch of pages for user definitions and call mprotect like so:</p>
  <pre><code>#define __NR_exit 0x2000001
  #define __NR_open 0x2000005
  #define __NR_close 0x2000006
  #define __NR_read 0x2000003
  #define __NR_write 0x2000004
  #define __NR_mprotect 0x200004a
  
  #define PROT_READ 0x01
  #define PROT_WRITE 0x02
  #define PROT_EXEC 0x04
  #define PROT_ALL (PROT_READ | PROT_WRITE | PROT_EXEC)
  #define PAGE_SIZE 4096
  
  
  // https://opensource.apple.com/source/xnu/xnu-201/bsd/sys/errno.h
  #define EACCES    13              /* Permission denied */
  #define EINVAL    22              /* Invalid argument */
  #define ENOTSUP   45              /* Operation not supported */
  
  
  /* Assembler entry point. */
          .text
          .globl start
  start:
          // Use mprotect to allow read/write/execute of the .bss section
          mov $__NR_mprotect, %rax                // mprotect
          lea user_defs_start(%rip), %rdi         // Start address
          and $-PAGE_SIZE,%rdi                    // Align at page boundary
          mov $USER_DEFS_SIZE, %rsi               // Length
          mov $PROT_ALL,%rdx
          syscall
          cmp $EINVAL, %rax
          je 1f
          cmp $EACCES,%rax
          je 2f
          test %rax,%rax
          je 4f                                   // All good, proceed:
  
          // must be ENOTSUP
          mov $2,%rdi                     // First parameter: stderr
          lea errENOTSUP(%rip),%rsi       // Second parameter: error message
          mov $8,%rdx                     // Third parameter: length of string
          mov $__NR_write,%rax            // Write syscall
          syscall
          jmp 3f
  
  1:
          mov $2,%rdi                     // First parameter: stderr
          lea errEINVAL(%rip),%rsi        // Second parameter: error message
          mov $7,%rdx                     // Third parameter: length of string
          mov $__NR_write,%rax            // Write syscall
          syscall
          jmp 3f
  
  2:
          mov $2,%rdi                     // First parameter: stderr
          lea errEACCES(%rip),%rsi        // Second parameter: error message
          mov $7,%rdx                     // Third parameter: length of string
          mov $__NR_write,%rax            // Write syscall
          syscall
  
  3:
          // did't work -- then exit
          xor %rdi,%rdi
          mov $__NR_exit,%rax     // syscall: exit
          syscall
  
  4:
  // All good, let's get started for real:
  
  .
  .
  .
  
          .set RETURN_STACK_SIZE,8192
          .set BUFFER_SIZE,4096
          .set USER_DEFS_SIZE,65536*2 // 128 kiB ought to be enough for everybody
  
          .bss
          .balign 8
  user_defs_start:
          .space USER_DEFS_SIZE
  
  </code></pre>
  <p>However, I get an EACCES return value. I suspect that this is because of some security policy apple has set up, but I do not find good documentation.</p>
  <p>Where is the source code for mprotect, and/or what are the methods to mark the data area executable and writable at the same time?</p>
  <p>I found that compiling with</p>
  <pre class="lang-sh prettyprint-override"><code>gcc -segprot __DATA rwx rwx
  </code></pre>
  <p>does mark the entire data segment rwx, so it must somehow be possible to do the right thing. But I would prefer to only make the area hosting Forth words executable, not the entire data segment.</p>
  <p>I found a similar discussion <a href="https://stackoverflow.com/questions/60497896/self-modifying-code-on-darwin-10-15-resulting-in-malformed-mach-o-image/60504320#60504320">here</a>, but without any solution.</p>