https://gist.github.com/gynvael/755f138ec4c6e656d2614b5749271f14 (writeup + emulator code)
## Notes on Infiltration (JerseyCTF III '23)
```A rogue AI has infiltrated a game server's custom VM run on PPC and its code is now traversing the user base. The developers have decompiled and given the current executing script the memory it was accessing at the time and opcode documentation. You are tasked with investigating the nature of this threat.```
You were given three files:
  * `opcodes.md` with an incomplete description of opcodes  * `ctf.xsa` with assembly (as in: text) for some architecture  * `memory.bin` with 4KB of binary data (high entropy apart from the header)
Apart from that the somewhat useful hint was who made the challenge (Native Function), as they had a lot of "RAGE VM" related repos/tools on their github.
  * https://github.com/NativeFunction
So yeah, this challenge was about the Rockstar Advanced Game Engine aka RAGE (which I've learnt a few hours into the challenge), or rather its scripting language. Or rather its assembly form in a dialect made by folks that made the RASM (dis)assembler. Actually the fact that it was a dialect made it a bit more difficult to Google for it, which made the whole challenge more confusing to solve.
Eventually the approach I took was to start implementing the assembly parser + emulator based on the instruction documentation for the actual instructions used only. And by that I mean I told GPT-4 to implement what it could and then started fixing that by analyzing the provided documentation and the actual code.
GPT-4 was actually also useful to translate the assembly code into Python based on documentation. It wasn't correct, but it gave me a decent idea what I was dealing with.
In the end there were 4 steps to this challenge:
  1. Load the header (4 big-endian ints: magic, data size, list start and initial key) and check some values.  2. Decrypt the memory and iterate the key (`decrypt` function).  3. Walk through the linked list and gather 5 ints (`set` function).  4. "Decrypt" these 5 ints and print them together in ASCII as the flag.
I got my emulator to do points 1 and 2, and then switched to re-writing the assembly code to Python since I understood it enough at that point.
So here's the Python code step 2:
```pyimport structimport sysimport os
def decrypt_2(sz, key):    local_2 = 0    local_3 = 0    local_4 = 0    local_5 = 0
    while local_4 < (sz - 16) // 4:      #print(hex(local_4*4 + 16), hex(key))
      static_0[4 + local_4] ^= key
      key = key * 2      key &= 0xffffffff      if key == 0:        key = local_4 % 15 + 1
      local_4 += 1
    print("final key:", hex(key))
def dumpstatic():  global static_0  with open("dump_plain.static_0", "wb") as f:    #for i in static_0:      #print(i)    #    f.write(struct.pack(">I", i))    f.write(struct.pack(f">{len(static_0)}I", *static_0))
def main():  with open("memory.bin", "rb") as f:    d = f.read()
  global static_0  static_0 = list(struct.unpack(f">{len(d)//4}I", d))
  decrypt_2(0x4000, static_0[3])  dumpstatic()
main()```
And here's the code for step 3 and 4:
```pyimport sysimport structimport os
with open("step2.bin", "rb") as f:  d = f.read()
start = 0x1830  # This is the list start from the header.
# This is the loop in main + (set) function translated to Python:off = startvalues = []for i in range(5):  v_ptr, next = struct.unpack(">II", d[off:off+8])  v = struct.unpack(">I", d[v_ptr:v_ptr+4])[0]  values.append(v)  print(hex(v_ptr), hex(v), hex(next))  off = next
print(values)
# And this is the decryption of the values in (main) function.
"""GetLocalP 6SetLocal 14GetLocal 5Push 936175996AddSetLocal 15"""local_5 = 0xf000  # Final key from previous step.local_15 = local_5 + 936175996  # 0x37cce97cprint("key", hex(local_15))
"""GetLocal 14DuppGetGetLocal 15XorpPeekSetDrop
this has to be jctf  6a637466"""values[0] = values[0] ^ local_15
"""GetLocal 14GetImmP 1DuppGetGetLocal 15XorpPeekSetDropGetLocal 14GetImmP 1DuppGetPush 7536640OrpPeekSetDrop"""values[1] = (values[1] ^ local_15) | 0x730000
"""GetLocal 14GetImmP 2DuppGetGetLocal 15XorpPeekSetDropGetLocal 14"""values[2] = values[2] ^ local_15
"""GetImmP 3DuppGetGetLocal 15XorpPeekSetDropGetLocal 14GetImmP 3DuppGetPush 28416OrpPeekSetDrop"""values[3] = (values[3] ^ local_15) | 0x6f00
"""GetLocal 14GetImmP 4pGetPush -16777216AndPush 8192000OrGetLocal 14GetImmP 4pSetGetLocalP 6"""values[4] = ((values[4] & 0xff000000) | 0x7d0000)
# And printing out the values and the flag.o = ""for i in range(5):  hv = hex(values[i])[2:]  print(hv, b''.fromhex(hv))  o += hv
f = b''.fromhex(o)print(f)print(len(f))
# b'jctf{stickie_bomb}\x00\x00'```
All in all I didn't get the emulator fully running mostly because it wasn't clear based on the provided opcode description whether the memory is treated as a list of ints or list of bytes, and apparently some instructions were doing this and some that. After finding the SC-CL source and especially this – https://github.com/NativeFunction/SC-CL/blob/master/bin/include/intrinsics.h – it became a bit clearer to me, but I decided to not rework the emulator, so just did the steps 2-4 in Python.
All in all it was a fun challenge, though I probably should have spent some more time initially trying to find out what architecture it was, and maybe find an emulator.
Anyway, in the end it worked, and after asking the admins to fix the flag in their system ? I got first blood on it! :)
-- Gynvael