First, download the `math` file. It's likely a Linux executable, since this is a reversing challenge, but let's run `file ./math` to be sure:```./math: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=71fce45eb1092a34d4e5f72881177f3d14d22be1, for GNU/Linux 3.2.0, not stripped```We know that this is a executable now. What does it do? Let's try it: ```Question #1: What is 1+1?Question #2: How many sides does a square have?Question #3: What is 8*30?Question #4: What is the remainder when 39 is divided by 4?Question #5: What is the 41st fibbonaci number?Question #6: How many questions are on this test?Question #7: What number am I thinking of?Question #8: What is the answer to this question?Question #9: Prove that the answer to the previous question is correct.Question #10: Give me an integer.You got [NUMBER OF CORRECT ANSWERS] out of 10 right!If you get a 10 out of 10, I will give you the flag!```Note that I've ommitted my answers and the number of correct answers, which I've replace with `[NUMBER OF CORRECT ANSWERS]`. (Partly because the results are different for different people, and partly because I don't want people seeing my score.)
Let's look at what the assembly instructions are with`objdump -Mintel -D ./math`.Once we have it disassembled, we can look at the assembly code and the data inside the executable. Let's start from `main`:```assembly0000000000001385 <main>:    1385:       55                      push   rbp    1386:       48 89 e5                mov    rbp,rsp    1389:       48 83 ec 10             sub    rsp,0x10    138d:       48 8d 05 5c 0e 00 00    lea    rax,[rip+0xe5c]        # 21f0 <_IO_stdin_used+0x1f0>    1394:       48 89 c7                mov    rdi,rax    1397:       e8 b4 fc ff ff          call   1050 <puts@plt>    139c:       48 8d 05 9d 0e 00 00    lea    rax,[rip+0xe9d]        # 2240 <_IO_stdin_used+0x240>    13a3:       48 89 c7                mov    rdi,rax    13a6:       e8 a5 fc ff ff          call   1050 <puts@plt>    13ab:       c7 45 fc 00 00 00 00    mov    DWORD PTR [rbp-0x4],0x0    13b2:       eb 6e                   jmp    1422 <main+0x9d>    13b4:       8b 45 fc                mov    eax,DWORD PTR [rbp-0x4]    13b7:       83 c0 01                add    eax,0x1    13ba:       89 c6                   mov    esi,eax    13bc:       48 8d 05 b4 0e 00 00    lea    rax,[rip+0xeb4]        # 2277 <_IO_stdin_used+0x277>    13c3:       48 89 c7                mov    rdi,rax    13c6:       b8 00 00 00 00          mov    eax,0x0    13cb:       e8 90 fc ff ff          call   1060 <printf@plt>    13d0:       8b 45 fc                mov    eax,DWORD PTR [rbp-0x4]    13d3:       48 98                   cdqe    13d5:       48 8d 14 c5 00 00 00    lea    rdx,[rax*8+0x0]    13dc:       00    13dd:       48 8d 05 1c 2f 00 00    lea    rax,[rip+0x2f1c]        # 4300 <questions>    13e4:       48 8b 04 02             mov    rax,QWORD PTR [rdx+rax*1]    13e8:       48 89 c7                mov    rdi,rax    13eb:       e8 60 fc ff ff          call   1050 <puts@plt>    13f0:       8b 45 fc                mov    eax,DWORD PTR [rbp-0x4]    13f3:       48 98                   cdqe    13f5:       48 8d 14 c5 00 00 00    lea    rdx,[rax*8+0x0]    13fc:       00    13fd:       48 8d 05 7c 2f 00 00    lea    rax,[rip+0x2f7c]        # 4380 <submitted>    1404:       48 01 d0                add    rax,rdx    1407:       48 89 c6                mov    rsi,rax    140a:       48 8d 05 75 0e 00 00    lea    rax,[rip+0xe75]        # 2286 <_IO_stdin_used+0x286>    1411:       48 89 c7                mov    rdi,rax    1414:       b8 00 00 00 00          mov    eax,0x0    1419:       e8 52 fc ff ff          call   1070 <__isoc99_scanf@plt>    141e:       83 45 fc 01             add    DWORD PTR [rbp-0x4],0x1    1422:       8b 05 38 2c 00 00       mov    eax,DWORD PTR [rip+0x2c38]        # 4060 <NUM_Q>    1428:       39 45 fc                cmp    DWORD PTR [rbp-0x4],eax    142b:       7c 87                   jl     13b4 <main+0x2f>    142d:       b8 00 00 00 00          mov    eax,0x0    1432:       e8 93 fe ff ff          call   12ca <grade_test>    1437:       90                      nop    1438:       c9                      leave    1439:       c3                      ret    143a:       66 0f 1f 44 00 00       nop    WORD PTR [rax+rax*1+0x0]```We can see that it does a lot of stuff, which is probably asking the question since we see a lot of `printf` and `scanf`s, then calls `grade_test` and exits.Now we want to look at `grade_test` since it probably has information about the answers (unless it's a decoy by the organizers, which it probably isn't):```assembly00000000000012ca <grade_test>:    12ca:       55                      push   rbp    12cb:       48 89 e5                mov    rbp,rsp    12ce:       48 83 ec 10             sub    rsp,0x10    12d2:       c7 45 fc 00 00 00 00    mov    DWORD PTR [rbp-0x4],0x0    12d9:       c7 45 f8 00 00 00 00    mov    DWORD PTR [rbp-0x8],0x0    12e0:       eb 3d                   jmp    131f <grade_test+0x55>    12e2:       8b 45 f8                mov    eax,DWORD PTR [rbp-0x8]    12e5:       48 98                   cdqe    12e7:       48 8d 14 c5 00 00 00    lea    rdx,[rax*8+0x0]    12ee:       00    12ef:       48 8d 05 8a 30 00 00    lea    rax,[rip+0x308a]        # 4380 <submitted>    12f6:       48 8b 14 02             mov    rdx,QWORD PTR [rdx+rax*1]    12fa:       8b 45 f8                mov    eax,DWORD PTR [rbp-0x8]    12fd:       48 98                   cdqe    12ff:       48 8d 0c c5 00 00 00    lea    rcx,[rax*8+0x0]    1306:       00    1307:       48 8d 05 72 2d 00 00    lea    rax,[rip+0x2d72]        # 4080 <answers>    130e:       48 8b 04 01             mov    rax,QWORD PTR [rcx+rax*1]    1312:       48 39 c2                cmp    rdx,rax    1315:       75 04                   jne    131b <grade_test+0x51>    1317:       83 45 fc 01             add    DWORD PTR [rbp-0x4],0x1    131b:       83 45 f8 01             add    DWORD PTR [rbp-0x8],0x1    131f:       8b 05 3b 2d 00 00       mov    eax,DWORD PTR [rip+0x2d3b]        # 4060 <NUM_Q>    1325:       39 45 f8                cmp    DWORD PTR [rbp-0x8],eax    1328:       7c b8                   jl     12e2 <grade_test+0x18>    132a:       8b 45 fc                mov    eax,DWORD PTR [rbp-0x4]    132d:       89 c6                   mov    esi,eax    132f:       48 8d 05 34 0e 00 00    lea    rax,[rip+0xe34]        # 216a <_IO_stdin_used+0x16a>    1336:       48 89 c7                mov    rdi,rax    1339:       b8 00 00 00 00          mov    eax,0x0    133e:       e8 1d fd ff ff          call   1060 <printf@plt>    1343:       83 7d fc 0a             cmp    DWORD PTR [rbp-0x4],0xa    1347:       75 2a                   jne    1373 <grade_test+0xa9>    1349:       48 8d 05 37 0e 00 00    lea    rax,[rip+0xe37]        # 2187 <_IO_stdin_used+0x187>    1350:       48 89 c7                mov    rdi,rax    1353:       e8 f8 fc ff ff          call   1050 <puts@plt>    1358:       48 8d 05 45 0e 00 00    lea    rax,[rip+0xe45]        # 21a4 <_IO_stdin_used+0x1a4>    135f:       48 89 c7                mov    rdi,rax    1362:       e8 e9 fc ff ff          call   1050 <puts@plt>    1367:       b8 00 00 00 00          mov    eax,0x0    136c:       e8 58 fe ff ff          call   11c9 <generate_flag>    1371:       eb 0f                   jmp    1382 <grade_test+0xb8>    1373:       48 8d 05 3e 0e 00 00    lea    rax,[rip+0xe3e]        # 21b8 <_IO_stdin_used+0x1b8>    137a:       48 89 c7                mov    rdi,rax    137d:       e8 ce fc ff ff          call   1050 <puts@plt>    1382:       90                      nop    1383:       c9                      leave    1384:       c3                      ret```Now we see a reference to `answers`, which sounds like it might have some interesting information. If we follow the assembly code, we can see that it's loading the pointer to `answers` into `rax`, and then setting rax to the value at the address `rax + rcx`. 
So what is this offset `rcx`? Well, if we look at location `12ff`, and the context (i.e. the instructions before it), we can see that it's the loop counter times 8. (If you don't get this, follow the program and see what numbers are being assigned to the registers and memory and what the program is doing with them.) 
Now why would it be multiplied by 8? Well this might signal that the answers are 8 bytes long integers. So let's examine what `answers` exactly is:```assembly0000000000004080 <answers>:    4080:       02 00                   add    al,BYTE PTR [rax]    4082:       00 00                   add    BYTE PTR [rax],al    4084:       00 00                   add    BYTE PTR [rax],al    4086:       00 00                   add    BYTE PTR [rax],al    4088:       04 00                   add    al,0x0    408a:       00 00                   add    BYTE PTR [rax],al    408c:       00 00                   add    BYTE PTR [rax],al    408e:       00 00                   add    BYTE PTR [rax],al    4090:       f0 00 00                lock add BYTE PTR [rax],al    4093:       00 00                   add    BYTE PTR [rax],al    4095:       00 00                   add    BYTE PTR [rax],al    4097:       00 03                   add    BYTE PTR [rbx],al    4099:       00 00                   add    BYTE PTR [rax],al    409b:       00 00                   add    BYTE PTR [rax],al    409d:       00 00                   add    BYTE PTR [rax],al    409f:       00 6d 8d                add    BYTE PTR [rbp-0x73],ch    40a2:       de 09                   fimul  WORD PTR [rcx]    40a4:       00 00                   add    BYTE PTR [rax],al    40a6:       00 00                   add    BYTE PTR [rax],al    40a8:       0a 00                   or     al,BYTE PTR [rax]    40aa:       00 00                   add    BYTE PTR [rax],al    40ac:       00 00                   add    BYTE PTR [rax],al    40ae:       00 00                   add    BYTE PTR [rax],al    40b0:       87 15 59 00 00 00       xchg   DWORD PTR [rip+0x59],edx        # 410f <flag+0x2f>    40b6:       00 00                   add    BYTE PTR [rax],al    40b8:       49 1e                   rex.WB (bad)    40ba:       a1 06 00 00 00 00 5f    movabs eax,ds:0x60e95f0000000006    40c1:       e9 60    40c3:       20 00                   and    BYTE PTR [rax],al    40c5:       00 00                   add    BYTE PTR [rax],al    40c7:       00 09                   add    BYTE PTR [rcx],cl        ...```Now you might see some instructions on the right side of this disassembly, but if you look at them what they're doing doesn't make sense. That's because the label (which is `answers`) isn't supposed to be instructions! They're just random instructions that just happen to correspond to the data in `answers` (there's a reason it's in the label `.data`!). 
This is because we used the `-D` flag which disassembles everything, instead of `-d` which disassembles executable sections, but doesn't show data sections. (There are flags to make `objdump` with `-d`, but imo it's simpler to just use `-D` here. Your choice)
Ok, so back to what we were doing. Let's look at the data inside `answers`, We see `2`, a bunch of `0` bytes, then `4`... well, if we look at the first few questions, we can obviously see that the answers are `2` and `4`. (unless you're a toddler. but then how are you reading this? pls don't spam comments thx)
Now we can see that each answer is 8 bytes long because there are a bunch of `0`s after the first few answers, and `0` wouldn't make sense as answers. Also, we saw that the loop count was being multiplied by `8` earlier, so it makes sense. 
Now we can find the answers! They're just 8 byte long integers. But we have to make sure to get the bytes in the right order, since x86 is little endian. So we just need to reverse the byte order for each integer.
Doing this, we get the answers:1. `2`2. `4`3. `0xf0=240`4. `3`5. `0x09de8d6d=165580141`6. `0x0a=10`7. `0x0x591587=5838215`8. `0x06a11e49=111222345` (this is one of those things where you feel really stupid after seeing the answer)9. `0x2060e95f=543222111`10. `9` (there aren't any `0`s after this so we can just assume that there's nothing else)
Now when we enter the answers (as decimal), we get the flag! It's `LITCTF{y0u_must_b3_gr8_@_m4th_i_th0ught_th4t_t3st_was_imp0ss1bl3!}`. Note that we couldn't have gotten the flag using purely math to answer the questions. (Well, technically everything is applied math, but I'm saying pure math here.)
You might ask, why couldn't we have just edited the code to generate the flag directly or faked it by using a debugger and modifying stuff to pass the checks?Well, if we look inside the `generate_flag` function, we see that it's computing the flag, and something named `submitted` is involved. Well, if we look through, we find out that this is the place where your inputted answers are stored! So the flag DEPENDS on the answers, and you can't get it without finding the answers in some way.