This is the only task that features a Win32 binary. A Win32 debugger is not strictly required but quite useful to test hypotheses.
The file ends with a block of data at 0x68000 that is outside of PE sections and thus not loaded with rest of the binary. The block starts with a string "1-abc.net File EncrypteR" and continues with high-entropy data which means some compression and/or encryption. The magic string allows to quickly find the handler 0x4027B0 inside the executable as well as Web site of the encryptor with the following feature overview:> * Easy-to-use interface> * Encrypt and decrypt files and folders on your hard drive, floppy disk or USB stick completely> * 11 encryption methods (Simple HEX Encoding, Ceasar-3, Ceasar-9, Simple XOR, Extended XOR, Advanced Encryption Standard (AES), Blowfish, Twofish, Serpent, MARS and Tiny Encryption Algorithm (TEA))> * 3 easy encrypting procedures (no password required)> * 8 secure encrypting procedures (password required)> * Encrypt your data to a self-extracting *.exe file for users who do not have this program installed on their systems> * Log your encrypting and decrypting activity to the history file by option> * Start program with Windows automatically and encrypt data from Windows Explorer directly> * Program can be installed on USB sticks, floppy disks or nearly all other re-writable media> * Very low system resources required
Mentioning hex-encoding, XOR and AES as equals suggests that the authors don't fully understand cryptography.
After comparing the signature, the code processes next bytes as the header:* space 0x20 seems to be ignored* byte 0x79 = 'y' seems to be the version marker that can be either 'x' or everything else* two next bytes, 0x3B and 0x3A, are some sort of checksum for the password* 0x14 is the length of the password* 0x05 encodes the encryption method (the encoding depends on the version marker)* then, the actual encrypted data begin
The checksum is not anything standard and not cryptographical, just a mix of simple arithmetic operations over bytes of the password. I think it is possible to generate a password with given checksum, but it would be a waste of time; two bytes can't tell much about 20-bytes password anyway, and while tracing in a debugger, it is easier to just skip the check.
The encryption method chosen in this particular file expands the password to 32 bytes by concatenating it to itself (several times, if needed). Then, it reads the data in pieces of 100 (0x64) bytes and passes each piece to the function at 0x4037B0 along with the expanded password. That function generates two values, first 16 bytes and reversed last 16 bytes, and calls several other functions where a code style is drastically different, with VMTs and exceptions. RTTI is enabled, so VMTs come with extra type pointer before function pointers that show two type names (shared by five VMTs):```.?AV?$BlockCipherFinal@$0A@VEnc@Rijndael@CryptoPP@@@CryptoPP@@.?AV?$CipherModeFinalTemplate_CipherHolder@V?$BlockCipherFinal@$0A@VEnc@Rijndael@CryptoPP@@@CryptoPP@@V?$ConcretePolicyHolder@VEmpty@CryptoPP@@V?$CFB_DecryptionTemplate@V?$AbstractPolicyHolder@VCFB_CipherAbstractPolicy@CryptoPP@@VCFB_ModePolicy@2@@CryptoPP@@@2@VCFB_CipherAbstractPolicy@2@@2@@CryptoPP@@```...in other words, something about Rijndael aka AES and CFB. A reasonable guess is that the function at 0x4037B0 implements AES decryption in CFB mode, and key and iv are taken from the expanded password (and a debugger confirms the guess, decryption results using a random password match decryption results in Python REPL).
This means that every 100-bytes piece is decrypted using the same key and IV. Looking at encrypted data more closely, we can see that the second and third pieces are exactly the same. We know that original data are some XLS file; new-style Office files (for some value of "new") are just ZIPs so should not have large repeated blocks, but old-style Office files have something like bitmap of free/allocated blocks, for not-very-large files this bitmap contains a lot of FFs near the beginning of a file. Since the second and third pieces are the same, they both are probably an encrypted all-FF piece.
In CFB, the first encrypted block is `Encrypt(IV) XOR plaintext[0]`, the second encrypted block is `Encrypt(ciphertext[0]) XOR plaintext[1]` and so on. This means that we can calculate `Encrypt(IV)` from a pair of known ciphertext/plaintext and then decrypt first block of every piece:```f = open('password.xls.exe', 'rb')f.seek(0x6801E)data = f.read()enciv = bytearray(16)for i in range(16):	enciv[i] = data[100+i] ^ 0xFFfor i in range(len(data) // 100):	dec = bytes(data[i*100 + j] ^ enciv[j] for j in range(16))	dec_printable = ''.join(chr(x) if 0x20 <= x <= 0x7E else '.' for x in dec)	print(dec.hex(), dec_printable)```This is enough to find the flag in the output:```d0cf11e0a1b11ae10000000000000000 ................ffffffffffffffffffffffffffffffff ................ffffffffffffffffffffffffffffffff ................ffffffffffffffffffffffffffffffff ................ffffffffffffffffffffffffffffffff ................fffffffffffffffffffffffffdffffff ................ffffffffffffffffffffffffffffffff ................ffffffffffffffffffffffffffffffff ................ffffffffffffffffffffffffffffffff ................ffffffffffffffffffffffffffffffff ................ffffffffffffffffffffffffffffffff ................ffffffff000000000000000000000000 ................00000000000000000000000000000000 ................00000000000000000000000000000000 ................00000000000000000000000000000000 ................00000000000000000000000000000000 ................11000000120000001300000014000000 ................2a0000002b0000002c0000002d000000 *...+...,...-...ffffffffffffffffffffffffffffffff ................ffffffffffffffffffffffffffffffff ................ffffffffffffffffffffffffffffffff ................2020202020202020202020202020202042000200b004610102000000c0010000 B.....a.........1a00c8000000ff7f9001000000020000 ................00000000000005014100720069006100 ........A.r.i.a.00000000c020e000140002000000f5ff ..... ..........c020e000140000000000f5ff200000f4 . .......... ...140000000000f5ff200000f400000000 ........ .......a4000100200000000000000000000000 .... ...........200000f00000000000000000c020e000  ............ ..07ff93020400148005ff600102000000 ..........`.....33000bf012000000bf00080008008101 3...............4354467b6e306e43335f52337535337d CTF{n0nC3_R3u53}6976652e636f6d0a0000726564646974 ive.com...reddit0900007477697463682e747609000061 ...twitch.tv...a0a0000000908100000061000bb0dcc07 ................02000100140023002000002643262254 ......#. ..&C&"T000800333333333333e93f2700080033 ...333333.?'...30b0f000000000000020e000000000014 ................0000010f000802100004000000020000 ................0000010f000802100009000000020000 ................0000010f00080210000e000000020000 ................0000010f000802100013000000020000 ................000100150001000000fd000a00030000 ................000f0007000000fd000a000600010015 ................0001000000fd000a000a0000000f000b ................000000fd000a000d0001001500010000 ................00fd000a00110000000f0012000000fd ................f00800000001000000000400000f0003 ................00000100010001000000670817006708 ..........g...g.ffffffff1008020000000000c0000000 ................00000000000000000000000000000000 ................000000000000000001000000e0859ff2 ................030000003130000040000000802db794 [email protected]..0200000002d5cdd59c2e1b1093970800 ................00000000000000000000000000000000 ................00000000000000000000000000000000 ................ffffffffffffffff0100000010080200 ................00000000000000000000000000000000 ................62006a00000000000000000000000000 b.j.............000000002e0000004900000000000000 ........I.......00000000000000000000000000000000 ................000000000000000028000200ffffffff ........(.......7200790049006e0066006f0072006d00 r.y.I.n.f.o.r.m.00000000000000000000000000000000 ................00000000000000000000000000000000 ................```