I'd like to manually "pull out" information of files of a specific file format. Needless to say, they were encoded in some form or another - I couldn't access a "plaintext" version of them. However, the program associated with the file obviously is able to read them - And I was wondering if I could somehow reverse engineer that program and thus find out how they're being encoded, to read the content myself.

It's a broad question - But I'm not sure where else to start.

  • Hello, try to specify what OS and what architecture the software is using! Access to a file is strongly linked to the OS and giving the architecture used is always a good idea. – perror Aug 16 '17 at 7:47
  • Which OS? Which file format? Maybe there's already a parser available online. – Megabeets Aug 16 '17 at 7:53

This is what I would do given the very limited information:

  1. Start by examining the file in a hex editor like HxD or 010 Editor. Take note of any "constants." For example, all Windows exe files start with the bytes 4D 5A or "MZ". Do these files start with a consistent byte signature? Scroll down in the hex editor and find out what other patterns you can find.
  2. Use a tool such as Detect It Easy or PortEx Analyzer to measure the entropy on the file's bytes and/or map them out. Is the entropy very high such that the tool indicates the file is packed, or is it lower? This can give you clues on whether or not is encoded and perhaps even what type of encoding may be used.

I would take the above information gathered and search the Internet for any constants or patterns that I found across the files. See if you can find anyone else discussing these files or any programs which use them. If you can find other programs using the files, try and locate source code or something on GitHub.

As you said, this is open-ended and there isn't much room for discussion here other than comments, but this is how I would start the process. Do you know which program is encoding the files? If so, get ahold of the program's executable and use a tool like PEiD KANAL and/or signsrch on it to search for common encoding algorithms, then get the addresses of those algorithms and open the program executable in a disassembler and examine the memory addresses. Last but not least, another trick you can use is if you can figure out which type of info may be in the file (such as URLs), you can use the Didier Stevens tool called XORSearch and search for "http" and the program will try a bunch of different XOR keys for that pattern and if it is able to crack an http somewhere, you may be able to find out the encoding that way.

| improve this answer | |
  • Thank you very much for your answer, it was very helpful! When viewing the file in the hex editor, it actually looked to me as if the "decoded" file was in plain JSON format (A first part of the file had small chunks of json in them, and curly brackets were scattered throughout the file). However, the "weird symbols" were still scattered throughout the file, so I'm not sure how to decode that. I already googled a lot, and the file is only used by that one program, which happens to be a game.. I'll try out to use the encoding-algorithm-searching tools, thanks a lot! – user2065501 Aug 16 '17 at 10:26
  • @user2065501 keep in mind, all the "weird symbols" mean is that you are looking at a binary file (rather than a text file) even if there is indeed some plaintext scattered throughout. This is totally normal and there's no guarantee that it is actually encrypted because of this. Measure the entropy of the file and that will tell you more. – the_endian Aug 16 '17 at 16:48

I would advise you to use a dynamic analysis of the software by tracing it using tools such as the Intel PIN tool (I am assuming you are dealing with x86 assembly).

With this tool you can trace each executed instruction by the software (see this tutorial on PIN).

Then, your point will be to find system call that open the correct file (one of the parameter of the system call will be the path to the file, so you should be able to find it with a simple grep).

Once you have located the instruction that open the file, I would advise you to start disassembling from this point (use software such as IDA pro, Radare2, or even gdb/Ollydbg could do it).

| improve this answer | |
  • That sounds very logical, thanks a lot for the advice! While I'm very very new at all of this (Even though I've been programming for a long time, I've barely touched assembly up to this point), I'll try to do my best. – user2065501 Aug 16 '17 at 10:28

Your Answer

