The basic approaches to reverse engineering

Figure 1: Reverse engineering with IDA ProAs I had started learning the reverse engineering, I was constantly lacking the information which would inform me about how to do it properly. There are many resources on the internet which are excellent for learning the basics, but I had some bad luck in finding those which would show me the entire reverse engineering sessions. This is what I would like to show in this series of write-ups.


Choosing a debugger to start learning the reverse engineering is an important step. It is important for a reverse engineer to learn the tools of his trade, which is usually done by meticulous use of said tools. If a tool is selected which doesn't have a long support plan, the engineer can come to a point where his tool becomes unusable in certain situations and he needs to adapt. While it is a usual state of things that tools get old and need switching, it is usually wise to select a good tool for the start to lessen the need for switching and relearning.


This happened to me. I was an avid user of Immunity debugger, which is an excellent tool but does not cover 64-bit programs. I had a good understanding of its scripting libraries and the shortcuts memorized. Now I'm switching to IDA, as it is the leading tool in disassembling and debugging these days. It also doesn't look like it's going away anytime soon.


Reversing basics


Approaches to learning reverse engineering


In this section, I list several of my own approaches to learning reversing and reversing in general. This is by no means the complete list, just my own personal experience.



  1. Be relentless. It is hard to start off with anything complicated, and believe me, reversing is complicated. There are many times when I felt really tired and deconcentrated after some time spent immersed into the debugged code, and even decided to just close the debugger and do something else. Reversing can drain you, so it is good to know this and not to give up because of it.

  2. Be meticulous. If you don't understand what a particular instruction does, or how some arithmetic works, research it. While learning, I believe it is imperative that you have the full understanding how things work, so that you have the required knowledge and, better yet, the required experience when the more complex projects come up. Once you feel comfortable with understanding the code and using the tool, you might get off with the complex cases.

  3. Try to start off with the static analysis. This will ensure you are trying to predict what the particular piece of code will do, thus pushing you harder. Don't be afraid of pen and paper, do your own calculations and then verify them. All this will benefit your experience and knowledge.

  4. When debugging, try to predict the outcome of your step. By this, I mean the step over a simple instruction or a well-defined function, not the function which is still unknown to you. This will help your debugging skills in the similar fashion as the point number three. When you do this, predict what the state of the registers will be, what parts of memory will change, and what the flags will look like when the instruction executes.

  5. Learn your tools. This stands true for any skill that requires tools. A skilled engineer concentrates on the job ahead of him without needing to focus on how he uses his tools to do the job. Putting it differently, he uses his tools automatically and subconsciously while keeping his concentration on the job.


In the end, I'd like to point out the truth which is true for any skill learning endeavor: the more you are mentally present while learning and the more you are trying hard, the better results will be.


Locating the part of interest


In any reversing session, the first order of business is to locate the part of interest. Notice how I stated 'the part' instead 'the code' or 'the function'. This is because these steps are very similar in any sort of the reverse engineering: the software, the hardware, the data, the signals etc.


This step can vary in complexity and required time depending on the reversed system's complexity and system designer's desire to make the reversing harder. In most cases, the system that is being reversed is not intended to be human readable. (An example where this is actually not the case is trying to reverse engineer an old legacy code to which we do have the access.)


Here are some examples how this step might look like in the real world:



  • reversing the code - we are searching the execution path branches for the method that we are interested in

  • reversing the program data - we are trying to locate the code which updates the memory locations that we are interested in.

  • reversing a protocol - we are searching to get as many data samples as possible

  • reversing the signals - we are trying to record the signals and retrieve the meaningful data from them (for instance, modulation and keying in wireless signals)


Reversing the item of interest


Once the item of interest has been found, the reverse engineer uses all the prowess to understand it. In many cases, there is no direct need to understand all the code details, but an overview is enough. The engineer can go through the function laying out the different library function calls and deduce the method behavior from that. The IDA debugger's graph view is a perfect tool for that.


On the other hand, there are times when the engineer needs to figure out the exact code behavior. An example would be reversing an algorithm.


Conclusion


This write-up was written in the hope that the people new to the reverse engineering skill might pick up some of my insights about it, comment them, or add some of their own. As this is a purely subjective approach, I would really like to see some good discussion.


The following articles in this series will try to showcase some of my reversing sessions in the hope that someone might get some interesting and helpful information (and that I might get some good feedback).

Reverse Engineering