Introduction to Reverse Engineering Software in Linux

http://www.ouah.org/RevEng/

Introduction to Reverse Engineering Software in Linux

Revision History
Revision $Revision: 1.26 $	$Date: 2002/09/18 06:54:57 $

Abstract

This document is an attempt to provide an introduction to reverse engineering software in Linux. Since reverse engineering is rapidly coming under legal fire, this author figures the best response is to make the knowledge widespread. The idea is that since discussing specific reverse engineering feats is illegal, we should then discuss general approaches, so that rather than downloading cracks or describing weaknesses for programs (yes, BOTH are now illegal), it is within every Linux user's ability to make them. Also, closed source programs piss me off. Resistance is futile. You will be Open Sourced.

---

Table of Contents

1. Introdution

1.1. What is reverse engineering?

1.2. Why reverse engineer?

1.3. Legal issues

2. The Linux Compilation Process

2.1. Intro

2.2. gcc

2.3. gcc -E (Preprocessor Stage)

2.4. gcc -S (Parsing+Translation Stages)

2.5. as (Assembly Stage)

2.6. ld/collect2 (Linking Stage)

3. Gathering Info

3.1. ldd

3.2. nm

3.3. /proc

3.4. netstat

3.5. lsof

3.6. fuser

4. Determining Program Behavior

4.1. strace/truss(Solaris)

4.2. ltrace

4.3. LD_PRELOAD

4.4. gdb

5. Determining Interesting Functions

5.1. Reconstructing function & control information

5.2. Consider the objective

5.3. Finding key functions

5.4. Plotting out program flow

6. Understanding Assembly

6.1. Registers

6.2. The stack

6.3. Two's complement

6.4. Reading Assembly

6.5. Know Your Compiler

7. Writing Standalone Assembly

7.1. Instructions with side-effects

7.2. Opcode Tables

7.3. Using GNU as

7.4. Conventions on saving registers

7.5. Using Library Functions

8. Working with the ELF Program Format

8.1. ELF Layout

8.2. Editing ELF

9. Understanding Copy Protection

10. Code Modification

10.1. Reasons for Code Modification

10.2. Instruction Modification

10.3. Single Instruction Insertion

10.4. Single Function Insertion

10.5. Multiple Function Insertion

10.6. Attacking copy protection

11. Buffer Overflows

11.1. Stack Overflows

11.2. 1-Byte Overflows

11.3. Returning to Libc

11.4. Attacking Countermeasures

11.5. Heap Overflows

11.6. Attacking hard copy protection

12. TODO (Contribute!)

12.1. Write assembly tutorial section

12.2. Create Diagrams & example outputs

12.3. More detail

12.4. Update disasm.pl

12.5. Do this for windows

12.6. Do this for protocols

12.7. Do this for hardware

13. Extra Resources

13.1. ELF Binary Specification

13.2. Other Resources and amusements

1. Introdution

1.1. What is reverse engineering?

Reverse engineering as this document will discuss it is simply the act of figuring out what software that you have no source code for does.

1.2. Why reverse engineer?

Answer: Because you can. Software that exists on your system that you do not have source code to is usually the most interesting kind of software. Sometimes you may be looking for security holes, sometimes you are curious how copy protection works, etc. I don't know about you, but to me, software that I don't have sourcecode to just pisses me off. So I figure: screw it, lets do some damage.

Also, it makes you a better programmer. This book will teach you a large amount about how your computer works on a low level, and the better an understanding you have of that, the more efficient programs you can write in general.

If you don't know assembly language, at the end of this book you will literally know it inside-out. While most first courses and books on assembly language teach you how to use it as a programming language, you will get to see how to use C as an assembly language generation tool, and how to look at and think about assembly as a C program. This puts you at a tremendous advantage over your peers not only in terms of programming ability, but also in terms of your ability to figure out how the black box works. In short, learning this way will naturually make you a better reverse engineer. Plus, you will have the fine distinction of being able to answer the question"Who taught you assembly language?" with "Why, my C compiler, of course!"

1.3. Legal issues

Pending...