Reverse-Engineering &
Exploitation
Fundamentals

GCC 2022

chick with laptop

whoarewe

lord idiot chicken

GCC 2018 Alumni

Pwn2Own 2021

daniao egg

Routers

macOS

whoareyou

  • Some programming experience (variables, loops, if-else, functions)
  • No security experience expected

Training Outline

  1. Assembly Basics (x86-64)
  2. Linux Reverse Engineering
  3. Buffer Overflows

What is Reverse-Engineering?

... the goal of a reverse engineering ... is to understand the functionality of a given program such that you can identify deeper issues.

~ ctf101.org

Is RE useful?

Learning reverse-engineering isn't just for those juicy CTF points

Game Hacking

Games are commonly closed-source and RE is required to understand their inner workings

Bug Hunting

Reverse-engineering a software to understand how it works at a low-level allows us to uncover software bugs

Malware

Reverse-engineering operating systems to understand how to best create malware / Reverse-engineering malware
Let's start with building blocks ...

Assembly Language (x86-64)

chick building blocks

How do programs run in the computer?

An analogy for the CPU

An introduction to Bob

Bob can only understand

... simple instructions 

fill <bowl>, <ingredient/bowl>
stir <bowl>
chop <bowl>
boil <bowl>, <time>

Let's prepare fried rice!


; cook rice
fill     A, rice
fill     A, water
boil     A, 18 minutes
stir     A

; ingredients
fill     B, onions
fill     B, carrots
chop     B
boil     B, peas

; mix to fry
fill     B, A

Too verbose

More complicated recipes would be way too tedious to write
The solution ... ?

Compilers!

Let's welcome Alice

Alice is far smarter than Bob

Alice can compile higher-level instructions into a format that Bob will be able to understand

Recipe Compilation

Alice


1. Boil rice for 18 minutes
2. Chop onions and carrots
3. Mix the onions, carrots
   and peas with the rice

Bob


; cook rice
fill     A, rice
fill     A, water
boil     A, 18 minutes
stir     A

; ingredients
fill     B, onions
fill     B, carrots
chop     B
boil     B, peas

; mix to fry
fill     B, A

The same is done for programs!

Program Compilation

Alice


int main(){
    int a = 1337;
    int b = 31337;
    return a+b;
}

Bob


push    rbp
mov     rbp, rsp
mov     DWORD PTR [rbp-4], 1337
mov     DWORD PTR [rbp-8], 31337
mov     edx, DWORD PTR [rbp-4]
mov     eax, DWORD PTR [rbp-8]
add     eax, edx
pop     rbp
ret

Program Compilation

gcc/compiler


int main(){
    int a = 1337;
    int b = 31337;
    return a+b;
}

x86-64 cpu


push    rbp
mov     rbp, rsp
mov     DWORD PTR [rbp-4], 1337
mov     DWORD PTR [rbp-8], 31337
mov     edx, DWORD PTR [rbp-4]
mov     eax, DWORD PTR [rbp-8]
add     eax, edx
pop     rbp
ret

Internally ...

x86-64 mnemonic


push    rbp
mov     rbp, rsp
mov     DWORD PTR [rbp-4], 1337
mov     DWORD PTR [rbp-8], 31337
mov     edx, DWORD PTR [rbp-4]
mov     eax, DWORD PTR [rbp-8]
add     eax, edx
pop     rbp
ret

hex representation


55
48 89 E5
C7 45 FC 39 05 00 00
C7 45 F8 69 7A 00 00
8B 55 FC
8B 45 F8
01 D0
5D
C3

What are instructions?

Flow of execution

chick pointing at a recipe with a ruler that is labelled RIP

A simple snippet

Baby's first instruction!

nop

Syntax:
nop
Functionality:
Does nothing.
Sample Code

nop
nop
nop

nop

A short tangent.

What are registers and memory?

Continuing the analogy...

Memory

mov

Syntax:
mov <dst> <src>
Functionality:
"Moves" the value from <src> to <dst>
Sample Code

mov     rax, 0x1337
mov     rbx, 0x2022
mov     rcx, rbx

mov

mov (memory)

add/sub

Syntax:
add <dst> <src>
Functionality:
Adds the value from <src> to <dst>, result in <dst>
Sample Code

mov     rax, 0x100
mov     rbx, 0x200
add     rax, rbx

add/sub

Mini Challenge #1

Mini Challenge #2

What about more complex code?

e.g.


if(some_condition){
	do_something();
}
else {
	do_something_else();
}

jmp

Syntax:
jmp <dst>
Functionality:
Jumps to execute code at <dst>
Sample Code

mov     rax, 0x100
jmp     skip
mov     rax, 0x1337
skip:
nop
nop

jmp

cmp

Syntax:
cmp <a>, <b>
Functionality:
Compares the values of <a> and <b> and sets EFLAGS
Sample Code

mov     rax, 0x100
mov     rbx, 0x200
cmp     rax, rbx

jmp (cond)

Syntax:
jcc <dst>
Functionality:
Does a conditional jump to <dst> based on EFLAGS
Sample Code

mov     rax, 0x100
mov     rbx, 0x200
cmp     rax, rbx
mov     rcx, rax
jl      rbx_lesser
mov     rcx, rbx
rbx_lesser:
nop

jmp (cond)

call

Syntax:
call <dst>
Functionality:
Calls the function located at <dst>
Sample Code

nop
call    foo
hlt
foo:
mov     rax, 0x1337
ret
nop

call

Mini Challenge #3