Meditation, The Art of Exploitation

Thinking? At last I have discovered it--thought; this alone is inseparable from me. I am, I exist--that is certain. But for how long? For as long as I am thinking. For it could be, that were I totally to cease from thinking, I should totally cease to exist....I am, then, in the strict sense only a thing that thinks.

Tuesday, May 16, 2006

Noteworthy i386 listing of frequently used subroutines and Typical function call protocol

Originally composed on June 13, 2004, editted formatting.


1. String length calculation and key comparison
mov ecx, -1 ; mov ecx, xx (xx=max possible length)
mov al, 0 ;xor eax, eax;sub eax, eax;mov eax, 0
mov edi, string_offset ; lea edi, [ebp+XX]
; ES:EDI -> string
repnz scasb ; repnz scasw
neg ecx ; xor eax, eax
sub eax, ecx
mov edi, real string ; lea edi, [ebp+XX]
mov esi, user string
repnz cmpsb ; repeat till ecx = 0 or [edi]!=[esi]
test ecx, ecx
jnz bad
xor eax, eax ; using eax=0 as good, could also use eax=1 as bad
jmp good
bad:
mov eax, 1
ret
good
do_good_stuff ; update registry, update ini file, update memory
ret

2. A==0? 1:0 translated to assembly
mov eax, A
neg eax ; eax = 0 - eax, unsigned, but sets CF if eax > 0
sbb eax, eax ; eax = eax - eax - CF
inc eax
ret
if A == 0,
neg eax => eax = 0, CF = 0
sbb eax, eax => eax = 0
inc eax => eax = 1
else
neg eax => eax = (UnSigned)-A, CF = 1
sbb eax, eax => eax = -1
inc eax => eax = 0
endif

An alternative form A == 0? 0:1
mov eax, A
cmp eax, 1 ; set CF if eax = 0, CF = 0 if A != 0
sbb eax, eax ; eax = -1 if A = 0, eax = 0 if A != 0
inc eax ; eax = 0 if A = 0, eax = 1 if A != 0
ret

A little bit mind boggling, isn't it? :-)
These are common code signatures you will find in assembly language
generated from high level language (typically C/C++) from a compiler.
A final note, the counterpart of sbb is adc (add with carry). As an

excersise, build A == 0? 1:0 using abc instead of sbb.

3. The __cdecl, __stdcall, __fastcall, thiscall signatures and
__declspec(naked)
(Reference: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclang/html/_core___stdcall.asp)

Normally a subroutine written in C/C++ is translated with the following
signatures, take
int sub_a(arg1, arg2, arg3)
call sub_a(arg1, arg2, arg3)

there is 4 ways to declare its prototype that will affect generated
assembly code.

3.1)
__cdecl:
This is the default calling convention for C and C++ programs.
Because the stack is cleaned up by the caller, it can do vararg functions.
The __cdecl calling convention creates larger executables than __stdcall,
because it requires each function call to include stack cleanup code.
The following list shows the implementation of this calling convention.

Element Implementation
Argument-passing order Right to left
Stack-maintenance responsibility Calling function pops the arguments from
the stack Name-decoration convention Underscore character (_) is prefixed to
names Case-translation convention No case translation performed

So what does all this babblying mean when it's in action?
Callee:
int __cdecl sub_a(arg1, arg2, arg3)
push ebp
mov ebp, esp
sub esp, 4 x number_of_local_automatic_variables <-- normally the case
; pointers could complicate it, and structure alignment requirement
; could also make the space required on stack seem strange.
; nevertheless, automatica variables are stored on stack.
; name it NOLAV: # of local automatic variables
arg1 = [ebp+8] ; Here we assume it's a near call (*flat memory model)
arg2 = [ebp+C] ; within the caller's own code segment linear space
arg3 = [ebp+10] ; normally true but not for some trickery code ...
add esp, 4 x NOLAV
; ebp+10 <- arg3 || stack high pop ebp
; ebp+0C <- arg2 || ret
; ebp+8 <- arg1 ||
; ebp+4 <- eip of the return address ||
; ebp <- previous ebp ||
<= esp = ebp pointer \/ stack low

Caller: call sub_a(arg1, arg2, arg3)
push arg3 ; see below
push arg2
push arg1
call sub_a
add esp, 0C

now it's of course simplified because arg3 cannot be directly referrenced in
assembly language. Very often it's something like this:
mov eax, [ebp+XX] ; passed on argument to this subroutine
push eax
or
mov eax, [ebp-XX] ; a LAV: local automatic variable
push eax
push XX ; a direct constant
mov eax, [XXXXXXXX] ; a global variable
push eax

I hope this clears up a lot of confusions around the myth around a subroutine
call in assembly language. Now we briefly describe __stdcall, __fastcall,
and thiscall

3.2) __stdcall: The __stdcall calling convention is used to call Win32 API
functions. The callee cleans the stack, so the compiler makes vararg functions
__cdecl. Functions that use this calling convention require a function prototype.
return-type __stdcall function-name[(argument-list)] The following list shows the
implementation of this calling convention. Element Implementation Argument-passing
order Right to left. Argument-passing convention By value, unless a pointer or
reference type is passed. Stack-maintenance responsibility Called function pops
its own arguments from the stack. Name-decoration convention An underscore (_)
is prefixed to the name. The name is followed by the at sign (@) followed by the
number of bytes (in decimal) in the argument list. Therefore, the function declared
as int func( int a, double b ) is decorated as follows: _func@12 Case-translation
convention None

Callee:
int __stdcall sub_a(arg1, arg2, arg3)
push ebp
mov ebp, esp
sub esp, 4 x number_of_local_automatic_variables <-- normally the case
; pointers could complicate it, and structure alignment requirement
; could also make the space required on stack seem strange.
; nevertheless, automatica variables are stored on stack.
; name it NOLAV: # of local automatic variables
arg1 = [ebp+8] ; Here we assume it's a near call (*flat memory model)
arg2 = [ebp+C] ; within the caller's own code segment linear space
arg3 = [ebp+10] ; normally true but not for some trickery code ...
add esp, 4 x NOLAV ; ebp+10 <- arg3 || stack high
pop ebp ; ebp+0C <- arg2 ||
ret 0C ; ebp+8 <- arg1 ||
; ebp+4 <- eip of the return address ||
; ebp <- previous ebp ||
<= esp = ebp pointer \/ stack low

The difference here is 'ret 0C' instead of 'ret' because in __stdcall the callee
is responsible to clean up the stack.

Caller: call sub_a(arg1, arg2, arg3)
push arg3 ; see below
push arg2
push arg1
call sub_a

Here after call sub_a, the caller code needs not to worry about stack clean up.
There are 2 important points I want to bring up about __stdcall. First, WINAPI
is __stdcall so they use this convention. This is the most frequently encountered
form on a windows platform. Second, __stdcall will mangle the subroutine name
which will cause trouble if you try to link against __stdcall subroutine.
Unless you are working in a consistent setting, try to avoid __stdcall
declaration. 3.3) __fastcall The __fastcall calling convention specifies that
arguments to functions are to be passed in registers, when possible. The following
list shows the implementation of this calling convention. Element Implementation
Argument-passing order The first two DWORD or smaller arguments are passed in ECX
and EDX registers; all other arguments are passed right to left. Stack-maintenance
responsibility Called function pops the arguments from the stack. Name-decoration
convention At sign (@) is prefixed to names; an at sign followed by the number of
bytes (in decimal) in the parameter list is suffixed to names. Case-translation
convention No case translation performed. Callee: int __stdcall sub_a(arg1, arg2,
arg3)

push ebp
mov ebp, esp
sub esp, 4 x NOLAV arg1 = ecx ; Here we assume it's a near call (*flat memory model)
arg2 = edx ; within the caller's own code segment linear space
arg3 = [ebp+8] ; normally true but not for some trickery code ...
add esp, 4 x NOLAV
pop ebp
ret 04 ; ebp+8 <- arg1 ||
; ebp+4 <- eip of the return address ||
; ebp <- previous ebp ||
<= esp = ebp pointer \/ stack low

The difference here is 'ret 04' instead of 'ret' in __cdecl or 'ret 0C' in __stdcall
because in __fastcall passes 2 arguments using registers and the callee is
responsible to clean up the rest of the stack.

Caller:

call sub_a(arg1, arg2, arg3)
push arg3 ; see below
mov edx, arg2
mov ecx, arg1
call sub_a

__fastcall is similar to __stdcall except when callee has less than 3 arguments,
no stack reference is needed for arguments so execution speed is improved. Like
__stdcall, subroutine names are also mangled in library. 3.4) thiscall The
__fastcall calling convention specifies that arguments to functions are to be
passed in registers, when possible. The following list shows the implementation of
this calling convention. Element Implementation Argument-passing order The first
two DWORD or smaller arguments are passed in ECX and EDX registers; all other
arguments are passed right to left. Stack-maintenance responsibility Called
function pops the arguments from the stack. Name-decoration convention At
sign (@) is prefixed to names; an at sign followed by the number of bytes
(in decimal) in the parameter list is suffixed to names. Case-translation
convention No case translation performed.

Callee: int __stdcall sub_a(arg1, arg2, arg3)
push ebp
mov ebp, esp
sub esp, 4 x NOLAV
arg1 = [ebp+10] ; Here we assume it's a near call (*flat memory model)
arg2 = [ebp+0C] ; within the caller's own code segment linear space
arg3 = [ebp+8] ; normally true but not for some trickery code this_pointer = ecx ...
add esp, 4 x NOLAV
pop ebp
ret 0C ; ebp+8 <- arg1 ||
; ebp+4 <- eip of the return address ||
; ebp <- previous ebp ||
<= esp = ebp pointer \/ stack low

Pretty much like __stdcall except that the caller secretly puts the "this" pointer
into ecx before it calls sub_a.

Caller: call sub_a(arg1, arg2, arg3)
push arg3 ; see below
push arg2
push arg1
mov ecx, this_pointer
call sub_a

Pretty much like __stdcall except that the caller secretly puts the "this" pointer
into ecx before it calls sub_a.

3.5) __cdeclspec(naked)
Callee: __cdeclspec(naked) int sub_a(arg1, arg2, arg3){
__asm{
do what ever but don't blow it up!
}
}
Unlike any of the above declaration decorations, this one is special that it
will not setup the base stack frame pointer (ebp) for the callee. The callee
sub_a will normally be written in assembly code and it's entirely upon sub_a to
not blow up anything! The complier trusts that sub_a knows what it is doing. On
the caller side, nothing special needs to be done. Although it's not unusual a
chain of naked subroutines are constructed to achieve some specific goal.

3.6) The default VC behavior when working with .C source code is __cdecl but
without the _ prefix. When working with .CPP source code, the function name is
automatically mangled unless prefixed with extern "C".

__declspec(dllexport) int a(int x, int y, int z){
int b = x+10;
return b+y+z;
}

.text:10001000
public a .text:10001000 a
proc near .text:10001000
.text:10001000 var_4 = dword ptr -4
.text:10001000 arg_0 = dword ptr 8
.text:10001000 arg_4 = dword ptr 0Ch
.text:10001000 arg_8 = dword ptr 10h
.text:10001000
.text:10001000 push ebp
.text:10001001 mov ebp, esp
.text:10001003 push ecx
.text:10001004 mov eax, [ebp+arg_0]
.text:10001007 add eax, 0Ah
.text:1000100A mov [ebp+var_4], eax
.text:1000100D mov eax, [ebp+var_4]
.text:10001010 add eax, [ebp+arg_4]
.text:10001013 add eax, [ebp+arg_8]
.text:10001016 mov esp, ebp
.text:10001018 pop ebp
.text:10001019 retn
.text:10001019 a endp

Table of name generation
.C __declspec(dllexport) int a -> a
.C __declspec(dllexport) int __cdecl a -> a
.C __declspec(dllexport) int __stdcall a -> _a@12
.CPP __declspec(dllexport) int a -> ?a@@YAHHH@Z
.CPP extern "C" __declspec(dllexport) int a -> a
.CPP extern "C" __declspec(dllexport) int __stdcall a -> _a@12

So be careful when you name your files. .C and .CPP produce
drastically different function tables and be sure to know what
you want and name your source code files accordingly.