Advanced Operating

System
Professor Mangal Sain
 Lecture 9

Security and Protection

Lecture 9 – Part 1

Security
THE SECURITY PROBLEM

 System secure if resources used and accessed as

intended under all circumstances
 Unachievable
 Intruders (crackers) attempt to breach security
 Threat is potential security violation

 Attack is attempt to breach security

 Attack can be accidental or malicious

 Easier to protect against accidental than

malicious misuse
SECURITY VIOLATION CATEGORIES

 Breach of confidentiality
 Unauthorized reading of data
 Breach of integrity
 Unauthorized modification of data
 Breach of availability
 Unauthorized destruction of data
 Theft of service
 Unauthorized use of resources
 Denial of service (DOS)
 Prevention of legitimate use
SECURITY VIOLATION METHODS

 Masquerading (breach authentication)

 Pretending to be an authorized user to escalate
privileges
 Replay attack
 As is or with message modification
 Man-in-the-middle attack
 Intruder sits in data flow, masquerading as sender
to receiver and vice versa
 Session hijacking
 Intercept an already-established session to bypass
authentication
STANDARD SECURITY ATTACKS
SECURITY MEASURE LEVELS

 Impossible to have absolute security, but make cost to

perpetrator sufficiently high to deter most intruders
 Security must occur at four levels to be effective:
 Physical
 Data centers, servers, connected terminals
 Human
 Avoid social engineering, phishing, dumpster diving
 Operating System
 Protection mechanisms, debugging
 Network
 Intercepted communications, interruption, DOS
 Security is as weak as the weakest link in the chain
 But can too much security be a problem?
PROGRAM THREATS
 Many variations, many names
 Trojan Horse
 Code segment that misuses its environment
 Exploits mechanisms for allowing programs written by
users to be executed by other users
 Spyware, pop-up browser windows, covert channels
 Up to 80% of spam delivered by spyware-infected systems
 Trap Door
 Specific user identifier or password that circumvents
normal security procedures
 Could be included in a compiler
 How to detect them?
PROGRAM THREATS (CONT.)

 Logic Bomb
 Program that initiates a security incident under
certain circumstances
 Stack and Buffer Overflow
 Exploits a bug in a program (overflow either the stack
or memory buffers)
 Failure to check bounds on inputs, arguments
 Write past arguments on the stack into the return
address on stack
 When routine returns from call, returns to hacked
address
 Pointed to code loaded onto stack that executes malicious
code
 Unauthorized user or privilege escalation
GREAT PROGRAMMING REQUIRED?
 For the first step of determining the bug, and second step
of writing exploit code, yes
 Script kiddies can run pre-written exploit code to attack
a given system
 Attack code can get a shell with the processes’ owner’s
permissions
 Or open a network port, delete files, download a program, etc
 Depending on bug, attack can be executed across a network
using allowed connections, bypassing firewalls
 Buffer overflow can be disabled by disabling stack
execution or adding bit to page table to indicate “non-
executable” state
 Available in SPARC and x86
 But still have security exploits
Lecture 9 – Part 2

Security
PROGRAM THREATS (CONT.)

 Viruses
 Code fragment embedded in legitimate program
 Self-replicating, designed to infect other computers
 Very specific to CPU architecture, operating system,
applications
 Usually borne via email or as a macro
 Visual Basic Macro to reformat hard drive
Sub AutoOpen()
Dim oFS
Set oFS = CreateObject(’’Scripting.FileSystemObject’’)
vs = Shell(’’c:command.com /k format c:’’,vbHide)
End Sub
PROGRAM THREATS (CONT.)

 Virus dropper inserts virus onto the system

 Many categories of viruses, literally many
thousands of viruses
 File / parasitic
 Boot / memory
 Macro
 Source code
 Polymorphic to avoid having a virus signature
 Encrypted
 Stealth
 Tunneling
 Multipartite
 Armored
A BOOT-SECTOR COMPUTER VIRUS
THE THREAT CONTINUES

 Attacks still common, still occurring

 Attacks moved over time from science experiments to
tools of organized crime
 Targeting specific companies
 Creating botnets to use as tool for spam and DDOS delivery
 Keystroke logger to grab passwords, credit card numbers

 Why is Windows the target for most attacks?

 Most common
 Everyone is an administrator
 Licensing required?
 Monoculture considered harmful
SYSTEM AND NETWORK THREATS
 Some systems “open” rather than secure by
default
 Reduce attack surface
 But harder to use, more knowledge needed to administer

 Network threats harder to detect, prevent

 Protection systems weaker
 More difficult to have a shared secret on which to base
access
 No physical limits once system attached to internet
 Or on network with system attached to internet
 Even determining location of connecting system difficult
 IP address is only knowledge
SYSTEM AND NETWORK THREATS (CONT.)

 Port scanning
 Automated attempt to connect to a range of ports on
one or a range of IP addresses
 Detection of answering service protocol
 Detection of OS and version running on system
 nmap scans all ports in a given IP range for a response
 nessus has a database of protocols and bugs (and
exploits) to apply against a system
 Frequently launched from zombie systems
 To decrease trace-ability
SYSTEM AND NETWORK THREATS (CONT.)

 Denial of Service
 Overload the targeted computer preventing it from
doing any useful work
 Distributed denial-of-service (DDOS) come from
multiple sites at once
 Consider the start of the IP-connection handshake
(SYN)
 How many started-connections can the OS handle?
 Consider traffic to a web site
 How can you tell the difference between being a target and
being really popular?
 Accidental – CS students writing bad fork() code
 Purposeful – extortion, punishment
CRYPTOGRAPHY AS A SECURITY TOOL
 Broadest security tool available
 Internal to a given computer, source and destination of
messages can be known and protected
 OS creates, manages, protects process IDs, communication
ports
 Source and destination of messages on network cannot
be trusted without cryptography
 Local network – IP address?
 Consider unauthorized host added

 WAN / Internet – how to establish authenticity

 Not via IP address

CRYPTOGRAPHY
 Means to constrain potential senders
(sources) and / or receivers (destinations) of
messages
 Based on secrets (keys)
 Enables
 Confirmation of source
 Receipt only by certain destination

 Trust relationship between sender and receiver

 Lecture 9 – Part 3

Security- Cryptography
ENCRYPTION
 Constrains the set of possible receivers of a message
 Encryption algorithm consists of
 Set K of keys
 Set M of Messages
 Set C of ciphertexts (encrypted messages)
 A function E : K → (M→C). That is, for each k  K, Ek is a
function for generating ciphertexts from messages
 Both E and Ek for any k should be efficiently computable
functions
 A function D : K → (C → M). That is, for each k  K, Dk is
a function for generating messages from ciphertexts
 Both D and Dk for any k should be efficiently computable
functions
ENCRYPTION (CONT.)
 An encryption algorithm must provide this essential
property: Given a ciphertext c  C, a computer can
compute m such that Ek(m) = c only if it possesses k
 Thus, a computer holding k can decrypt ciphertexts to the
plaintexts used to produce them, but a computer not
holding k cannot decrypt ciphertexts
 Since ciphertexts are generally exposed (for example, sent
on the network), it is important that it be infeasible to
derive k from the ciphertexts
SYMMETRIC ENCRYPTION
 Same key used to encrypt and decrypt
 Therefore k must be kept secret
 DES was most commonly used symmetric block-encryption algorithm (created
by US Govt)
 Encrypts a block of data at a time
 Keys too short so now considered insecure
 Triple-DES considered more secure
 Algorithm used 3 times using 2 or 3 keys
 For example
 2001 NIST adopted new block cipher - Advanced Encryption Standard (AES)
 Keys of 128, 192, or 256 bits, works on 128 bit blocks
 RC4 is most common symmetric stream cipher, but known to have
vulnerabilities
 Encrypts/decrypts a stream of bytes (i.e., wireless transmission)
 Key is a input to pseudo-random-bit generator
 Generates an infinite keystream
SECURE COMMUNICATION OVER INSECURE MEDIUM
ASYMMETRIC ENCRYPTION

 Public-key encryption based on each user

having two keys:
 public key – published key used to encrypt data
 private key – key known only to individual user
used to decrypt data
 Must be an encryption scheme that can be made
public without making it easy to figure out the
decryption scheme
 Most common is RSA block cipher
 Efficient algorithm for testing whether or not a
number is prime
 No efficient algorithm is know for finding the prime
factors of a number
CRYPTOGRAPHY (CONT.)

 Note symmetric cryptography based on

transformations, asymmetric based on
mathematical functions
 Asymmetric much more compute intensive
 Typically not used for bulk data encryption
AUTHENTICATION
 Constraining set of potential senders of a message
 Complementary to encryption
 Also can prove message unmodified
 Algorithm components
 A set K of keys
 A set M of messages
 A set A of authenticators
 A function S : K → (M→ A)
 That is, for each k  K, Sk is a function for generating
authenticators from messages
 Both S and Sk for any k should be efficiently computable
functions
 A function V : K → (M × A→ {true, false}). That is, for
each k  K, Vk is a function for verifying authenticators
on messages
 Both V and Vk for any k should be efficiently computable
functions
AUTHENTICATION – HASH FUNCTIONS

 Basis of authentication
 Creates small, fixed-size block of data message
digest (hash value) from m
 Hash Function H must be collision resistant on m
 Must be infeasible to find an m’ ≠ m such that H(m) = H(m’)
 If H(m) = H(m’), then m = m’
 The message has not been modified
 Common message-digest functions include MD5,
which produces a 128-bit hash, and SHA-1, which
outputs a 160-bit hash
 Not useful as authenticators
 For example H(m) can be sent with a message
 But if H is known someone could modify m to m’ and recompute H(m’)
and modification not detected
 So must authenticate H(m)
AUTHENTICATION - MAC
 Symmetric encryption used in message-
authentication code (MAC) authentication
algorithm
 Cryptographic checksum generated from message
using secret key
 Can securely authenticate short values
 If used to authenticate H(m) for an H that is
collision resistant, then obtain a way to securely
authenticate long message by hashing them first
 Note that k is needed to compute both Sk and Vk, so
anyone able to compute one can compute the other
AUTHENTICATION – DIGITAL SIGNATURE
 Based on asymmetric keys and digital signature algorithm
 Authenticators produced are digital signatures
 Very useful – anyone can verify authenticity of a message
 In a digital-signature algorithm, computationally
infeasible to derive ks from kv
 V is a one-way function
 Thus, kv is the public key and ks is the private key
 Consider the RSA digital-signature algorithm
 Similar to the RSA encryption algorithm, but the key use is
reversed
 Digital signature of message Sks (m) = H(m)ks mod N
 The key ks again is a pair (d, N), where N is the product of two
large, randomly chosen prime numbers p and q
 Verification algorithm is Vkv(m, a) (akv mod N = H(m))
 Where kv satisfies kvks mod (p − 1)(q − 1) = 1
KEY DISTRIBUTION

 Delivery of symmetric key is huge

challenge
 Sometimes done out-of-band
 Asymmetric keys can proliferate – stored
on key ring
 Even asymmetric key distribution needs
care – man-in-the-middle attack
DIGITAL CERTIFICATES

 Proof of who or what owns a public key

 Public key digitally signed a trusted party

 Trusted party receives proof of

identification from entity and certifies that
public key belongs to entity
 Certificate authority are trusted party –
their public keys included with web browser
distributions
 They vouch for other authorities via digitally
signing their keys, and so on
MAN-IN-THE-MIDDLE ATTACK ON ASYMMETRIC CRYPTOGRAPHY
IMPLEMENTATION OF CRYPTOGRAPHY

 Can be done at various

layers of ISO Reference
Model
 SSL at the Transport layer
 Network layer is typically IPSec
 IKE for key exchange

 Basis of Virtual Private

Networks (VPNs)

 Why not just at lowest Source: http://en.wikipedia.org/wiki/

level? OSI_model

 Sometimes need more knowledge

than available at low levels
 i.e. User authentication

 i.e. e-mail delivery

ENCRYPTION EXAMPLE - SSL
 Insertion of cryptography at one layer of the ISO network
model (the transport layer)
 SSL – Secure Socket Layer (also called TLS)
 Cryptographic protocol that limits two computers to only
exchange messages with each other
 Very complicated, with many variations
 Used between web servers and browsers for secure
communication (credit card numbers)
 The server is verified with a certificate assuring client is
talking to correct server
 Asymmetric cryptography used to establish a secure
session key (symmetric encryption) for bulk of
communication during session
 Communication between each computer then uses
symmetric key cryptography
USER AUTHENTICATION
 Crucial to identify user correctly, as protection systems depend on
user ID
 User identity most often established through passwords, can be
considered a special case of either keys or capabilities
 Passwords must be kept secret
 Frequent change of passwords
 History to avoid repeats
 Use of “non-guessable” passwords
 Log all invalid access attempts (but not the passwords
themselves)
 Unauthorized transfer
 Passwords may also either be encrypted or allowed to be used only
once
 Does encrypting passwords solve the exposure problem?
 Might solve sniffing

 Consider shoulder surfing

 Consider Trojan horse keystroke logger

 How are passwords stored at authenticating site?
PASSWORDS
 Encrypt to avoid having to keep secret
 But keep secret anyway (i.e. Unix uses superuser-only readably file
/etc/shadow)
 Use algorithm easy to compute but difficult to invert
 Only encrypted password stored, never decrypted
 Add “salt” to avoid the same password being encrypted to the same
value
 One-time passwords
 Use a function based on a seed to compute a password, both user and
computer
 Hardware device / calculator / key fob to generate the password
 Changes very frequently

 Biometrics
 Some physical attribute (fingerprint, hand scan)
 Multi-factor authentication
 Need two or more factors for authentication
 i.e. USB “dongle”, biometric measure, and password
IMPLEMENTING SECURITY DEFENSES
 Defense in depth is most common security theory – multiple layers of
security
 Security policy describes what is being secured
 Vulnerability assessment compares real state of system / network compared
to security policy
 Intrusion detection endeavors to detect attempted or successful intrusions
 Signature-based detection spots known bad patterns
 Anomaly detection spots differences from normal behavior
 Can detect zero-day attacks

 False-positives and false-negatives a problem

 Virus protection
 Searching all programs or programs at execution for known virus
patterns
 Or run in sandbox so can’t damage system
 Auditing, accounting, and logging of all or specific system or network
activities
 Practice safe computing – avoid sources of infection, download from only
“good” sites, etc
FIREWALLING TO PROTECT SYSTEMS AND NETWORKS

 A network firewall is placed between trusted and

untrusted hosts
 The firewall limits network access between these two security
domains
 Can be tunneled or spoofed
 Tunneling allows disallowed protocol to travel within allowed
protocol (i.e., telnet inside of HTTP)
 Firewall rules typically based on host name or IP address which
can be spoofed
 Personal firewall is software layer on given host
 Can monitor / limit traffic to and from the host
 Application proxy firewall understands application
protocol and can control them (i.e., SMTP)
 System-call firewall monitors all important system calls
and apply rules to them (i.e., this program can execute
that system call)