I 3YSTEMSAND)NTERNET

I
)NFRASTRUCTURE3ECURITY
.ETWORKAND3ECURITY2ESEARCH#ENTER
$EPARTMENTOF#OMPUTER3CIENCEAND%NGINEERING
0ENNSYLVANIA3TATE5NIVERSITY5NIVERSITY0ARK0!

CSE543 - Introduction to
Computer and Network Security
Module: Operating System Security

Professor Trent Jaeger

CSE543 - Introduction to Computer and Network Security Page 1

 OS Security
• So, you have built an operating system that enables
user-space processes to access hardware resources
‣ Thru various abstractions: files, pages, devices, etc.
• Now, you want your operating system to enforce
security requirements for your application processes
‣ What do you do?

CSE543 - Introduction to Computer and Network Security Page 2

 OS Security
• We learned about a few things that will help you
• Your OS must implement a
‣ (Mandatory) Protection system
• That can enforce a
‣ MAC policy
• How do we implement such an OS mechanism?
‣ Multics
‣ Linux Security Modules

CSE543 - Introduction to Computer and Network Security Page 3

 Access Policy Enforcement
• A protection system uses a reference validation
mechanism to produce and evaluate authorization
queries
‣ Interface: Mediate security-sensitive operations by building
authorization queries to evaluate
‣ Module: Determine relevant protection state entry (ACLs,
capabilities) to evaluate authorization query
‣ Manage: Install protection state entries and reason about
labeling and transition states

• How do we know whether a reference validation

mechanism is correct?
CSE543 - Introduction to Computer and Network Security Page 4
 Security-Sensitive Operations
• Broadly, operations that enable interaction among
processes that violate secrecy, integrity, availability
• Which of these are security-sensitive? Why?
‣ Read a file (read)
‣ Get the process id of a process (getpid)
‣ Read file metadata (stat)
‣ Fork a child process (fork)
‣ Get the metadata of a file you have already
opened? (fstat)
‣ Modify the data segment size? (brk)
• Require protection for all of CIA?
CSE543 - Introduction to Computer and Network Security Page 5
 Reference Monitor
• Defines a set of requirements on reference
validation mechanisms
‣ To enforce access control policies correctly
• Complete mediation
‣ The reference validation mechanism must always be
invoked (before executing security-sensitive operations)
• Tamperproof
‣ The reference validation mechanism must be tamperproof
• Verifiable
‣ The reference validation mechanism must be small
enough to be subject to analysis and tests, the
completeness of which can be assured
CSE543 - Introduction to Computer and Network Security Page 6
 Multiprocessor Systems
• Major Effort: Multics
‣ Multiprocessing system -- developed many OS concepts
• Including security
‣ Begun in 1965
• Research continued into the mid-70s
‣ Used until 2000
‣ Initial partners: MIT, Bell Labs, GE (replaced by Honeywell)
‣ Other innovations: hierarchical filesystems, dynamic linking

• Subsequent proprietary system, SCOMP, became the

basis for secure operating systems design (XTS-400)
CSE543 - Introduction to Computer and Network Security Page 7
 Multics Goals
• Secrecy
‣ Multilevel security
• Integrity
‣ Rings of protection
• Resulting system is
considered a high point in
secure systems design

CSE543 - Introduction to Computer and Network Security Page 8

 Protection Rings
• Successively less-privileged “domains”
• Modern CPUs support 4 rings
‣ Use 2 mainly: Kernel and user
• Intel x86 rings
‣ Ring 0 has kernel
‣ Ring 3 has application code

• Example: Multics (64 rings in theory, 8 in practice)

CSE543 - Introduction to Computer and Network Security Page 9
 What Are Protection Rings?
• Coarse-grained, Hardware Protection Mechanism
• Boundary between Levels of Authority
‣ Most privileged -- ring 0
‣ Monotonically less privileged above
• Fundamental Purpose
‣ Protect system integrity
• Protect kernel from services
• Protect services from apps
• So on...

CSE543 - Introduction to Computer and Network Security Page 10

 Protection Ring Rules
• Program cannot call code of
higher privilege directly
‣ Gate is a special memory
address where lower-privilege
Ring 3
code can call higher No
• Enables OS to control where gate
applications call it (system calls) Gate

Ring 0

CSE543 - Introduction to Computer and Network Security Page 11

 Multics Interpretation
• Kernel resides in ring 0
7
• Process runs in a ring r
‣ Access based on current ring ---
6
• Process accesses data (segment)
5
‣ Each data segment has an access a2
bracket: (a1, a2) 4
• a1 <= a2
R-X
‣ Describes read and write access to Ring 3
segment
• r is the current ring 2
• r <= a1: access permitted a1
1
• a1 < r <= a2: r and x permitted; w denied RWX
• a2 < r: all access denied 0

CSE543 - Introduction to Computer and Network Security Page 12

 Multics Interpretation (con’t)
• Also different procedure segments Denied
‣ with call brackets: (c1, c2), c1 <= c2 7
c2
‣ and access brackets (a1, a2) Allow
6
‣ The following must be true (a2 == c1) with
‣ Rights to execute code in a new procedure segment 5 gate
• r < a1: access permitted with ring-crossing fault a2
• a1 <= r <= a2 = c1: access permitted and no fault 4 c1
• a2 < r <= c2: access permitted through a valid gate
• c2 < r: access denied Ring 3 No ring
• What’s it mean? fault
2
‣ case 1: ring-crossing fault changes procedure’s ring
• increases from r to a1
1 a1
‣ case 2: keep same ring number Ring
‣ case 3: gate checks args, decreases ring number 0 fault
• Target code segment defines the new ring
CSE543 - Introduction to Computer and Network Security Page 13
 Examples
• Process in ring 3 accesses data segment
‣ access bracket: (2, 4)
‣ What operations can be performed?
• Process in ring 5 accesses same data segment
‣ What operations can be performed?
• Process in ring 5 accesses procedure segment
‣ access bracket (2, 4)
‣ call bracket (4, 6)
‣ Can call be made?
‣ How do we determine the new ring?
‣ Can new procedure segment access the data segment
above?
CSE543 - Introduction to Computer and Network Security Page 14
 Now forward to UNIX ...

CSE543 - Introduction to Computer and Network Security Page 15

 UNIX Security Limitations
• Circa 2000 Problems
‣ Discretionary access control
‣ Setuid root processes
‣ Network-facing daemons vulnerable
• What can we do?

CSE543 - Introduction to Computer and Network Security Page 16

 UNIX Security Limitations
• Circa 2000 Problems
‣ Discretionary access control
‣ Setuid root processes
‣ Network-facing daemons vulnerable
• What can we do?
‣ Reference validation mechanism that satisfies reference
monitor concept

‣ Protection system with mandatory access control

(mandatory protection system)

CSE543 - Introduction to Computer and Network Security Page 17

 Linux Security Modules
• Reference validation mechanism for Linux
‣ Upstreamed in Linux 2.6
‣ Support modular enforcement - you choose
• SELinux, AppArmor, POSIX Capabilities, SMACK, ...

• 150+ authorization hooks

‣ Mediate security-sensitive operations on
• Files, dirs/links, IPC, network, semaphores, shared memory, ...

‣ Variety of operations per data type

• Control access to read of file data and file metadata separately

• Hooks are restrictive

CSE543 - Introduction to Computer and Network Security Page 18
 LSM & Reference Monitor
• Does LSM satisfy reference monitor concept?

CSE543 - Introduction to Computer and Network Security Page 19

 LSM & Reference Monitor
• Does LSM satisfy reference monitor concept?
‣ Tamperproof
• Can MAC policy be tampered?
• Can kernel be tampered?

CSE543 - Introduction to Computer and Network Security Page 20

 LSM API
Linux Security Modules

• Register (install) module

• Load policy (open and write to special file)
• Produce authorization queries at hooks
Systems and Internet Infrastructure Security (SIIS) Laboratory Page 14

CSE543 - Introduction to Computer and Network Security Page 21

 LSM API
Linux Security Modules

• Attacks on “register”
• Attacks on “install policy”
• Attacks on “system calls”
Systems and Internet Infrastructure Security (SIIS) Laboratory Page 14

CSE543 - Introduction to Computer and Network Security Page 22

 LSM API
Linux Security Modules

• To prevent attacks on registration

• And attacks on function pointers of LSM
• LSMs are now statically compiled into the kernel
Systems and Internet Infrastructure Security (SIIS) Laboratory Page 14

CSE543 - Introduction to Computer and Network Security Page 23

 LSM & Reference Monitor
• Does LSM satisfy reference monitor concept?
‣ Tamperproof
Can MAC policy be tampered?
•
• Can kernel be tampered?
‣ Verifiable
• How large is kernel?
• Can we perform complete testing?

CSE543 - Introduction to Computer and Network Security Page 24

 LSM & Reference Monitor
• Does LSM satisfy reference monitor concept?
‣ Tamperproof
Can MAC policy be tampered?
•
• Can kernel be tampered? By network threats?
‣ Verifiable
• How large is kernel?
• Can we perform complete testing?
‣ Complete Mediation
• What is a security-sensitive operation?
• Do we mediate all paths to such operations?

CSE543 - Introduction to Computer and Network Security Page 25

 LSM Security
Linux Hooks Modules

Systems and Internet Infrastructure Security (SIIS) Laboratory Page 1

CSE543 - Introduction to Computer and Network Security Page 26
 LSM & Complete Mediation
• What is a security-sensitive operation?
‣ Instructions? Which?
‣ Structure member accesses? To what data?
‣ Data types whose instances may be controlled
Inodes, files, IPCs, tasks, ...
•
• Approaches
‣ Mediation: Check that authorization hook
dominates all control-flow paths to structure
member access on security-sensitive data type
‣ Consistency: Check that every structure member
access that is mediated once is always mediated
• Several bugs found - some years later
CSE543 - Introduction to Computer and Network Security Page 27
 LSM &
Analysis
Complete Mediation THREAD-A:
(1) fd1 = op
(2) fd2 = op
(3) fcntl(fd

KERNEL
/* from fs/fcntl.c */ (4) fi

•  Static analysis of Zhang, Edwards,

long sys_fcntl(unsigned int fd, (5) se
unsigned int cmd,
unsigned long arg) (6) fc
{

and Jaeger [USENIX Security

struct file * filp; THREAD-B:
... /* this clos
filp = fget(fd); * and assig
... */

2002]!
(7) dup2( fd
err = security ops->file ops
->fcntl(filp, cmd, arg); KERNEL
... /* thi
err = do fcntl(fd, cmd, arg, filp); * fil
*/

Based on a tool called CQUAL!

...

‣ 
(8) fi
} (9) lo

static long
do_fcntl(unsigned int fd, Fig
unsigned int cmd,

•  Found a TOCTTOU bug!

unsigned long arg,
struct file * filp) {
...
switch(cmd){ chance of race
... not properly sy
case F_SETLK: tial exploits.

‣  Authorize filp in sys_fcntl! ...

err = fcntl setlk(fd, ...);
Here we presen
curity checks th
} are performed o
... ple, the followin

‣  But pass fd again to fcntl_getlk! }

/* from fs/locks.c */
dentry structu
on the inode s
fcntl_getlk(fd, ...) {
struct file * filp;

•  Many supplementary analyses

...

filp = fget(fd); /* from fs/a

...
/* operate on filp */ security_ops

were necessary to support

... ->setattr(
} ...
inode = dent
inode_setatt

CQUAL!
Figure 8: Code path from Linux 2.4.9 containing an ex-
...
ploitable type error.

It is also quite c
Systems
CSE543and Internet Infrastructure
- Introduction to ComputerSecurity (SIIS) Laboratory
and Network Security Page
Page 21 28 an
data structure
 LSM Enforcement
• Several LSMs have been deployed
‣ Most prominent: AppArmor, SELinux, Smack,
TOMOYO
• The most comprehensive is SELinux
‣ Used by RedHat Fedora and some others

CSE543 - Introduction to Computer and Network Security Page 29

 LSM Enforcement
• Several LSMs have been deployed
‣ Most prominent: AppArmor, SELinux, Smack,
TOMOYO
• The most comprehensive is SELinux
‣ Created by the NSA - Result of many years work
‣ Used by RedHat Fedora and some others

CSE543 - Introduction to Computer and Network Security Page 30

 LSM Enforcement
SELinux Policy Rules
Several LSMs
• • SELinux Ruleshave beenan
express deployed
MPS
‣ Most prominent: AppArmor, SELinux, Smack,
‣  Protection state – ALLOW subject-label object-label ops
TOMOYO
Labeling
• ‣  The moststate – Assign new is
comprehensive objects labels on creation
SELinux
‣  ‣ Transition
Createdstate – Define
by the NSAhow a process
- Result may change
of many label
years work
Used
‣ are
•  All by RedHat
defined Fedora and some others
explicitly
‣  Tens of thousands of rules are necessary for a standard
Linux distribution
•  Remember, we are ignoring user processes too (other than
confining them relative to the system)

•  Enforces a Least Privilege Policy

CSE543 and
Systems - Introduction to ComputerSecurity
Internet Infrastructure and Network Security
(SIIS) Laboratory 31
Page 9
Page
 SELinux
LSM Transition State
Enforcement
Several
• • For LSMs
user to run have been
passwd deployed
program
‣  Most
‣ Only prominent:
passwd AppArmor,
should have SELinux,
permission to Smack,
modify /etc/shadow
TOMOYO
•  Need permission to execute the passwd program
• The most comprehensive is SELinux
‣  allow user_t passwd_exec_t:file execute (user can exec /usr/bin/passwd)
‣ Created by the NSA - Result of many years work
‣  allow user_t passwd_t:process transition (user gets passwd perms)
‣ Used by RedHat Fedora and some others
•  Must transition to passwd_t from user_t
‣  allow passwd_t passwd_exec_t:file entrypoint (run w/ passwd perms)
‣  type_transition user_t passwd_exec_t:process passwd_t

•  Passwd can the perform the operation

‣  allow passwd_t shadow_t:file {read write} (can edit passwd file)

Systems
CSE543and Internet Infrastructure
- Introduction to ComputerSecurity (SIIS) Laboratory
and Network Security Page
Page 10
32
 Take Away
• Goal: Build authorization into operating systems
‣ Multics and Linux
• Requirements: Reference monitor
‣ Satisfy reference monitor concept
• Multics
‣ Hierarchical Rings for Protection
‣ Call/Access Bracket Policies (in addition to MLS)
• Linux
‣ Did not enforce security (DAC, Setuid, root daemons)
‣ So, the Linux Security Modules framework was added
‣ Approximates reference monitor assuming network threats
only -- some challenges in ensuring complete mediation
CSE543 - Introduction to Computer and Network Security Page 33