Security Paradigms in Operating Systems

Diogo Ditzel Kropiwiec,1*Paulo Lício de Geus 1

1
Laboratory of Administration and System Security
Institute of Computing - State University of Campinas
P.O. Box 6176 – 13084-971 Campinas, SP - BRAZIL
diogo,[email protected]
Abstract. Security has become one of the principal concerns during the development.
development of applications in general. However, the continual growth in the number
The number of security incidents reported demonstrates that such efforts have not been sufficient.
efficient to contain the hacker’s advances. In this paper, the security paradigms
used by the more common operating systems are presented and their vulnera-
bilities discussed, highlighting the factors that have contributed to the growing
number of attacks. New security paradigms are also presented and the factors
that hinder their fast adoption are analyzed.
Summary. Security has become one of the main focuses in development.
application of applications in general. The increase in the number of incidents of
security, however, shows that the efforts are being insufficient to
counter the advance of hackers. This article presents the paradigms of
security on which the most commonly used operating systems are based
mum, and its failures, thus alerting to the reasons that have led to the growth
number of attacks. New paradigms are also presented,
security analyzing the aspects that hinder a rapid adoption of the mes-
mos.

1. Introduction
Operating systems emerged with two main goals: to create a layer
the abstraction between hardware and applications, and manage resources effectively
[Tanenbaum and Woodhull, 1997]. In the first stage of development, the
security concerns regarding these operating systems were minimal,
if not non-existent, since the computers were isolated, and the only way to
The possible option was in-person, that is, it was necessary to be in front of the terminal to
perpetrate some malicious activity. Additionally, the processing power of the
computers were very limited, and the time to be spent on other activities that were not them
primary objectives—hardware abstraction and resource management—was highly
costly.
As time goes by, computers, just like operating systems
national organizations continued to evolve and spread. Organizations with various com-
computers began to set up networks to interconnect them, and the operating systems p
people to support the execution of multiple tasks simultaneously, allowing two
∗ CNPq support.
or more people shared the processing time of the same machine
[Garfinkel and Spafford, 1996, Tanenbaum and Woodhull, 1997]. In parallel, the capability-
Processing power has grown enormously, and applications have become more elaborate and complex.
could be developed.
With the advent of the Internet, many of these organizational networks have come together. The exchange
information was greatly benefited, and business transactions between organizations
were simplified and expanded. Domestic users began to be part of the
and a vast amount of information has become accessible to anyone
[Nakamura and de Geus, 2002].
Unfortunately, this computational expansion and the ease of obtaining information
brought a forgotten problem in the early moments: the security of information.
Recent data clearly shows that the number of reported security incidents
increases each year, from just 2,412 in 1995 to 137,529 in 2003 [CERT, 2003].
Meanwhile, the concern for security has been accompanying the development of
operating systems and their applications have been around for a long time, as can be seen in
[Harrison et al., 1976], contrary to what the statistics seem to demonstrate. Almost 30
Years have passed, and despite all efforts aimed at eliminating vulnerabilities, they have
have shown to be insufficient.
As the first step of a larger ongoing project, this article analyzes
isolate the paradigms of computer security, both those present in the systems op-
current operational usages as those present in operating systems still restricted to
academic environment, as well as its limitations and complications. The goal is to understand what
has caused the continuous proliferation of the security problems currently reported and
What can be done to reverse this situation.
In Section 2, the security paradigms present in the systems will be discussed.
operational current use. In Section 3, the failures of these paradigms will be presented.
but, which lead to the picture presented above. In Section 4, the new ones will be presented
security paradigms, of which there are already available implementations for use, but
as will be seen in Section 5, they are still likely to take time to become standard. Finally,
In Section 6, the next research areas necessary to enable will be pointed out.
adoption of new paradigms, aimed at strengthening the security of operating systems.

2. Security Paradigms of Current Operating Systems

Current operating systems are being included.
most widespread in organizations and personal computers. In this group, they are categorized
the Unix derivatives (Linux, BSD and their various distributions), Windows (with
highlighting to NT, 2000 and XP) and MacOS. These systems are derived from the operating systems
third generation operating systems [Tanenbaum and Woodhull, 1997], especially Unix—
what safety concerns were limited to the few users who had access
to the computer, directly via terminal, or through the isolated internal network that belonged to
machine—and the DOS—in which resource security concerns did not exist.
The security of these operating systems lies in the access rights of a
user (or group of users) to a resource (file, peripheral, memory, etc.), based on
on the concept of access permissions in Multics rings [Feiertag and Organick, 1971], and
subsequently adapted for users on Unix [Ritchie and Thompson, 1974]. In this last
In the article, the author warns that the proposed security model is simple, but adequate.
when it comes to his needs. Windows originally presented a well security
inferior to this, for using DOS as a support system (and consequently, the sys-
FAT16/32 file system, which did not provide any type of access control), and only
subsequently, Windows came to use a file system that allowed for control
of user access, the NTFS.
Three paradigms commonly present in the systems will now be presented.
operational, which will be numbered only to simplify the subsequent explanations
break the same.
The first security paradigm present in these systems consists of the user
as a security limiter. The user, when creating files and applications, associates with
these permissions relate to reading, writing, and execution—in some models, these
permissions are expanded to a larger set with more specific actions - for
if own, for users of the same group, and for other system users
[Garfinkel and Spafford, 1996], and this set of permissions associated with the files
it will be called domain. Thus, it is the user's responsibility to restrict
the files/resources that you consider relevant. Similarly, when running an application,
this will have access to the user's domain, being able to interact with applications and/or files of the
same form as the user from whom the domain was obtained. Additionally, there are situations
in which it is desired to allow a user, through an appropriate application,
can interact with the operating system as if it were another user, and in these cases are the-
sociadas flags1that allow the switching of rights [Garfinkel and Spafford, 1996].
The second security paradigm consists of concentration and isolation.
of the privileged rights of the computer. Operating systems have a
user, referred to as super-user or administrator, who has access to all the
resources of machines, and to which all privileged operations are restricted
[Garfin kel and Spafford, 1996]. It´ is important to emphasize that the super-user's domain
it involves unrestricted access to all computer resources, including files
and applications belonging to other users. To common users and their respective
only a subset of these privileged operations is allowed for access to applications, and
still limited by the decisions of the operating system core, through a
well-defined interface with the operating system, known as API - Application
Programming Interface. This way, it ensures that users cannot cause
any operational failure of the operating system, whether accidentally or intentionally
Mind. As can be observed, access to privileged operations is total or none.
since the API is responsible for controlling the indirect access carried out by the applications.
In this context, there are applications that require access to some operation.
privileged not provided by the API, end up having to use the superuser's domain,
having unrestricted access to all privileged operations.
A third commonly used paradigm consists of security through encryption.

1 InLinux, for example, there is obitSetUId associated with applications. When activated, the application is
executed in the owner's domain, and not of the user who commanded the execution
 the information. There is a varied set of encryption techniques used both
for messages to be sent over the network securely [Tanenbaum, 2003] as much
to store files on hard disk [Silbertschatz et al., 2002]. With encryption,
the user is guaranteed that, even if someone gains unauthorized access to
File or message, this will be illegible without the appropriate encryption key. Only
vulnerability to encryption the superuser who, during the encryption process, can
access the memory unrestrictedly and, consequently, obtain the message/file, or
still the encryption key. Once the encryption application is closed, even the superuser
you will not have access to the content if you have not stored the key.
Derived from these paradigms is the concept that applications must provide
the security that is lacking and that can adequately ensure such security without
operating system auxiliary [Loscocco et al., 1998]. Once compromised a
application, all the domain to which the application was associated will be compromised.
For this reason, every time a vulnerability is discovered, they are made available.
corrections (commonly referred to as patches) to the compromised application.

3. Failures of Current Paradigms

The paradigms presented are effective if restricted to the context in which they were designed.
returned, with isolated computers or small isolated networks. In these cases, the number
the attackers' power is extremely reduced, restricted to people who have access to
machines. However, the picture changes drastically when interconnecting these systems or networks.
Thky.enlchaoleitudcw
arktm
seatrednpela,rghIotibneftr
the attackers' technician varies across a much wider range of areas, applications, and systems op-
Operational. The difficulty of tracking an attack gives the attacker a relative sense of security.
that he cannot be held responsible for his actions.
Before presenting the flaws present in the three paradigms, it is necessary to
categorize them according to their security characteristics. In 1985, the Department of
The American defense presented the Trusted Computer System Evaluation Criteria (TCSEC),
a classification in four levels of security to be used when classifying systems
operational security [DoD, 1985]. However, the most important point of
Document regarding this article and the classification of types of access control:
DAC - Discretionary Access Control, eMAC - Mandatory Access Control.
first, the security of the files is at the discretion of the user who owns them, and these can
change the permissions of your files in relation to yourself and others freely.
In the second, there are security policies, common or specific to each user, that
we restrict the actions that this can perform on the file, aiming not only at the security of the
user, but also the security of the entire operating system.
As can be observed, the three paradigms presented in Section 2
quadram in the DAC classification. It is the user's responsibility to take care of the
security of your files, through the correct configuration of permissions, or through
from the encryption of important documents. Even the super-user is subject to this
thank you, given the type of access you have to the operating system.
Regarding the first paradigm, there are two main points to be analyzed.
´
It is common for the user to neglect the correct configuration of permissions on their
files, whether due to ignorance of how to do it, carelessness or disregard of the
even. Consequently, if another user's domain is compromised, and the
permissions of the first user as they allow, from the other it will be possible to obtain
access to the files, or even destroy them. The second point is related to the domain
associated applications. These have access to the same user domain as the ex-
executed, and if they are compromised by a successful attack, the attacker will have access
to the same domain of the user. This point has greater implications when together
with the second paradigm.
When it comes to an attack on applications of common users, only the files
to which they have access may be copied and/or destroyed. The greater risk
there is a possibility that applications requiring additional privileges
they must be committed, as a result of the second paradigm, once this happens,
the entire system will be compromised. The problem, therefore, is the lack of granularity in
distribution of the privileged operations of the operating system. This lack of granularity
age arises from the fact that all privileges are centralized and isolated in a
the only domain, that of the super-user, which makes it impossible to restrict the application only to
minimum subset of privileged operations necessary.
Relying exclusively on applications to ensure security is extreme.
dangerous mind, as can be observed by the reported security incidents
[CERT, 2003]. Applications are becoming increasingly complex, they have
larger and larger codes, and ensuring the non-existence of failures is practically impossible.
From the identification of a vulnerability to the availability of a fix, it takes
a considerable time, during which the affected applications can be explored by ata-
You sing. Add to this the fact that, with the addition of new features, new
failures may arise, and it is clear that continuing to rely on the applications
too risky.
The third paradigm presented can be considered safely in isolation.
With the appropriate keys, the desired security for the files will ensure that no attacker will be able to.
access the content. However, as previously presented, the super-user can
access the content in memory during the encryption process and compromise the
security and, as seen, it is possible for an attacker to gain access to the super domain
user. Additionally, even if the content cannot be read, it can be deleted,
only with access to the domain of the user who owns the file.
In [Loscocco et al., 1998], arguments are presented that demonstrate that,
without adequate security support from the operating system, the problems
The ones presented above will continue to occur. Thus, it is possible to understand
because security incidents grow every year, and because all the effort spent
computer security has proven to be insufficient to contain the advancement of
Therefore, new security paradigms for systems are necessary.
operational, that can support a secure computational development, without
depend exclusively on applications.
4. New Security Paradigms

The need for the operating system to ensure the security of files, applications and
computer resources against the eventual compromise of an unmanageable application
[Loscocco et al., 1998]. Based on this principle, three paradigms can be derived
security: mandatory access control, principle of least privilege, and protection
two mechanisms of the operating system API.
Mandatory Access Control (MAC), as briefly seen in Section
3, consists of a set of security policies that limit the permissions that the
the user provides information about files and applications. In this context, the need for a
security policy administrator, distinct and isolated from the system administrator
operational, whose responsibility is to define which policies to be applied, aiming
prevent users, including the super-user, from accidentally exposing files or
applications that may compromise the operating system. The need for isolation
The occurrence between the policy administrator and the system administrator is due to the failure.
presented by the second paradigm of Section 2. Otherwise, the compromise of the
super-user would imply the compromise of policies, and consequently, of everything
the operating system. Note that, in this context, the super-user also submits
e.schpietol
A need arising from the use of security policies is the inclusion of
two modules to the operating system: the policy manager and the re-
security enforcer (security enforcer or reference monitor [DoD, 1985]). The first is
responsible for managing security policies, using an understandable language
to the administrators, and by converting the security policies into an intelligible format
to the safety restraining device. The second must ensure that the policies are applied,
asking any user, including the administrator, to perform some unauthorized action
Additionally, the operating system must ensure that the restrictor is always
consulted, avoiding an attempt to bypass it to subvert the operating system.
The principle of least privilege, presented in [Ferraiolo and Kuhn, 1992],
it consists of the idea that a user or application is not given any privilege
more than necessary to carry out your activities. As seen, the problem of
the compromise of an application consists in it having access to the same
domain that the user who is executing it. However, when restricting the application to a
The damage that may arise will be smaller in this subdomain of this domain.
by linking this paradigm to the MAC paradigm, it is possible to ensure that a vulnerability
explored does not even affect the user. For example, an internet browser needs
read access to the operating system settings pertaining to itself,
read and write access to a temporary directory and nothing more. It can contain
the application to this subdomain, even if the browser is compromised, the attacker
you will only have access to the browser settings and the temporary directory,
usually used only to cache pages opened by the user. Even if
tried to implant a trojan horse in a temporary directory in the hope that the user would
accidentally execute a policy that prevents the execution of applications in the directory
temporary would eliminate this risk.
Finally, the operating system responsible for protecting the mechanisms of
API against spoofing attacks, bypassing and tampering. Many
Applications depend on the operating system mechanisms to perform their function.
correctly, as communication channels, cryptography mechanisms, control of
access, access to input and output devices (keyboards, mouse, video), among others.
If they can have their behavior changed, either by another application or by...
to explore some vulnerability of the operating system, there is no guarantee that the
information is not being destroyed or recorded by third parties. Even the presence
other security methods will not be sufficient to prevent the invasion.
There are already several operating systems that have been developed with a focus
in the new security paradigms. Among these, those based on micro-
kernel, which foresees primitive protection mechanisms aimed at supporting the
construction of a flexible and high-level security architecture. Some apprehensions
they encounter a limitation regarding the support of mandatory access control with
high degree of flexibility and validation, such as Fluke [Ford et al., 1996] and Exok-
kernel [Mazieres and Kaashoek, 1997]. Others, like L4 [Liedtke, 1996], support
mandatory access control with 'classes and chiefs' mechanisms and identification
of emitters/receivers in communication between processes, however they are harmed
due to the lack of a coherent framework to use these mechanisms with
mandatory policies, and for providing for a small number of security domains.
Two of the most relevant operating systems are Flask (a variation of Fluke)
[Secure Computing Corporation, 1998]) and the DTOS (a variation of the operating system
Both present a framework consistent with a policy and restriction manager.
security tower [Loscocco et al., 1998].
Regarding security policies, there are several models that represent
the relationships between users, applications, and resources. One of the oldest, the Multi-level Se-
security (MLS [Bell and Padula, 1973]), is based on user release decisions and
classifications for objects. However, it is very limited to frame several requirements
of security. Domain and Type Enforcement (DTE [Walker et al., 1996]), in turn,
associates types to files and system resources and domains applications and users, being
that the domains define what access is allowed to different types. Another model is the
Role-based Access Control (RBAC [Ferraiolo and Kuhn, 1992]), which uses the concept of
papers to define the operations that can be performed on system files,
very similar to the DTE. Finally, in [Swift et al., 2002] it is presented a
set of extensions of the Access Control List (ACL) model, which allows for the mechanism
access control system to protect the system, resources, applications, and users
used by Windows 2000.

5. Difficulties in the Implementation of New Security Paradigms

As seen in the previous section, there are already more robust security paradigms than
the gifts in common operating systems, under which they were developed
next generation operating systems, where security is a primary factor. Among
So, when observing any organization, it will be noticeable that the computers of its
networks still largely use, if not completely, operating systems based
in the old security paradigms. As security is a growing concern
In organizations [Nakamura and de Geus, 2002], it is necessary to analyze the reasons
the delay in the adoption of these systems.
One of the factors that hinder the implementation of new paradigms is 'inertia'.
two operating systems. By 'inertia', understand resistance to migration. Whether due to
part of the users, either because of the applications or due to financial issues
Considering operating system licenses, their replacement with another will not be easy.
It will be focused on this issue regarding applications, given that it imposes the greatest complication.
in the long term. They are associated with the activities of organizations and depend heavily on
mind of the structure provided to them by the operating system for which they were
involved. The replacement of the operating system creates the need for adaptation of the
applications to the new system, or the development of new applications that allow us
organizations will continue to carry out their activities. However, both processes
They involve high financial costs and a long development and training time.
Since the replacement of the inviable operating system, the solution would be to adapt the
existing operating systems to adapt to new paradigms. However,
due to the fact that they have developed from older models, the structure-
it requires a restructuring of the system for the new models,
with an emphasis on the core, which should be responsible for providing the necessary data for
the decision-making of the safety restrainer and respond according to the decision.
Other auxiliary systems, such as the file system, will need to be redesigned, to
to be able to store additional information (the types of files, the policies related
necessary security. In this sense, it is already in an advanced stage of development.
I am setting up a nucleus for Linux with support for mandatory control, SELinux (Security-
Enhanced Linux) [Loscocco and Smalley, 2001a, Loscocco and Smalley, 2001b] that pos-
a policy manager with support for DTEeRBAC. Windows also seems
walk in this direction, since it already has greater granularity access control
and the mechanisms of MAC [Swift et al., 2002].
Another factor that weighs on the adoption of new paradigms is the complexity of management.
Implementation of security policies. The policies are difficult to understand, little...
they are limited in terms of users, functions, operations, and classes [Baker, 1996].
There are specific tools to assist in identifying conflicts between policies and
in the identification of regions not covered by the set of policies (both restrictively,
as permissibly), as in [Jaeger et al., 2003], however such tools only
they assist in the verification of policies, and not in their definition.
To reduce the complexity of implementing SELinux, some
distributions now provide the core of this as optional, and instructions on how to
stala-lo [Gilburd, 2003], providing even some basic policies for the main
programs used. However, the use of these policies makes the administrator's work difficult
policy administrator, who first needs to understand the meaning of the thousands of
pre-configured definition lines before you can modify and extend them.

6. Final Considerations
As seen throughout the article, the security issues in operating systems
that are currently occurring are associated not only with the applications but also with
security paradigms on which the most commonly used operating systems are based
common, that cannot contain the failure solely the application. Originating from a
generation where the risks were much lower and isolated, these systems do not have the
necessary security for the current computational framework, whose expansion of networks in character
global has created an environment where attacks are becoming increasingly frequent. However,
the substitution of operating systems cannot be carried out quickly, due to the
applications already developed and widely used.
To ensure the security of operating systems, it is necessary to focus on
attention to new paradigms. Not that network security should be relaxed or
that either the applications should not go through stages of development aiming at
the identification and elimination of errors, because even individually these applications still
can cause problems for users. In this sense, secure programming techniques
must be improved and disseminated, and there is still much to be done in terms of
network topologies, firewalls, among others.
The key point of this article is that it is time to look more closely.
the security that can be obtained from operating systems, without relying on
auxiliary applications. It´ is necessary for practices such as those being carried out in
Linux to adapt to the new paradigms (SELinux) to be adopted by others
operating systems in use, reinforcing the existing barriers and limiting access
uncontrolled obtained from successful attacks on applications.
Additionally, security policies should be studied, aiming to obtain
clearer and easier to define policies, to enable their validation.
tools that simplify the work of defining and verifying policy conflicts
should be developed and improved, aiming not only at the management in a
compact but complete security policies, but also extending to the user
the possibility of creating additional policies to meet your needs.
Future work that needs to be carried out involves more in-depth studies
of the new operating systems (including SELinux), research of new ways
of representation and definition of policies and the research and development of tools
that simplify policy management. In this way, the use will be enabled
of new large-scale paradigms.

References

Baker, D. (1996). Fortresses built upon sand. In Proceedings of the New Security
Paradigms Workshop.
Bell, D. E. and Padula, L. J. L. (1973). Secure computer systems: Mathematical foundation
Technical report, The MITRE Corporation, Bedford, MA, USA.
CERT (2003). Cert/cc statistics 1988-2003. Available at Invalid text for translation.
org/stats/>. Accessed on: 02/16/04.
DoD (1985). Trusted Computer Security Evaluation Criteria. Department of Defense.
DOD 5200.28-STD.
Holiday, R. J. and Organick, E. (1971). The multics input-output system. In Proceedings
of the Third Symposium on Operation Systems Principles, pages 35–41, New York.
Ferraiolo, D. and Kuhn, R. (1992). Role-based access control. In Proceedings of The 15th
National Computer Security Conference.
Ford, B., Hibler, M., Lepreau, J., and Godmar Back, P. T., and Clawson, S. (1996). Micro-
kernels meet recursive virtual machines. In Proceedings of the 2nd USENIX Symposium
on Operating Systems Design and Implementation.
Garfinkel, S. and Spafford, G. (1996). Practical Unix & Internet Security. O’Reilly &
Associates, Inc., USA, 2nd edition. 971 p.
Gilburd, Z. (2003). Gentoo x86 SELinux installation guide. Available
http://www.gentoo.org/proj/en/hardened/selinux/
selinux-x86-install.xml>.
Harrison, M. A., Ruzzo, W. L., and Ullman, J. D. (1976). Protection in operating systems.
Commun. ACM, 19(8):461–471.
Jaeger, T., Zhang, X., and Cacheda, F. (2003). Policy management using access control
spaces.ACM Trans. Inf. Syst. Secur., 6(3):327–364.
Liedtke, J. (1996). L4 Reference Manual. IBM T. J. Watson Research Center. RC 20549.
Loscocco, P. and Smalley, S. (2001a). Integrating flexible support for security policies
into the linux operating system. In Proceedings of the FREENIX Track: 2001 USENIX
Annual Technical Conference, Boston Mass.
Loscocco, P. and Smalley, S. (2001b). Meeting critical security objectives with security-
enhanced linux. In Proceedings of The 2001 Ottawa Linux Symposium, Ottawa, Ont.,
Canada.
Loscocco, P.A., Smalley, S. D., Muckelbauer, P.A., Taylor, R. C., Turner, S. J., and
Farrel, J. F. (1998). The inevitability of failure: The flawed assumption of security
in modern computing environment. In Proceedings of the 21st National Information
Systems Security Conference, pages 303–314.
Mazieres, D. and Kaashoek, M. (1997). Secure applications need flexible operating systems.
tems. InProceedings of the 6th Workshop on Hot Topics in Operating Systems.
Nakamura, E. and de Geus, P.L. (2002). Network security in cooperative environments.
Berkeley Publishing, São Paulo, first edition.
Ritchie, D. M. and Thompson, K. (1974). The unix time-sharing system. Commun. ACM,
17(7):365–375.
Secure Computing Corporation (1998). Assurance in the fluke microkernel: Formal se-
curity model. Technical report, Secure Computing Corporation. MD A904-97-C-3047
CDRL A003.
Silberschatz, P.B., Galvin, P.B., and Gagne, G. (2002). Operating System Concepts.
John Wiley & Sons, Inc., New York, 6th edition. 887 p.
Swift, M. M., Hopkins, A., Brundrett, P., Van Dyke, C., Garg, P., Chan, S., Goertzel, M.
and Jensenworth, G. (2002). Improving the granularity of access control for windows
2000.ACM Trans. Inf. Syst. Secur., 5(4):398–437.
Computer Networks
edition.
Tanenbaum, A. S. and Woodhull, A. S. (1997). Operating systems: Design and Implementation
mentation. Prentice Hall, New Jersey - USA, 2nd edition.
Walker, K. W., Bagder, D. F., Petkac, M. J., Sherman, L., and Oostendorp, K. A. (1996).
Confining root programs with domain and type enforcement. In Proceedings of The
6th USENIX Security Symposium, San Jose, California.