Most discussions of software protection in the OS community focus on memory protection: guaranteeing a program will not access memory or execute code for which it is not authorized. Software fault isolation , proof-carrying code , and type-safe languages are three popular ways to ensure memory protection.
Lucco, et al., introduced software fault isolation  in 1993. They showed they could rewrite load, store, and branch instructions to validate all memory access with an aggregate 5-30% slowdown. By eliminating these checks where possible, and optimizing the placement of the remaining checks, a perhaps acceptable slowdown is achieved in a software-only implementation. The theory is still the same as hardware-based protection: potentially dangerous operations are dynamically checked for safety before their execution.
Necula and Lee introduced proof-carrying code  in 1996. Proof-carrying code eliminates the slowdown associated with software fault isolation by statically verifying a proof that a program respects an agreed upon security policy when the program is loaded. After the proof is verified, the program can run at full speed. Few limits are put on the form of the program; it can be hand-tuned assembly language. The major difficulty at the moment is generating the proofs: for now, they are written by hand with the help of a non-interactive theorem prover. Research is underway to automate proof generation. One promising idea is to have the compiler for a safe language generate a proof of memory safety during the compilation process.
The language-based protection model, exemplified by Java, can be traced back at least to the Burroughs B-5000 in 1961  and is also used in current research operating systems such as SPIN . Memory accesses can be controlled with a safe language . If the language has a mechanism for enforcing abstraction boundaries (in Java, this means ensuring that private variables and methods are not accessible outside their class), then the system can force control to flow through security checks before executing any dangerous code. Language-based safety can use either dynamic type checking [38,6] or static type checking (as is mostly the case in Java [18,9]). The major tradeoff is performance versus theoretical complexity. The static system should be faster, because it only checks safety once, before the program starts. Dynamic checking is simpler, because there is no need to prove a type soundness theorem: at each step during execution, a local check is sufficient to ensure the absence of type errors. Note that while static type checking may be theoretically complex, the implementation can be quite simple.
While memory protection is necessary and important, there is much more to security than memory protection. Memory protection is sufficient for programs that do calculations without invoking system services, but more interesting programs use a rich set of operating system services such as shared files, graphics, authentication, and networking. The operating system code implementing these services must define and correctly enforce its own security policy pertaining to the resources it is managing. For example, the GUI code must define and enforce a policy saying which code can observe and generate which GUI events. Memory protection can keep hostile code from directly reading the GUI state, but it cannot keep a hostile program from tricking the GUI code into telling it about events it shouldn't see.
Seltzer, et al. , studied some of the security problems involved in creating an extensible operating system. They argue that memory protection is only part of the solution; the bulk of their paper is concerned with questions of how to provide secure services. Consider the security flaws that have been found in Unix. Very few are related to hardware memory protection: almost all of the flaws have been in trusted services such as sendmail and fingerd. Perfect memory protection would not prevent these flaws.
The challenge, then, is not only getting memory protection but providing secure system services. This paper considers how secure services can be built solely with software protection.