Even though it is fairly well known that system call wrappers are dangerous, many security schemes continue to use them, including several well-known commercial security products from major vendors.

Briefly, the technique involves intercepting user requests at the system call level, then performing some security function and perhaps failing the call. A common example is re-vectoring the system call table so that a virus scanner is invoked when a file is opened. The Linux system call table was unexported long ago to discourage such use, although there’s no way to actually prevent it.

There are many problems with such techniques. One significant class arises from the combination of concurrency, and gaps in time between when an object is checked and when it is used. For example, an application may subvert the security mechanism by passing clean data to be checked, then replacing it with malicious data before it is used. This is called a Time of Check to Time of Use race, or TOCTTOU.

Robert Watson, of FreeBSD and TrustedBSD fame, recently presented a good paper on the topic, which he discusses here: USENIX WOOT07, Exploiting Concurrency Vulnerabilities in System Call Wrappers, and the Evil Genius.

In the paper, problems with system call wrapping are comprehensively detailed, while the slides also include several examples of exploit code.

One interesting case, which I think would surprise many, is the ease with which their sudo could be subverted.

With Sudo on MP systems, the window for execve() arguments was over 430K cycles. We were able to successfully exploit this vulnerability, replacing command lines so that they were incorrectly logged, masking all further attacker activity in the audit trail.

The paper also discusses ways to mitigate the problem, including not using system call wrapping at all, and instead directly integrating mediation into the kernel; which is what SELinux does.