I’ve been poking at Rusty’s hypervisor, lhype, which he’s developing as example code for the upcoming paravirt ops framework. The lhype patch applies cleanly to recent -mm kernels, and I’m queuing up some patches for Rusty here (or browse the git repo). The latest patch there adds support for Thread Local Storage, allowing things like Fedora binaries to run without segfaulting.

[ NOTE: an updated version of this procedure is now published here ]

If anyone wants a very quick and simple recipe to get up and running with lhype, try this:

1. Obtain an appropriate kernel and lhype patches¹:

    $ git clone git://git.infradead.org/~jmorris/linux-2.6-lhype

2. Check out the ‘patches’ branch:

    $ cd linux-2.6-lhype
    $ git-checkout patches

3. Configure the kernel:

    $ make menuconfig

Keep it simple initially, to get up and running. Also keep in mind that lhype currently only works for i386, no high memory and no SMP. Enable the paravirt ops framework and select lhype as a module:

    CONFIG_PARAVIRT=y
    CONFIG_LHYPE=m

4. Build and install the kernel:

    $ make -j12 && sudo make modules_install && sudo make -j12 install

Or whatever usually works for you. Note that the kernel you’re building will be for both the guest and the host domains.

5. Boot into host kernel (dom0)

This should just work. Now the fun really begins.

6. Grab a nice Linux OS image²:

    $ wget http://fabrice.bellard.free.fr/qemu/linux-test-0.5.1.tar.gz
    $ tar -xzf linux-test-0.5.1.tar.gz linux-test/linux.img

7. Create a file for shared network i/o:

    $ dd if=/dev/zero of=netfile bs=1024 count=4

8. Launch a guest!

    $ sudo modprobe lhype
    $ sudo linux-2.6-lhype/drivers/lhype/lhype_add 32M 1 linux-2.6-lhype/vmlinux \
            linux-test/linux.img netfile root=/dev/lhba

If all went well, you should see the kernel boot messages fly past and then something like:

Linux version 2.6.19-rc5-mm2-gf808425d (jmorris@xeon.namei)
(gcc version 4.1.1 20060928 (Red Hat 4.1.1-28)) #1 PREEMPT Tue Nov 28 00:53:39 EST 2006

QEMU Linux test distribution (based on Redhat 9)

Type 'exit' to halt the system

sh: no job control in this shell
sh-2.05b#

It’s already useful for kernel hacking. Local networking works. It’d also likely be useful for teaching purposes, being relatively simple yet quite concrete.

‘lhype_add’ is an app included with the kernel which launches and monitors guest domains. It’s actually a simple ELF loader, which maps the guest kernel image into the host’s memory, then opens /proc/lhype and writes some config info about the guest. This kicks the hypervisor into action to initialize and launch the guest, while the open procfile fd is used for control, console i/o, and DMA-like i/o via shared memory (using ideas from Rusty’s earlier XenShare work). The hypervisor is simply a loadable kernel module. Cool stuff.

It’s a little different to Xen, in that the host domain (dom0) is simply a normal kernel running in ring 0 with userspace in ring 3. The hypervisor is a small ELF object loaded into the top of memory (when the lhype module is loaded), which contains some simple domain switching code, interrupt handlers, a few low-level objects which need to be virtualized, and finally an array of structs to maintain information for each guest domain (drivers/lhype/hypervisor.S).

The hypervisor runs in ring 0, with the guest domains running as host domain tasks in ring 1, trapping into the hypervisor for virtualized operations via paravirt ops hooks (arch/i386/kernel/lhype.c) and subsequent hypercalls (drivers/lhype/hypercalls.c). Thus, the hypervisor and host kernel run in the same ring, rather than, say, the hypervisor in ring 0 with the host kernel in ring 1, as is the case with Xen. The advantage for lhype is simplicity: the hypervisor can be kept extremely small and simple, because it only needs to handle tasks related solely to virtualization. It’s just 463 lines of assembler, with comments. Of course, from an isolation point of view the host kernel is effectively part of the hypervisor, because they share the same hardware privilege level. It has also been noted that in practice, a typical dom0 has so much privileged access to the hypervisor that it’s not necessarily meaningful to run them in separate rings. Probably a good beer @ OLS discussion topic.

Overall, it seems like a very clean and elegant design. I’ll see if I can write up some more detailed notes on what I’ve learned about it soon.

Note that Rusty will be giving a presumably canonical talk on lhype at LCA 2007.

The OSDL Virtualization list is probably the best place to keep up with development at this stage, as well as Rusty’s accurately titled bleeding edge page.

¹ You can also roll your own via the paravirt patch queue, where the core development takes place.
²Qemu image suggested for simplicity. e.g. an FC6 image will now work, although it won’t get past single user mode due to lack of support for initrd.

***

On a hypervisor-related note, IBM researcher Reiner Sailer has posted an ACM/sHype HOWTO for Fedora Core 6, which explains how to enable the current Xen security extensions.

***

While I was doing some ELF reading, I found this great document, A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux.

***

Looks like FOSS.IN/2006 was another big success.
o Photos
o Google news query

The nutters who did this:

resulting in the best bugzilla ever,

want to go here:

How to help.

About nine months ago, I set up selinuxnews.org to host a blog aggregator and news blog for the SELinux project. Today, I thought it’d be interesting to see how many hits the site was getting and ran the logs through webalizer:

(more detail …)
It’s been fairly consistent, with about 2300 hits and 600 visits per day, not including syndicated access, which seems difficult to estimate. Also, it’s interesting to discover that most hits on the site were for RSS and similar feeds. So much for HTML.

SELinux

Dan Walsh has been giving some RHEL security presentations recently, and I thought the slides were pretty cool, so here they are in PDF form. They include a nice overview of the comprehensive security features available in RHEL, including SELinux, ExecShield and PIE. The slides are obviously written from a Red Hat point of view, but also apply to Fedora and any of the distros incorporating the same features.

I’ve been reviewing papers for the SELinux Symposium, and am very impressed with the quality and scope of the activity within and around the SELinux community. Much progress continues to be made in many areas, including usability (as can also be seen by tracking SELinux News).

The release of Fedora Core 6 has unleashed many of the SELinux infrastructure and usability improvements, such as modular policy and setroubleshoot. There’ve been some nice reviews, like Fedora Core 6: Innovations Continue from eweek:

Review: The fast-moving Red Hat distribution polishes SELinux, adds new tools and improves performance.

In its first five releases, Red Hat’s Fedora Core has represented the Linux technology vanguard. And so it is with Fedora Core 6.

During tests, Fedora Core 6 impressed eWEEK Labs with the progress it has made toward making Security-Enhanced Linux—and the dramatically improved security protections that SELinux helps afford—more palatable.

Of course, it’s not perfect yet, but it’s reassuring to see that people will come back and re-evaluate things you’ve been working to improve.

One analogy I like to use is to think of SELinux as a major, fundamental change in computer security, with the integration of Mandatory Access Control into general purpose computing. Like any significant change, it can be a long and difficult road, and really a matter of effort and persistence once you have the fundamentals in place. So, the progress of SELinux is perhaps like the development of programming languages, where you had a progression from machine code to assembler, then to higher level languages like C, Perl and Ajax. I’d say, with SELinux, that we’re past the assembly and 3GL phase and are moving to scripting and graphical IDEs. Just like you wouldn’t say computers are too complicated because Joe Random can’t understand the Intel® 64 and IA-32 Architectures Software Developer’s Manual, it is nonsensical to say that SELinux is fundamentally too complicated because of its underlying architecture. In fact, SELinux is merely revealing the true extent of the complexity of the security interactions in the OS, and providing a flexible mechanism to control it. Usability is entirely a matter of developing the right abstractions. You can’t pretend complexity isn’t there, but you can hide it. On the other hand, if your fundamentals are wrong, no amount of eye candy can help.

Virtualization

FC6 also features a bunch of improvements in virtualization (e.g. virt-manager and Cobbler), and I hope that we can make Fedora the place to go for the latest and greatest in this area, too (if it’s not already?) I’ve created a virtualization category on the Fedora Project wiki, to collect together the various related pages there. One area that I’m directly involved in is hypervisor research & development. We’ve moved some of our internal (RH) project planning & tracking stuff out to the wiki: see here. If you’re interested in this area, the Fedora project will warmly welcome any contributions.

Conferences

I’m sad to say I won’t be able to make it to FOSS.IN this year, due to family commitments. It’s one of a few top-tier technical conferences, alongside the likes of LCA and Linux-Kongress, and is especially exciting because of the stunning rate of growth of the community in India. (btw, these global Internet growth stats are very interesting, where India has had over 1000% Internet growth since 2000, which I suspect correlates strongly with FOSS growth). The Final List of Talks looks fantastic.

I wonder which country will be the next to establish a major Linux technical conference. (Japan?)

Rusty’s Moustache

Is reportedly making a comeback.

I am one of Jim Gettys’ Jedi mind trick victims, with a record breaking answer of ten seconds when asked how long I thought it takes for Linux to wake up from suspend-to-ram. Out by a mere four orders of magnitude. It’s always good to remember: never assume anything, and don’t be a guesser.

I usually write up some notes on all the SELinux changes in a new kernel release, but I’ve been a little busy with RHEL5 coming up, and the notes for 2.6.18 will have to wait a while. I’m happy, though, to see that secmark (reworked SELinux network controls, integrated with iptables) went in.

The FOSS.IN/2006 CFP is open for another week. I’ve submitted a couple of talks and hope to attend one way or another. Due to a change of venue, I don’t think there’ll be cows walking around the conference this year, but you can’t have everything. It’s a truly amazing event, and I’d say it ties with LCA as favorite Linux conference.

Jeff Waugh linked to an interesting presentation blog, which features an inspired Darth Vader vs. Yoda example, which is itself a great example of presentation “zen”.

There’s a week left, too, on the SELinux Symposium 2007 CFP. A lot of great work has been done on usability and developer tools during the year, and the symposium will be an useful opportunity to see how the project has advanced overall and where its heading.

Erich Schubert has been keeping the SELinux effort in Debian moving forward; and SELinux itself, with Debian leading the deployment of auto-configuring modular policy. (Note: see comments for more details on this). As one of my sparc boxes is running Debian, I hope to get it updated to a development version and have a closer look at what’s happening there. If you’re interested in SELinux on Debian, see the project page.

The BSD securelevels LSM has finally been terminated.

In an ideal world, I’d love to see more of the Linux security development community rally around SELinux, rather than be split across various projects. Aside from believing that SELinux is the right model on a fundamental level, with its policy-flexibility and complete coverage of security-relevant interactions in the kernel, it takes a lot of work to make the transition from DAC to MAC in a general purpose operating system. It’s such a hard problem that it’s never succeeded before, and I guess I see SELinux as the big opportunity to nail it; with the availability of the OS source code, a global developer team and a solid architecture based on decades of research by the NSA — all of the building blocks are there.

Competing ideas are of course essential, but it’s one thing to convert a research idea into an LSM and publish some papers; and quite another to build developer, user and academic communities, to continually evolve the software on every level, and to support many tens of millions of users across several widely deployed OSs, in environments ranging from consumer telephones, to Internet servers, through to operational systems protecting national security and all manner of amazing things that I wouldn’t be able to talk about, even if I knew about them.

So, it’s not that I think BSD securelevels or some other security project is a bad thing (although, honestly, some are not so wonderful), it’s a matter of what’s practical and suitable for inclusion in the mainline Linux kernel. To make an analogy from the networking world: it’s good that things like STREAMS exist, and that people have done real work on things like TCP offload engines, but it doesn’t mean that they belong in the mainline Linux kernel.

Slides from all of the talks at Netconf 2006, and photos, are being made available at
http://vger.kernel.org/netconf2006.html (bottom of the page).

It’s been a great summit again, and it’s very sad to be leaving Tokyo. Thanks to all of the sponsors, and especially Yoshifuji-san of USAGI.

DaveM made a new friend at the University of Tokyo:

Update:
Uploaded a photo set from the conf: click here.

Here are the slides from my Netconf 2006 talks.

• Better IPsec SA Resolution (PDF)
  Some discussion of the case where we need to send a packet, but have no IPsec security association for it. Generally, we currently return EAGAIN to applications and initiate an SA negotiation, but this seems less than optimal. One approach is to not return an error (which some implementations do) so that applications don’t die from surprise. We could also queue the packet and wait for the negotiation to complete, with appropriate semantics for various scenarios.

  The big question at this point is: in what situations are people seeing this? Generally, the key manager will maintain SAs as needed, and the case of not having one established is believed to be unusual, but we don’t really know for sure. Opportunistic encryption would likely benefit from proper queuing, but it’s not clear how widely it’s deployed.

• OLPC Networking Overview (PDF)
  Brain dump of what I understand currently of the networking aspects of the OLPC project, which I thought might be of interest to other kernel network developers.
• Mandatory Access Control Networking Update (PDF)
  Coverage of developments in MAC networking over the last year, including the new Secmark controls & SELinux, completion of the native xfrm IPsec labeling to meet LSPP/EAL4 , and CIPSO support. Paul Moore from HP will be giving a more detailed talk on CIPSO today or tomorrow.

Also, met up with some local SELinux developers at lunch time: Kaigai Kohei , Kazuki Omo and Yuichi Nakamura. After noticing a six-page SELinux article in a Japanese Linux magazine last night while shopping in Akihabara Electric Town, I asked them if they’d seen it and it turns out that Nakamura-san wrote it, and that he writes a monthly SELinux column for the magazine. We had some discussion about SELinux usability, SE postgresql and JFFS2 support. Unfortunately, we ran out of time — hopefully we’ll get to talk again at the next SELinux symposium.

(Which reminds me to note that the CFP submissions are due on October 9th).

I wonder how many people know that there’s an official Netfilter Song.

I’m in Tokyo for Netconf 2006.

Harald Welte told me at breakfast that Motorola A1200 phones run SELinux, and that a million of these have been sold in China. It’d be interesting to see their security policy.

Update:
The A1200 is not running real SELinux, it’s a derived LSM called “Motorola Access Control”. The source is available here:
https://opensource.motorola.com/sf/frs/do/listReleases/projects.a1200/frs.sample_package

Wade Mealing blogged Do the Fedora Developers run with SELinux enabled ?, after encountering a lot of AVC messages on his Fedora Core box. Looking at the messages, it seems that the system is out of whack generally and needs to be relabeled. The cleanest way to do this is:

# yum update
# touch /.autorelabel
# reboot

and the early init scripts should fix the labels.

The hint that it’s a general labeling problem is the presence of file_t and unlabeled_t labels in the AVCs, which are generic fallback values and should not typically be seen in the wild. In fact, any AVC messages or SELinux issues for a normal user should be regarded as a bug.

This type of thing should not normally ever happen and we’d really like to know how the system got into this state. It could be that the policy has not been kept up to date with the rest of the system, which should only really be an issue for people who are playing with development versions of the distro and selectively upgrading rpms. It’s also possible that the presence of hard disk error messages in the logs has something to do with it. Another possible cause is mounting a non-labeled disk somewhere critical in the fs. Without more detailed information, we don’t know for sure, so please always report bugs.

Any of the following mailing lists are good for this:

• http://www.redhat.com/mailman/listinfo/fedora-list
• http://www.redhat.com/mailman/listinfo/fedora-selinux-list (subscriber-only)
• http://www.nsa.gov/selinux/info/list.cfm

SELinux-related bugzilla entries are typically resolved very quickly:

• https://bugzilla.redhat.com/bugzilla/index.cgi

Bugs are good.

Send us some!

Note that soon, setroubleshoot will have a plugin for this specific issue, and explain to the user what’s wrong and what to do about it.

A great security book, Security Engineering is now available online for free. The author, Ross Anderson, convinced Wiley to make it available so that as many people as possible can have access.

Bruce Schneier is quoted `If you’re even thinking of doing any security engineering, you need to read this book’. I’d go further and extend this to any kind of engineering. I think it’s one of those rare books that pretty much everyone involved in computers or technology in general should read (another might be Brooks’ The Mythical Man Month). Security Engineering is packed with knowledge and deep insight across a wide range of topics. It’s also comprehensively referenced, making it a great stepping off point for further study.

For those interested in learning more about historical Mandatory Access Control, MLS and other advanced security models, the Multilevel Security chapter is worth reading. While only touching on Type Enforcement, it provides a lot of background material for understanding the historical context of SELinux.

Also, Ross has a blog.