About
Articles

A Minimal OS X kext Written (Partly) in Rust

Originally published on 30th March 2014.

Preamble

Most of my work involves writing C, C++ and/or Objective-C code. For the most part, this isn't my preferred choice of language[1] - it happens the only practical choice of the problem and solution domain I tend to inhabit: operating system kernels and systems programming. For some reason, all the programming languages which enable you to go to such a low level seem to be decades old and were created before software security was more than an academic curiosity.

There are now some new kids on the block though, which are trying to take a more sensible approach of blending safety with power, and expressiveness with simplicity. One that has been gathering a lot of buzz lately is Mozilla's Rust. Another that has caught my eye is ATS, but I shall focus on that one another time.

I do a lot of kernel-level development, and safety there is of even more interest than in applications. So every now and then I take a look to see if there is a realistic alternative to C (and C++) for use in kernel development. Rust has been used to create both a basic Linux kernel module, rust.ko and more recently Julia Evans' standalone (toy) OS kernel. As it happens, I've picked up more clients who want me to write or maintain OSX kernel extensions than Linux kernel code, so if there's any chance I might be able to use a new language for serious work, it will need to work on OSX/xnu. I've wanted to try to "port" rust.ko to OSX for some months now, and have only just got around to trying. Here's what I've found.

The core: kernel hello world in Rust

My first step was to grab the rust compiler, version 0.9, from the website and build and install it. Easy enough. Then I read the first few sections of the rest of the tutorial and tried the few working examples that it does provide, to give me a basic feel for the language. Then, on to kernel business.

rust.ko defines a rust_main() function, written in Rust, which writes "hello from rust" to the kernel log. Printing is done via a print() function, which is set up to call down to the C function printk(). The code for all this is in main.rs, and my goal was to replicate that for OSX.

My first goal was compiling rust.ko's main.rs file to a standalone object file. The tutorial only covered regular executables, so my next port of call was rust.ko's Makefile. Its rustc invocation of $(RC) -O --lib -o $@ -c $< didn't quite work. 0.9 rustc doesn't like -o. Okay, easily fixed, I'm happy with the default name anyway.

Next problem: #[fixed_stack_segment] is apparently obsolete. Well, removing it certainly fixes the compile error. Apparently, Rust used to use a segmented stack approach, and #[fixed_stack_segment] switched to a stack segment large enough to call into C code. Using one big stack is now the default in Rust, but the comment on this page about 2MB stacks did worry me somewhat, as kernel stacks in xnu are all of 16KiB.

Anyway, that now produced a nice little main.o file. Inspecting it with nm revealed that rust_main was indeed exported, and printk was listed as a dependency. There's no printk in the OSX kernel, there's only printf/IOLog and kprintf. Replacing kprint with printf caused problems. It seems that rustc, or possibly llvm, have some kind of hardcoded knowledge of a function with this name. Luckily, IOLog does the same thing, so I used that instead.

N.B.:

  • The first argument to IOLog/printf is of course a format string, not a raw string. As I'm only printing a single literal string in this example, I've chosen to overlook this vulnerability for now.
  • Rust strings aren't null-terminated, so I think the original code might incorrectly be relying on there being a 0 byte in place. I added a \0 to the end of the string in my version just to be sure. There is a string to C string conversion mechanism in the Rust standard library, and probably in rust-core, which would be the tidy solution, but for now this will do.

Setting the stage

The other part of rust.ko is stub.rs. The directive #[no_std]; at the top of main.rs disables the whole of the Rust standard library. Much like the kernel doesn't contain a full libc, we can't expect to encounter a full Rust standard library. Even very basic rust code has some dependencies, and kernel modules (and kexts) need some extra boilerplate to satisfy the kernel's dynamic linker.

Starting from the top:

Line 7

char __morestack[1024];

Uh oh, another ominous reference to the stack. It's not mentioned in the rest of the file, so it must be satisfying the linker somehow. Searching the web yielded this assembly code and associated commentary in the standard Rust runtime. The name seems to be a legacy of dynamic stack expansion, which is no longer done, and now is called when stack overflow is detected. I don't know why the author of rust.ko implemented it as an array when it's expected to be a function, but I suspected that even implementing it as a function was hiding a bigger problem. So I took a look at the generated code:

ko-main.o:
(__TEXT,__text) section
_rust_main:
0000000000000000        cmpq    %gs:0x330, %rsp
0000000000000009        ja      0x25
000000000000000b        movabsq $0x8, %r10
0000000000000015        movabsq $_rust_main, %r11
000000000000001f        callq   ___morestack
0000000000000024        ret
0000000000000025        pushq   %rbp
0000000000000026        movq    %rsp, %rbp
0000000000000029        leaq    _str111(%rip), %rdi
0000000000000030        popq    %rbp
0000000000000031        jmpq    _printk

Slightly odd calling convention notwithstanding, my understanding of what is happening here is that the incoming stack pointer is compared to the thread-local stack base, and if we have reached the base of the available stack memory, __morestack() is called to handle the overflow situation. (presumably the exact size it checks for will depend on the actual stack frame size of the current function) Presumably by terminating the process or thread; judging by the ret instruction after the call, it is not expected that __morestack will fix the overflow.

Long story short, thread-local storage doesn't exist for kernel threads, and %gs:0x330 does not hold the stack base pointer in the xnu kernel. In my tests, there was a NULL pointer there, so if that were guaranteed to be the case, this would mean the test always passes, as it makes the code believe there is an effectively infinite stack.

I used the following code to inspect it:

void* base = NULL;
__asm__("movq %%gs:0x330, %0" : "=r"(base) : : );
void* rsp = NULL;
__asm__("movq %%rsp, %0" : "=r"(rsp) : : );
IOLog(
    "%%gs:0x330: %p\n"
    "%%rsp:      %p\n", base, rsp);

I don't know what that memory area is used for in the kernel, though, so who knows if we can rely on this behaviour. Besides, it's a waste of wired memory and CPU cycles to keep on doing this check when it doesn't even work. In my experience, stack overflows in the kernel instantly kernel panic with a triple-fault as the preceding virtual memory page is not mapped. (essentially the guard area model of overflow detection) [2]

Now, my web searches on the topic revealed that the stack overflow check can be disabled by annotating function definitions with #[no_split_stack]. With the annotation, rust_main is reduced to the following, as you'd expect.

(__TEXT,__text) section
_rust_main:
0000000000000000        pushq   %rbp
0000000000000001        movq    %rsp, %rbp
0000000000000004        leaq    _str112(%rip), %rdi
000000000000000b        popq    %rbp
000000000000000c        jmpq    _printk

This needs to be added to every single function though, I couldn't find a file-wide or compiler flag equivalent. Trying not to forget this (and retrofitting libraries with it) will become annoying quickly, I'm sure. Apparently, the stack check is added late enough in compilation that it can be avoided altogether by compiling to LLVM bytecode instead, and compiling that bytecode with clang. (more on this later on in the article)

Line 8

char _GLOBAL_OFFSET_TABLE_;

This is used for position independent code (PIC) in the ELF binary format. OS X uses Mach-O, so I removed it from the kext version and crossed my fingers.

Lines 10-23

Function definitions for abort(), malloc() and free() that call down to their Linux kernel equivalents.

These are (obviously) basic libc function implementations. They aren't actually needed in the current version of main.rs, so presumably, these are leftovers of the original authors' experiments? Or maybe earlier compiler versions' output required them? In any case, these will need to change for OSX kexts, so while they're not needed I've removed them altogether.

Line 25

extern void rust_main(void);

This is the C prototype for the Rust function of the same name. We'll keep it!

This works because the function is annotated with #[no_mangle] in Rust. Much like C++, Rust encodes a function's type and namespace in its symbol name, "mangling" its raw name. #[no_mangle] is the equivalent of extern "C" in C++, which sidesteps this mechanism, making the function easily callable from C.

Lines 27-42

  • kmod init/exit functions - These generate some debug output, and the init function calls rust_main.
  • kernel module boilerplate macros.

There are direct OS X kext equivalents for all of these; the Xcode "Generic Kernel Extension" project template generates them for us, and we just need to declare and call rust_main():

#include <mach/mach_types.h>

void rust_main(void);

kern_return_t RustyKext_start(kmod_info_t * ki, void *d);
kern_return_t RustyKext_stop(kmod_info_t *ki, void *d);

kern_return_t RustyKext_start(kmod_info_t * ki, void *d)
{
    rust_main();
    return KERN_SUCCESS;
}

kern_return_t RustyKext_stop(kmod_info_t *ki, void *d)
{
    return KERN_SUCCESS;
}

Easy enough. We could probably have skipped the indirection via C for the start function and directly implemented it in Rust. That's probably a nicer long term solution, but it does mean we need Rust definitions for the kmod_info_t and kern_return_t types and values.

Compiler flags

There are some differences between binaries for kernel extensions and user space code. Code for kexts is therefore compiled using the following extra clang arguments:

-fno-builtin -msoft-float -fno-common -mkernel

Some of these are actually redundant, as -mkernel expands to -static, -fno-common, -fno-builtin, -fno-cxa-atexit, -fno-exceptions, -fno-non-call-exceptions, -fno-asynchronous-unwind-tables, -fapple-kext, -fno-weak and -fno-rtti. It also implies -mno-red-zone on x86-64.

Most of these deal with C++-specific things (disabling exceptions and RTTI and causing slightly different vtables), or are only relevant at the linking stage, when we can use Apple's linker anyway. -msoft-float isn't strictly required: on OSX, it is possible to use floating-point (or SIMD) code in the kernel. However, the -msoft-float flag changes the calling convention for floating point function arguments, and the rest of the kernel does use the flag. Moreover, executing an FP/SIMD instruction causes a permanent change in the way a thread's context changes are done, so this flag is there to avoid generating FP code by accident, as you want to limit its use to specific threads only. The rust compiler understands a -Csoft-float flag, which has a similar effect, so I recommend using that unless you know what you're doing.

The only compiler flag that really worried me is -mno-red-zone. Not enabling this in C code used in a kext on x86-64 will cause spurious stack memory overwrites. The Red Zone exists in some x86-64 ABIs and dictates that signal or interrupt handlers move the stack pointer (%rsp) by 128 bytes before clobbering stack memory. This means that regular leaf functions can put temporary data on the stack past the top of the stack without having to change %rsp and then change it back again. Kernel stacks are tiny, and 128 bytes are precious, so Apple decided to knock the red zone on the head in xnu. Interrupt handlers will overwrite stack memory starting directly after %rsp, so any code assuming that memory is safe is playing with fire.

rustc is LLVM-based, so I assumed I might be able to tell the backend about the lack of red zone: -mno-red-zone is originally from gcc so clang copied its name, but llc knows it as -disable-red-zone. The mailing list thread about Rust stacks also mentions a rustc argument -Cllvm-args= which lets you pass arguments through to the LLVM backend. I had to compile the latest compiler version from git as 0.9 doesn't seem to support this yet. No matter how hard I tried, it would not accept -disable-red-zone though, and from searching through the source code I conclude that although that flag exists internally, it is never parsed or used in rustc. I wrote a few functions in rust that encouraged values to be put on the stack. All such experiments generated code that moved the stack pointer for such temporaries, and I never saw an instance where the existance of a red zone was assumed.

So I think at least as of the current version, the red zone is never used or assumed, so we should be safe. That's not to say that this might change in the future, or that I might have missed something.

Much like the stack overflow check issue mentioned earlier, we can use clang to perform the final stage of compilation though. So compiling rust code for xnu becomes:

rustc -O --emit bc --crate-type staticlib -Csoft-float ko-main.rs
clang -mkernel -msoft-float  -c ko-main.bc

For some reason, in this combination, -Csoft-float and -msoft-float seem to be ignored, at least with the compiler versions I'm using, and floating-point instructions are emitted when floats are used in the Rust code. Fortunately, this isn't something you're likely to do accidentally in kernel code.

Putting it all together

So, now we have some Rust code, the requisite compiler flags, and some glue code in C. We just need to link the Rust code with the C code, add some kext linkage boilerplate and put it in a .kext bundle with an info.plist. This is kind of tedious to do by hand, and Xcode does this all out of the box, so I decided to use its build system for this experiment as well.

Starting with a suitably named, clean "Generic Kernel Extension" project, I did the following:

  • Disabled code signing for the kext target.
  • Updated the C code to call the Rust function as described earlier.
  • Added a "custom script" build rule for Rust source files (*.rs) to the kext target.

Script:

mkdir -p "${DERIVED_FILE_DIR}/rust-bin"

/usr/local/bin/rustc --out-dir "${DERIVED_FILE_DIR}/rust-bin" -O --emit bc --crate-type staticlib "${INPUT_FILE_PATH}" && "${DT_TOOLCHAIN_DIR}/usr/bin/clang" -mkernel -msoft-float -c "${DERIVED_FILE_DIR}/rust-bin/${INPUT_FILE_BASE}.bc" -o "${DERIVED_FILE_DIR}/rust-bin/${INPUT_FILE_BASE}.o"

Irritatingly, Xcode mangles line-continuation characters (\) in the script box, so you have to put the compilation on one giant, horrible line.

(These compiler arguments are for the 0.10-pre rustc, git master at time of writing. These things seem to change from week to week.)

The 2 output files are:

$(DERIVED_FILE_DIR)/rust-bin/${INPUT_FILE_BASE}.bc
$(DERIVED_FILE_DIR)/rust-bin/${INPUT_FILE_BASE}.o
  • Added the main.rs file to the project, with the adapted code from rust.ko's equivalent file.
  • Xcode added it to the "Copy Bundle Resources" build phase, so I moved it to "Compile Sources"
  • Set up the bundle dependencies. Remember we used IOLog() from the Rust code? This means we need to add both com.apple.kpi.iokit and com.apple.kpi.libkern to the OSBundleLibraries dictionary in the Info.plist with appropriate versions (10.0.0 aka Snow Leopard will do the trick).

That's it, the kext should now build. To load it, copy it to the destination machine (or VM), ensure it has root/wheel permissions (I use sudo cp -r RustyKext.kext /tmp/) and load it (sudo kextutil /tmp/RustyKext.kext). Assuming your system hasn't crashed by this point, look at the system log in Console.app or so, and you should find a "hello from rust" message!

Conclusion, limitations and further work

While very satisfying, this is only the very first step. You'll discover very quickly that using almost any Rust language feature, and attempting to write useful code of any kind will quickly lead to compile or link errors because various language infrastructure is missing in this freestanding mode. This is where rust-core comes in, and probably writing a considerable amount of xnu-specific glue code, for example for memory allocations, etc. Certain things will presumably never work, such as Rust's user thread model (switching stacks in the kernel is a no-no) and thus probably also some of its thread communication techniques. There are of course also a large number of kernel APIs that will need to be imported into Rust. I don't know if there are any mechanisms for #including C headers directly, and if not, replicating those APIs by hand will be slow and error-prone.

Finally, beware that Rust and its compiler, especially the lower level bits, are still very much in flux, so expect your code to frequently be broken by new compilers. Getting code to compile for the kernel is a bit of a challenge - C compilers like GCC and clang are heavily influenced by Linux and OS X kernel development, respectively, so support for kernel mode is very good. I hope kernel development eventually becomes a priority in Rust as well, as it could be an interesting alternative.

I'm still a Rust newbie myself, but I hope to expand on this base to do something useful with Rust in the kernel at some point. I'll try to write it up if I do, but if you get there before me, do let me know! Same goes for any mistakes or omissions discovered in the above of course.

I have placed the sample Xcode project, RustyKext, on GitHub, feel free to use it as a base for experimentation or even serious work.

Acknowledgements

Thanks to Chris Williams for reading a draft of this article and for his helpful input.

Footnotes:

1.

Many people seem to hate C++ but claim that C or Objective-C are far superior. I have found that none of the languages in this group are what they could, maybe should be. In particular, safety continues to be a problem - it's 2014 and buffer overflows are still a security issue. It's just so easy to shoot yourself in the foot with C, and while some say this is by design, it seems that most of the time you don't really want to shoot yourself in the foot. What you want is very low level access to memory and other system resources, but in practice only in very localised parts of your code. Manual memory management is another one of these things that makes C what it is. In practice, this frequently means memory leaks due to forgotten deallocation, or worse, dangling pointers. What you actually want is tight control over resource usage: no accidental memory allocation, direct control over lifetime when you need it. All of these things and more are requirements for system-level software, at least on these von Neumann machines that are popular with the kids today.

2.

I'll admit that a kernel panic isn't exactly "failing cleanly." The only alternative would be to unwind the stack (longjmp or exception style) to a point in the code that can do something about it. It's probably more practical to just write your code so that it uses as little stack space as possible and comes nowhere near the 16KiB limit. Deeply recursive functions are out anyway (except when tail-recursive).