Porting Linux KVM to FreeBSD, step by step

This page

This page is an edited version of the commit log for the project source code repository. Here you will find a detailed history for the project and some notes on what has been done, in addition to links to the web interface of the SCM used.

Even if Perforce should have been used for that (and it will be, of course,) by now we are using git for two main reasons: we need the ability to work offline for long periods, and we want to import easily changes made on the main KVM repository, that is tracked with git.

Prepare for code revision

Mon Aug 20 17:34:05 2007 +0000

This is the snapshot of the code that will be reviewed. Still misses proper locking and SMP operation.

commit

Big ugly changes to make things work

Sun Aug 19 01:18:31 2007 +0000

Fix a lot of small and not-so-small bugs and problems, and implemented what was left. Now it should work in UP, on SMP it makes the system crash if qemu executed unpinned, death-slow otherwise... that's definitely not good, anyway it's something to work on.

commit

Implement some missing calls

Thu Aug 9 23:32:16 2007 +0000

Now every call should have been implemented, we just have to get things working...

commit

Reflect into userspace the API mmap() change

Tue Aug 7 22:09:07 2007 +0000

commit

Implement the mmap() on the VM file descriptor as an ioctl()

Tue Aug 7 22:03:48 2007 +0000

Since we cannot hook the mmap() call to the open VM file descriptor we change the API adding an ioctl() doing the same things. Since from a module we cannot execute our code when a pagefault occurs without writing great amounts of code, and we know in advance the physical frames that will be mapped (assuming that the physical memory is not reconfigured before it is first used, but this is the case with the userspace client,) we immediately insert the pages into the newly mapped areas.

commit

Call lkvm module exit functions on unload

Sun Aug 5 15:05:56 2007 +0000

commit

Revised page allocation/deallocation

Sat Aug 4 20:13:23 2007 +0000

Now we use always vm_phys_alloc_contig(), and we keep track of allocated pages in a hash table, to implement pfn_to_page(). Assuming this is working the last issue should be allowing calls to kfree()/vfree()/__free_pages() inside IPI handlers, since those calls can block. A deferred deallocation mechanism should be a good solution.

commit

Use the safe copy mechanism for ioctl()s

Sat Aug 4 20:09:04 2007 +0000

commit

Make get_cpu()/put_cpu() use sched_pin()/sched_unpin()

Sat Aug 4 20:07:39 2007 +0000

The differences between Linux and FreeBSD strike again. On Linux to pin a task to a cpu we use preempt_disable(), while on FreeBSD the same thing is handled at the scheduler level.

commit

Implement get_cpu()/put_cpu()

Thu Aug 2 20:55:58 2007 +0000

commit

Implement __free_pages()

Thu Aug 2 20:39:31 2007 +0000

The whole memory allocation thing should be revised, along with locking. By now this works, even if multi-page allocations are not treated in the same way as one-page ones.

commit

Support __GFP_ZERO allocations

Thu Aug 2 13:38:41 2007 +0000

This also contains a fix to a typo in __GFP_WAIT handling. __GFP_WAIT is still a little bit meaningless because we still don't do proper locking.

commit

Execute ioctl()s on falloc() created files

Thu Aug 2 01:56:00 2007 +0000

Now we are able to start the execution of the remaining ioctl()s used by KVM, mapping each Linux struct file into its FreeBSD equivalent. The common code with the miscdevice ioctl() implementation has been factored out.

commit

Comment out spinlock and mutex implementations

Wed Aug 1 11:39:21 2007 +0000

The current spinlock and mutex implementations have several problems, mainly related to the expected behavior of codepaths protected by locks and to lock deallocation (shown under WITNESS.) By now just ignore the problem, in the near future sleep mutexes will be used for Linux spinlocks and sema(9) primitives will be used for Linux mutexes.

commit

Let KVM_CREATE_VM ioctl() complete

Wed Aug 1 11:35:08 2007 +0000

Add some fs related functions, such as file/inode allocation and file descriptor allocation and installation. This implementation is KVM oriented and is not general.

commit

Introduce an implementation for get_current()

Sat Jul 28 19:27:17 2007 +0000

We just use a statically allocated struct task_struct and return a pointer to it. This only works when only a single thread accesses linux-kmod-compat, but a correct solution would be too complex for the actual uses of the current pointer in KVM.

commit

Let kvm_init_arch() terminate

Fri Jul 27 13:56:10 2007 +0000

commit

Comment out semaphore implementation

Tue Jul 24 12:36:16 2007 +0000

Using sleep mutexes as a replacement for Linux semaphores is incorrect, since sleeping inside them is not allowed and they don't work (obviously) for semaphores with count not starting from one. Will move to sema(9).

commit

Reintroduce a Linux compatible redefinition of msleep()

Tue Jul 24 12:08:32 2007 +0000

To reduce the modifications needed to compile Linux drivers we reintroduce msleep(), even if that may cause problems in the general case.

commit

Update types in ldev_stub.c to CURRENT

Tue Jul 24 11:08:47 2007 +0000

A couple of macros are not defined anymore in the USB core code, so update ldev_stub.c to reflect this change. Also include usb_port.h that now must be used explicitly.

commit

Accept USB short transfers by default

Sun Jul 22 18:51:46 2007 +0000

This is the standard Linux behavior, replicate it in linux-kmod-compat.

commit

Use the old copy_from_user/copy_to_user implementation

Fri Jul 13 18:06:05 2007 +0000

The new, ioctl() safe, version imposes too much overhead and is often not needed, so if a module needs it, that module has to explicitly ask for its use.

commit

Fix a bad typo on mutex destruction

Wed Jul 11 02:38:08 2007 +0000

Those things should not happen.

commit

Sync userspace with changes in kernel interface

Sat Jul 7 23:35:08 2007 +0000

Make kvmctl.c use the new interface we had to use for KVM_GET_MSR_INDEX_LIST ioctl().

commit

Change kernel interface of KVM_GET_MSR_INDEX_LIST ioctl()

Sat Jul 7 23:19:27 2007 +0000

Since KVM_GET_MSR_INDEX_LIST returns an error while copying to userspace a return value in its argument we need the copyout in kern/sys_generic.c ioctl() to take place. That needs kern_ioctl() to return 0, so the KVM error is passed through a field of the argument structure.

commit

Uniform recursive (g)make calls

Sat Jul 7 22:45:09 2007 +0000

commit

Call initialization/exit of compatibility library from modules

Sat Jul 7 19:57:17 2007 +0000

The new copy_from_user() and copy_to_user() implementation requires functions to be called on module load and unload, make our three modules call them.

commit

Make copy_from_user() and copy_to_user() work in ioctl()s

Sat Jul 7 17:41:26 2007 +0000

Linux does not special case the copy in/out of user parameters of ioctl()s, so the implementation of copy_from_user() and copy_to_user() has to avoid the copyin/copyout if called inside an ioctl(). This is done registering the parameter areas allocated by the (FreeBSD) kernel on ioctl() entry and on each copy from/to userspace check if the requested area is in that list; if so use memcpy instead of copyin/copyout. This breaks old users of linux-kmod-compat.

commit

Add support for miscdevice calls

Tue Jun 26 04:33:04 2007 +0000

Now the calls to the registered miscdevice reach the Linux code. We allow miscdevices to be opened at any given time by at most one user, since this simplifies memory handling and it is not a problem at all with KVM. Add also deregistering a miscdevice. Here we also sync the definition of alloc_pages with CURRENT, using vm_phys_alloc_contig to allocate contiguous physical pages.

commit

Make Linux KVM userspace compile on FreeBSD

Sat Jun 23 19:37:26 2007 +0000

Compile libkvm and then enable KVM support on qemu, so that it uses the library.

commit

Make KVM userspace configure and Makefile work on FreeBSD

Sat Jun 23 17:16:01 2007 +0000

Still do not compile libkvm, only modify the build environment to work on FreeBSD.

commit

Patched qemu to compile on FreeBSD

Sat Jun 23 16:52:38 2007 +0000

We applied the patches of ports/emulators/qemu to the userspace client in kvm-17 to make it compile on FreeBSD. Since the patches needed slight modifications their updated version is tracked on the git repository. We did not use the qemu port since KVM patches are still not in mainline.

commit

Imported Linux KVM userspace support

Sat Jun 23 16:20:42 2007 +0000

The userspace support is composed by a modified qemu, used basically to load the guest OS, initialize the system (memory, devices, etc...) and then handle all the exits from the emulation mode that the kernel cannot handle (mainly IO operations,) and a library that interfaces qemu with the ioctl() calls exported by the kernel modules. As it has been done with the kernel module, we stick to a stable release of the Linux code (compatible with the code used in the kernel modules,) and use kvm-17.

commit

Added misc_register support, with stub cdevsw functions

Wed Jun 20 10:40:19 2007 +0000

The Linux misc device descriptor is extended with two more fields to link it to a character device created with make_dev(9). By now still do not call the Linux file ops, but only stubs.

commit

Add page_address() implementation

Sun Jun 17 22:15:57 2007 +0000

commit

Add some SMP support functions

Sun Jun 17 22:14:32 2007 +0000

commit

Add rdmsrl and wrmsrl, using FreeBSD equivalents

Sun Jun 17 22:13:02 2007 +0000

commit

Add alloc_pages implementation

Fri Jun 15 15:03:15 2007 +0000

When allocating multiple physical pages we use vm_page_alloc_contig. There are locking issues to be resolved, but they need a better understanding of the locking mechanisms used in the vm subsystem than the one I have now. GFP_ATOMIC would allow memory allocations inside a spinlock, while vm_page_alloc* functions lock the page queues, that are protected by sleepable mutexes.

commit

Add cpuid() implementation

Thu Jun 14 16:57:16 2007 +0000

We don't use the FreeBSD equivalent to avoid unnecessary parameter copies and the code is pretty straightforward.

commit

Add support for boot_cpu_data

Thu Jun 14 16:35:04 2007 +0000

This poses the problem of global variables, since boot_cpu_data should be a shared global symbol, that would need a) a global linux-kmod-compat module, or b) a copy of each global as a static variable of each module (works for readonly vars.) Since this point needs to be discussed a little, just let boot_cpu_data work by now, reading the correct value on the fly.

commit

Fixed comments and style for alloc_page()

Sat Jun 9 21:25:06 2007 +0000

commit

Let kvm_init terminate regularly

Thu Jun 7 03:25:58 2007 +0000

commit

Add an initial version of alloc_page

Thu Jun 7 03:24:30 2007 +0000

This can really be improved. For now just use wired pages to prevent problems with paging, as this is the way Linux KVM works. This changeset brings in another dependency on amd64: Linux uses a direct map for physical pages (when possible,) so the definition of __va() can be remapped quite easily on amd64, but not on i386.

commit

Implement rdmsr_safe, amd64 only

Sat Jun 2 10:12:34 2007 +0000

commit

Let the modules load, even if they won't do anything useful

Tue May 29 13:03:21 2007 +0000

commit

Compile three modules

Tue May 29 12:34:21 2007 +0000

The modules built still don't load correctly.

commit

Give modules the ability to exclude ldev_stub.c

Mon May 28 21:42:22 2007 +0000

I needed this to add explicit module dependencies between the lkvm components without passing from the USB support.

commit

Adapt KVM to linux-kmod-compat requirements

Sun May 13 20:18:23 2007 +0000

commit

Add stubs to linux-kmod-compat to compile KVM

Sun May 13 20:14:56 2007 +0000

commit

Fix linux_msleep using the FreeBSD macro definition

Sat May 12 20:54:11 2007 +0000

This is not really a fix, since linux_msleep worked on FreeBSD 6.x; switching to 7.0 it can not be defined as a function anymore.

commit

Add semicolons at the end of module_init/module_exit definitions

Sat May 12 20:52:02 2007 +0000

commit

Fix the name clash for list manipulation macros

Sat May 12 20:50:11 2007 +0000

A _compat suffix is appended to Linux identifiers when there is a name collision. This breaks backwards compatibility for sources using linux-kmod-compat.

commit

Add module makefile to lkvm

Sat May 12 20:46:17 2007 +0000

commit

Move KVM user interface headers into lkvm module source tree

Sat May 12 16:42:06 2007 +0000

commit

Import Linux KVM as in kernel 2.6.21

Sat May 12 16:41:03 2007 +0000

The idea is to stick with it until we get something working, then, if there is enough time before the end of the SoC, rebase the work trying to track Linux major releases.

commit

Import linux-kmod-compat-20070326

Sat May 12 16:40:33 2007 +0000

commit

Script last updated June 7th, 2007