Porting Linux KVM to FreeBSD, step by step

This page is an edited version of the commit log for the project source code repository. Here you will find a detailed history for the project and some notes on what has been done, in addition to links to the web interface of the SCM used.

Even if Perforce should have been used for that (and it will be, of course,) by now we are using git for two main reasons: we need the ability to work offline for long periods, and we want to import easily changes made on the main KVM repository, that is tracked with git.

Mon Aug 20 17:34:05 2007 +0000

This is the snapshot of the code that will be reviewed. Still misses proper locking and SMP operation.

commit

Sun Aug 19 01:18:31 2007 +0000

Fix a lot of small and not-so-small bugs and problems, and implemented what was left. Now it should work in UP, on SMP it makes the system crash if qemu executed unpinned, death-slow otherwise... that's definitely not good, anyway it's something to work on.

commit

Thu Aug 9 23:32:16 2007 +0000

Now every call should have been implemented, we just have to get things working...

commit

Tue Aug 7 22:09:07 2007 +0000

commit

Tue Aug 7 22:03:48 2007 +0000

Since we cannot hook the mmap() call to the open VM file descriptor we change the API adding an ioctl() doing the same things. Since from a module we cannot execute our code when a pagefault occurs without writing great amounts of code, and we know in advance the physical frames that will be mapped (assuming that the physical memory is not reconfigured before it is first used, but this is the case with the userspace client,) we immediately insert the pages into the newly mapped areas.

commit

Sun Aug 5 15:05:56 2007 +0000

commit

Sat Aug 4 20:13:23 2007 +0000

Now we use always vm_phys_alloc_contig(), and we keep track of allocated pages in a hash table, to implement pfn_to_page(). Assuming this is working the last issue should be allowing calls to kfree()/vfree()/__free_pages() inside IPI handlers, since those calls can block. A deferred deallocation mechanism should be a good solution.

commit

Sat Aug 4 20:09:04 2007 +0000

commit

Sat Aug 4 20:07:39 2007 +0000

The differences between Linux and FreeBSD strike again. On Linux to pin a task to a cpu we use preempt_disable(), while on FreeBSD the same thing is handled at the scheduler level.

commit

Thu Aug 2 20:55:58 2007 +0000

commit

Thu Aug 2 20:39:31 2007 +0000

The whole memory allocation thing should be revised, along with locking. By now this works, even if multi-page allocations are not treated in the same way as one-page ones.

commit

Thu Aug 2 13:38:41 2007 +0000

This also contains a fix to a typo in __GFP_WAIT handling. __GFP_WAIT is still a little bit meaningless because we still don't do proper locking.

commit

Thu Aug 2 01:56:00 2007 +0000

Now we are able to start the execution of the remaining ioctl()s used by KVM, mapping each Linux struct file into its FreeBSD equivalent. The common code with the miscdevice ioctl() implementation has been factored out.

commit

Wed Aug 1 11:39:21 2007 +0000

The current spinlock and mutex implementations have several problems, mainly related to the expected behavior of codepaths protected by locks and to lock deallocation (shown under WITNESS.) By now just ignore the problem, in the near future sleep mutexes will be used for Linux spinlocks and sema(9) primitives will be used for Linux mutexes.

commit

Wed Aug 1 11:35:08 2007 +0000

Add some fs related functions, such as file/inode allocation and file descriptor allocation and installation. This implementation is KVM oriented and is not general.

commit

Sat Jul 28 19:27:17 2007 +0000

We just use a statically allocated struct task_struct and return a pointer to it. This only works when only a single thread accesses linux-kmod-compat, but a correct solution would be too complex for the actual uses of the current pointer in KVM.

commit

Fri Jul 27 13:56:10 2007 +0000

commit

Tue Jul 24 12:36:16 2007 +0000

Using sleep mutexes as a replacement for Linux semaphores is incorrect, since sleeping inside them is not allowed and they don't work (obviously) for semaphores with count not starting from one. Will move to sema(9).

commit

Tue Jul 24 12:08:32 2007 +0000

To reduce the modifications needed to compile Linux drivers we reintroduce msleep(), even if that may cause problems in the general case.

commit

Tue Jul 24 11:08:47 2007 +0000

A couple of macros are not defined anymore in the USB core code, so update ldev_stub.c to reflect this change. Also include usb_port.h that now must be used explicitly.

commit

Sun Jul 22 18:51:46 2007 +0000

This is the standard Linux behavior, replicate it in linux-kmod-compat.

commit

Fri Jul 13 18:06:05 2007 +0000

The new, ioctl() safe, version imposes too much overhead and is often not needed, so if a module needs it, that module has to explicitly ask for its use.

commit

Wed Jul 11 02:38:08 2007 +0000

Those things should not happen.

commit

Sat Jul 7 23:35:08 2007 +0000

Make kvmctl.c use the new interface we had to use for KVM_GET_MSR_INDEX_LIST ioctl().

commit

Sat Jul 7 23:19:27 2007 +0000

Since KVM_GET_MSR_INDEX_LIST returns an error while copying to userspace a return value in its argument we need the copyout in kern/sys_generic.c ioctl() to take place. That needs kern_ioctl() to return 0, so the KVM error is passed through a field of the argument structure.

commit

Sat Jul 7 22:45:09 2007 +0000

commit

Sat Jul 7 19:57:17 2007 +0000

The new copy_from_user() and copy_to_user() implementation requires functions to be called on module load and unload, make our three modules call them.

commit

Sat Jul 7 17:41:26 2007 +0000

Linux does not special case the copy in/out of user parameters of ioctl()s, so the implementation of copy_from_user() and copy_to_user() has to avoid the copyin/copyout if called inside an ioctl(). This is done registering the parameter areas allocated by the (FreeBSD) kernel on ioctl() entry and on each copy from/to userspace check if the requested area is in that list; if so use memcpy instead of copyin/copyout. This breaks old users of linux-kmod-compat.

commit

Tue Jun 26 04:33:04 2007 +0000

Now the calls to the registered miscdevice reach the Linux code. We allow miscdevices to be opened at any given time by at most one user, since this simplifies memory handling and it is not a problem at all with KVM. Add also deregistering a miscdevice. Here we also sync the definition of alloc_pages with CURRENT, using vm_phys_alloc_contig to allocate contiguous physical pages.

commit

Sat Jun 23 19:37:26 2007 +0000

Compile libkvm and then enable KVM support on qemu, so that it uses the library.

commit

Sat Jun 23 17:16:01 2007 +0000

Still do not compile libkvm, only modify the build environment to work on FreeBSD.

commit

Sat Jun 23 16:52:38 2007 +0000

We applied the patches of ports/emulators/qemu to the userspace client in kvm-17 to make it compile on FreeBSD. Since the patches needed slight modifications their updated version is tracked on the git repository. We did not use the qemu port since KVM patches are still not in mainline.

commit

Sat Jun 23 16:20:42 2007 +0000

The userspace support is composed by a modified qemu, used basically to load the guest OS, initialize the system (memory, devices, etc...) and then handle all the exits from the emulation mode that the kernel cannot handle (mainly IO operations,) and a library that interfaces qemu with the ioctl() calls exported by the kernel modules. As it has been done with the kernel module, we stick to a stable release of the Linux code (compatible with the code used in the kernel modules,) and use kvm-17.

commit

Wed Jun 20 10:40:19 2007 +0000

The Linux misc device descriptor is extended with two more fields to link it to a character device created with make_dev(9). By now still do not call the Linux file ops, but only stubs.

commit

Sun Jun 17 22:15:57 2007 +0000

commit

Sun Jun 17 22:14:32 2007 +0000

commit

Sun Jun 17 22:13:02 2007 +0000

commit

Fri Jun 15 15:03:15 2007 +0000

When allocating multiple physical pages we use vm_page_alloc_contig. There are locking issues to be resolved, but they need a better understanding of the locking mechanisms used in the vm subsystem than the one I have now. GFP_ATOMIC would allow memory allocations inside a spinlock, while vm_page_alloc* functions lock the page queues, that are protected by sleepable mutexes.

commit

Thu Jun 14 16:57:16 2007 +0000

We don't use the FreeBSD equivalent to avoid unnecessary parameter copies and the code is pretty straightforward.

commit

Thu Jun 14 16:35:04 2007 +0000

This poses the problem of global variables, since boot_cpu_data should be a shared global symbol, that would need a) a global linux-kmod-compat module, or b) a copy of each global as a static variable of each module (works for readonly vars.) Since this point needs to be discussed a little, just let boot_cpu_data work by now, reading the correct value on the fly.

commit

Sat Jun 9 21:25:06 2007 +0000

commit

Thu Jun 7 03:25:58 2007 +0000

commit

Thu Jun 7 03:24:30 2007 +0000

This can really be improved. For now just use wired pages to prevent problems with paging, as this is the way Linux KVM works. This changeset brings in another dependency on amd64: Linux uses a direct map for physical pages (when possible,) so the definition of __va() can be remapped quite easily on amd64, but not on i386.

commit

Sat Jun 2 10:12:34 2007 +0000

commit

Tue May 29 13:03:21 2007 +0000

commit

Tue May 29 12:34:21 2007 +0000

The modules built still don't load correctly.

commit

Mon May 28 21:42:22 2007 +0000

I needed this to add explicit module dependencies between the lkvm components without passing from the USB support.

commit

Sun May 13 20:18:23 2007 +0000

commit

Sun May 13 20:14:56 2007 +0000

commit

Sat May 12 20:54:11 2007 +0000

This is not really a fix, since linux_msleep worked on FreeBSD 6.x; switching to 7.0 it can not be defined as a function anymore.

commit

Sat May 12 20:52:02 2007 +0000

commit

Sat May 12 20:50:11 2007 +0000

A _compat suffix is appended to Linux identifiers when there is a name collision. This breaks backwards compatibility for sources using linux-kmod-compat.

commit

Sat May 12 20:46:17 2007 +0000

commit

Sat May 12 16:42:06 2007 +0000

commit

Sat May 12 16:41:03 2007 +0000

The idea is to stick with it until we get something working, then, if there is enough time before the end of the SoC, rebase the work trying to track Linux major releases.

commit

Sat May 12 16:40:33 2007 +0000

commit