It has taken us some time, but yesterday we migrated the last of the Xen virtual machines over to KVM! Everything is running smoothly and so far all customers seem to be happy.
As far as performance is concerned: I was expecting KVM to be just a little bit slower than Xen, because it doesn’t support paravirtualization. If you believe the benchmarks all over the web this should cost between 5% and 10% CPU performance.
This doesn’t really matter to us, because the bottleneck over here (and I guess almost everywhere) is disk (and sometimes network) performance, and those can be paravirtualized with Linux/Windows virtio drivers anyway (which we do, of course).
Our experience was actually quite the contrary: We’ve seen a performance increase! This can be easily explained by the fact that with Xen, all memory that is not assigned to a domain is not used, whereas with KVM all that memory can be used by the Linux kernel of the hypervisor as diskcache. This means less hits on the disks, hence more disk performance. Great!
(P.S: That last thing isn’t strictly true for Xen in all cases, but we had to configure our Xen dom0’s with a max memory of 1GB, because we noticed that the Xen hypervisor wasn’t always able to take away memory from the dom0 when it needed to create a new domain.)