Currently Tilaa runs on the Xen 3.3 hypervisor with a custom built Linux kernel. When we started building our infrastructure in the beginning of 2009 it was the most mature opensource option there was, even though were was this new thing called KVM, which seemed like it had a long way to go before it could replace Xen. And there were also plans to merge the Xen dom0 specific bits, which are needed for the privileged Linux domain that manages the Xen hypervisor, upstream in the Linux kernel. The future looked bright indeed!
A lot has happened since then. To make a long story short, the patches were rejected because they were either too intrusive, caused a performance hit on native Linux kernels or simply because the Linux kernel hackers thought the design was flawed. We’ve been patiently waiting for a miracle to happen, but it looks like the patches are not going to make it into mainline any day soon.
To make things worse, it appears that the Xen management daemons have some locking issues and crash once in a while when the system is under load. We made a great effort trying to iron these issues out and we now have some workarounds in place to ensure our customers don’t suffer from these issues, but it’s not an acceptable solution in the long term. (For the record: We’re perfectly stable and haven’t had a single crash. We’re talking long term strategy here:)
In the meanwhile Citrix opensourced it’s Xen Cloud platform, which differs from the community Xen hypervisor. Citrix rewrote the Xen management daemons in Ocaml, which greatly improves performance and stability apparently. The downside it that they only provide an ISO, which includes everything you need to build the stack. But it’s close to impossible to get this properly packaged for a common Linux distribution. Also, all development is apparently still behind closed Citrix doors and there’s almost no community involvement in the development process. Not a good solution to bet your company’s future on.
In other news it looks like the OpenSolaris community is moving away from xVM (which is based on Xen) to VirtualBox, Novell announces it will provide KVM in their next SLES product, RedHat is pushing RHEL5 customers over to KVM as we speak (and will probably drop Xen support in RHEL6) and no other mainstream Linux distribution supports Xen out of the box (but most do provide KVM).
So we have currently started testing with KVM and started to adapt our toolchain to support it. This was actually easier than we expected, but fortunately we anticipated a move to another virtualization solution early in our development stage.
As it looks now we will probably start migrating our own Tilaa virtual machines to a server running KVM somewhere between 2 and 3 months from now and let it run for a couple of weeks to ensure everything is stable. After that, we will migrate all customer VM’s to the new platform. Ofcourse, when we have a more definitive timeschedule we will inform our customers. We will try to keep our blog up to date with the most recent developments.
The bottom line of this post: If you are working on a Linux based open-source virtualization project, save yourself some time and use KVM. Xen is still an interesting platform, but it’s future is too uncertain right now (at least for us).