The arrow of time

Ivan Voras' blog

VMWare + FreeBSD + ZFS soft-raid with SATA drives - performance

VMWare ESXi is a great free-as-in-beer product bringing professional virtualization within everyone's grasp. Contrasted to VMWare Server 2, I'm yet to find something really serious in ESXi,to rant about, but the lack of software RAID support comes very near. For some strange reason ESXi doesn't support software RAID configurations (officially it's all about "reliability" but I suspect they're either not interested in building management products for sw RAID into the various management "consoles", because sw RAID appears to be at the low end of the market or it's something to do with the ESX's dubious interaction with the GPL).

Because of chronic money shortage at this job, I've configured one single-CPU VMWare host with 2 SATA drives and was later surprised when I couldn't mirror them within the host OS (i.e. ESXi). The alternative is atrocious but in these circumstances it's the best I could do: I create virtual hard drives, one on each physical drive, and do software mirroring within the guest OS. I have no idea how this setup will behave on drive failure; I suppose at the very least there is absolutely no chance of doing anything resembling hot replacement of hardware drives.

Anyway, in this particular instance I've set up 8.0-RC1 and used ZFS for mirroring. One of the reasons for this is to enable ZFS compression for actual data sets, which turned out nicely though not as nice as expected (only 1.25x compresssion ratio - I guess the used data simply doesn't compress much). The IRL performance is, as expected, nothing to brag about in any aspect, and bonnie++ just confirms it.

I have two cases within the same pool: without compression and with gzip compression and the (interesting parts of...) benchmark results are:

Sequential output: without compression: 30 MB/s / with compression: 23 MB/s
Sequential rewrite: without compression: 16 MB/s / with compression: 21 MB/s
Sequential input: without compression: 60 MB/s / with compression: 96 MB/s
Create files (random or sequential): without compression: 7600 files/s / with compression:

The SATA drives individually (and without VMWare emulation) easily pull between 50 and 90 MB/s. Bonnie++ reports CPU usages of around 20% which are completely bogus; measuring with 'top' reveals  much larger CPU usage, almost completly in "sys" time (50% for uncompressed, 100% for compressed dataset). Obviously, bonnie++ writes easily compressible data - possibly zeroed blocks, which results in higher read rates for compressed data.

I really do not recommend this configuration to anyone who can pull an alternative - absolutely every aspect of it is suboptimal: two SATA drives, VMWare introducing its own IO slowdowns, mirroring inside the guest VMs - both performance and reliability suffer greatly. Additionally, this setup is very problematic to use with rdiff-backup (or, I suspect, any such utility working over ssh) because ZFS in combination with VMWare's slow IO results in frequent "pauses" in the operating system where it is for all purposes locked up for several seconds.

On a bit older "raw" (no virtualization) system running 7.1 with the old ZFS (ZFSv6) with three SATA drives in RAIDZ, achieved performance is around 40 MB/s for writes and 125 MB/s for reads (both without compression).

#1 Re: VMWare + FreeBSD + ZFS soft-raid with SATA drives - performance

Added on 2010-11-23T03:37 by jadz

I appreciate your posting. I'm tring to do something like this at home to have flexibility in adding and removing disks while maintaining a Windows 7 host system (for some dependancies I have on the host).

#2 Re: VMWare + FreeBSD + ZFS soft-raid with SATA drives - performance

Added on 2011-10-06T22:52 by Orleen

I created similar design with FreeNAS 8x64 over ESXI 5.0. This allowed me to create a ZFS array of four SATA drives, 2TB Each. As you can guess, this is my backup storage, so speed is not a priority. I'm replicating virtual hosts with VEEAM Backup and Replication 5, which reports speed of !00 mb/s or slightly more at night, when the network is iddle, and between15-50 MB/s at busy time. It would be great, if VMWare was able to create ZFS arrays.

Post your comment here!

Your name:
Comment title:
Type "xxx" here:

Comments are subject to moderation and will be deleted if deemed inappropriate. All content is © Ivan Voras. Comments are owned by their authors... who agree to basically surrender all rights by publishing them here :)