I recently built a VMware ESXi host at home. When I was researching the hardware, I learned there are a number of things to consider when choosing a RAID card for use under ESXi. This article covers those things and offers advice for anyone who is building a similar system.
Anyone who has stood up ESXi in a enterprise data center would probably scoff at the idea of putting storage directly in the host machine. That's ok, this article isn't for you. This article is for people building ESXi in a lab environment or maybe in a small business setting where a) clustering/HA isn't a consideration and b) there is no shared storage present.
My plan was to use inexpensive, high capacity SATA drives to store all the VM images. Since I wasn't willing to accept the risk of running everything on a single drive, my plan was to have two drives setup in a mirror and present a mirrored volume to ESXi. I was already familiar with presenting LUNs off a SAN for ESXi to use, but hadn't yet seen a host built using local storage so I wasn't sure what my options were or their pros and cons.
There are basically three types of solutions that I explored, listed here in order of lowest to highest in terms of cost:
- On-board SATA controller RAID
- SATA/SAS host bus adapter with RAID support
- Dedicated RAID card
On-board SATA Controller RAID⌗
Every new motherboard for the past number of years has offered RAID support using the on-board SATA chipset. This is usually seen as an easy, low cost (could even consider it "no cost") way of getting RAID into your machine. Even server class motherboards made by companies like Super Micro and Intel support RAID on their SATA ports. These ports are connected to the southbridge on the motherboard with the southbridge acting as the SATA controller. Modern motherboards have southbridge chipsets with names like ICH10R for Intel boards and SB750 for AMD boards. When enabling RAID on these chipsets, they utilize a combination of the southbridge hardware and a software driver to operate the RAID array. This means that ICH10R and SB750 based RAID arrays are heavily dependent on the type of operating system that's running — the OS must have an appropriate driver that enables the RAID functionality. This is what's known as "software RAID" because of the reliance on a software component.
ESXi's support for software RAID is pretty clear: there is none. The only way you can use on-board SATA ports is in non-RAID mode. This eliminates on-board RAID as an option.
SATA/SAS Host Bus Adapter With RAID Support⌗
The next best option to using the on-board ports is to add additional ports via a SATA or SAS HBA. Some really common examples of such cards include LSI's SAS30x1E-R or Super Micro's AOC-USAS-x cards as well as LSI's 921x or Super Micro's AOC-USAS2-x cards. These cards are branded as RAID adapters and utilize LSI's 1068E and SAS2008 chipsets, respectively. Despite the marketing of these cards, they operate similarly to the on-board SATA controllers — they require a software driver in order to operate in RAID mode which makes them ineligible for use as RAID controllers with ESXi. They do however provide between 4 and 8 high performance SATA/SAS ports complete with more features and options than the ICH10R/SB750 ports.
When reviewing SATA/SAS HBAs, consider these points when trying to determine if it's a software RAID card:
- Does the card utilize one of these very popular chipsets which only support software RAID? (eg, LSI 1064E, 1068E or SAS2008) (Yeah I'm picking on LSI chipsets here but that's only because I researched them the most)
- Is the card simply rebranded and really using one of the above chipsets under the hood (such as the Intel SASWT4I/SASUC8I or Super Micro cards)?
- Does the card lack its own processor (aka, RAID On Chip, ROC) for doing parity calculations?
If you can answer "yes" to any of these questions then you are likely looking at a card that only does software RAID.
Dedicated RAID Card⌗
A dedicated RAID card is just that: a card that does all RAID functions itself, no software required. These cards are often identified by being able to answer "no" to the set of questions posed in the section above. You can also usually spot a RAID card at twenty paces just by looking at its price tag. These cards typically start at 1.5x - 2x the price of a SATA/SAS HBA with the same number of ports and go up from there.
Generally speaking, hardware RAID cards are well supported under ESXi which, after ruling out the previous two solutions, made them the only viable solution for my build. The only thing you need to check on is that the card you're considering is listed on the VMware HCL.
Other Considerations for a Dedicated RAID Card⌗
Do a search in almost any VMware related forum and you'll see posts from people talking about poor performance when running their data store on a directly attached RAID array. The response is almost always "do you have write-back turned on?" Enabling write-back tells the RAID controller to cache writes in its on-board memory and indicate to the OS (ie, ESXi) that the data has been successfully written to disk. Storing the data in memory takes a fraction of the time required to send the data to the disk. At some point in the future, the card then sends that data from memory down to the disk, but the advantage is that ESXi isn't sitting waiting for that to happen, its moved on to other things by then. Write-back greatly increases random write operations which are highly characteristic in a VM environment.
Now, you may wonder, since the data is stored for a time in volatile memory, what happens if the machine crashes or the power is lost before the data is sent to the disk? That's where the battery backup unit (BBU) comes in. The BBU is like a small UPS for your RAID card; it provides battery power so that the memory on the card stays online even if the computer itself is off. That keeps the non-committed data safe in memory until the machine is back online and the data can be written to disk.
A hardware RAID card that is listed on the VMware HCL is the only viable solution for running an ESXi host with local storage. Consider how many SATA/SAS ports you need when sizing the card and also carefully consider whether you should install a battery backup-up unit to eliminate the risk of power supply issues. Be aware that not all SATA/SAS cards are true RAID cards, even if the marketing material says they are. Ask the vendor and do your research online before you purchase.