Bad news: there is a bewildering number of choices for the type of flash which are typically classified by:
- the underlying storage technology (e.g., SLC, MLC, TLC, etc.)
- the expected use case (e.g., consumer, data center, enterprise, read-optimized, write-optimized, etc.)
- device interconnect (e.g., SATA, SAS, PCI-e, NVMe, etc.)
Good news: with a Datrium DVX solution you can use modern, read-optimized, higher capacity drives from your server vendor to keep costs low and usable capacity high.
Let’s examine the details.
- Prices have been steadily declining so you can add what you need today at today’s price and add more if needed tomorrow at tomorrow’s (lower) price.
- Cost per GB is relatively linear within a family of devices so you can add 2x as much without paying more than 2x the cost.
- DVX provides inline data reduction on the host with ranges we have observed from 2x to 6x. Allowing for some operational overhead for the hyperdriver and using an average ~4x reduction, we can see that server side flash effective costs are getting dramatically better.
- If the workload is flash friendly like DVX is, then you won’t need to invest in higher end, high endurance, expensive devices to support the host caching operations.
- Read performance is the primary driver for using a flash host cache. Put the data closer to the application and avoid network overhead. Flash devices that use a particular device technology (e.g., SATA) achieve roughly similar performance. Thus, a consumer (read intensive) MLC drive likely achieves similar performance as an enterprise (write intensive) MLC drive.
- Write performance is important too and pretty much all shared, durable storage architectures will need to go to the network, as well as the local cache (flash), to store new data. For DVX, fast NVRAM acknowledgement of durable data on the NetShelf precedes caching into local flash. In general, write performance of the local flash does not introduce an impedance on the overall write operations.
- Write-optimized drives use more/faster DRAM/SLC cache to improve write performance. They also use high flash over-provisioning to guard against write amplification. Faster components and over-provisioning is expensive and hence increases the drive cost. Flash investment is better directed at more capacity or more affordable drives.
- Think of host flash resources much like you would server memory – more is better. Typically, if you have high performance servers with a lot of compute and RAM then design these hosts with more flash than hosts with less compute and memory.
- With current market availability, drives in the 1-2 TB size seem to be the sweet spot and very cost effective – especially when you consider there is no RAID overhead required on the hosts and DVX hyperdriver delivers inline data reduction with compression and deduplication.
- The Datrium DVX can support on each host from 1-8 SSD devices and up to 8 TB of total flash capacity.
- Flash devices only support a limited number of writes. Drive endurance is rated using workloads such as JEDEC that emphasize small writes. Small writes are the norm when data is written directly from typical operating systems such as Microsoft Windows or Linux. Small writes are particularly problematic because they lead to high write amplification.
- Datrium uses a log structured storage to eliminate write amplification; small writes are gathered up and written as a large write, and it does not overwrite in place, it writes to new space.
- The DVX also uses compression and deduplication to further reduce the write amplification to a fraction of the actual data.
- If a flash device fails, the durable data is still on the NetShelf ready to be cached again on a working flash component – eliminating any need for flash device recovery. Simply replace the device.
Type / Connection
- “Consumer” vs. “Enterprise” SSDs: These terms are not always used consistently. Branded server vendors (e.g Cisco, Dell, HP) will select Enterprise drives as supportable server options, and they work fine for DVX.
- The inter-connect type (e.g., 6 Gbps SATA, 12 Gbps SAS or 4/8 lane PCI-e) fundamentally improves the speed with which data is delivered from the flash to the host. In general, faster interconnect improves performance.
- SATA drives are perfectly adequate for a vast majority of the use cases. Using 2 devices per host improves performance and availability more than a single device with a faster interconnect.
- Devices may be connected directly to the system bus (PCI-e), internal storage ports (AHCI) or more sophisticated controllers (HBA/RAID). Conventional storage systems need to build a software RAID solution to aggregate storage from multiple devices. However, the Datrium software architecture does not impose such an overhead and host level RAID is not required.
- Using 2 drives is better than 1 – it is preferable to use multiple smaller drives to achieve the target cache size. For example, two 1 TB drives costly roughly the same as a single 2 TB drive (of the same device family). Yet, two drives offer twice the device interconnect bandwidth as a single drive. They also provide added resiliency against failed drives.
- The best RAID controller is no RAID controller. Minimizing the number of IO subsystems between the flash and the host reduces the potential for unexpected behavior. When possible, directly attach SSD to the system board instead of routing via RAID controllers.
- Use at least 2 flash devices for better performance and availability considerations.
- SATA/SAS interconnect are adequate for the vast majority of use cases.
- “Read intensive” class of devices work well with the DVX hyperdriver.
- In general, good choices for drives are:
- supported by VMware’s hardware compatibility list (HCL)
- pre-installed in new servers – or available from server vendors
- Rely on DVX for reliability – the data is always safe on the NetShelf, the software uses checksums, referential integrity checks and large writes to achieve high endurance and performance.
- Use more flash; buy the best flash that is practical and economical. Here are some examples from suppliers that work with many major server platforms:
** prices are indicative of a quick internet search – actual prices may vary