Redshift’s design is the harbinger of how future storage-related products are going to be designed in the cloud. The new design was announced at re:Invent 2019.
The design is not cool just for technology’s sake; it solves a real business problem. In the real world, the common conundrum is: “I have lots of data, and a lot of historical data as well, and I want to keep all of it because my data has insights that benefit my business. So, I want cheap and deep storage, but yet I want high performance when I run queries on that data.”
Cheap-n-Deep and High Performance are like water and oil. In the cloud world, there is a way to solve this problem elegantly. Let’s see how.
The 2-Layer Filesystem Design
Andy Jassy introduced the new 2-Layer Redshift design in his keynote. The idea is simple: Store lots of data in S3 for cost efficiency and durability and launch on-demand compute nodes with solid-state drive (SSD) caching for high performance. Basically, scale storage and performance independently.
AWS S3 has many interesting properties: Cost-efficient, highly durable, Exabyte scale, high bandwidth, and simple to use. However, there are downsides too: long latencies and poor random I/O performance. EBS has the opposite properties and ends up being 10x more expensive for a high performance and durable solution.
The key design element is to couple S3 with on-demand compute. By leveraging the elastic scaling properties of the cloud, the on-demand compute with SSD caching solves the performance problem when needed. The compute nodes can be auto-scaled up and down as the business needs change. The 2-Layer design can potentially provide 90% cost savings, which makes it a very compelling business technology.
Another new product announcement at re:Invent, AWS UltraWarm, also follows the same design principles.
2-Layer Design Adoption is Increasing
Others have also recently adopted the 2-layer storage system design in the cloud before Redshift. However, the reach of AWS will make this new design much more widely known, and it will provide a blueprint for future cloud-based storage system designs. Here are a couple of other recent examples.
Snowflake is a data warehouse in the cloud. It uses S3 as the backing store, with elastic on-demand compute clusters for running queries.
Cloud Backup and DR SaaS services with VMware Cloud on AWS. VM and container backups are deduped and stored in S3, with elastic on-demand compute that instantly turns backups into live VMs.
What about HCI and AFA?
Traditional Hyperconverged infrastructure (HCI) designs will be challenged to adapt to the public cloud. The core HCI design principle was to converge compute and storage into single nodes – basically a 1-Layer design. This model is extremely inefficient in the cloud because the HCI design requires lots of compute nodes, and these nodes must always be powered on. The cost is further increased by having replicas to protect against compute node failures.
All-Flash-Arrays (AFAs) have a different design problem in the cloud. All the algorithms in AFAs are designed around SSDs. So, you need both expensive compute and SSDs to be always powered on, which makes it extremely expensive. Additionally, most AFAs are not designed as scale-out systems, and that limits their scope in the cloud.
The Future: S3 + On-Demand Compute
AWS S3 provides a strong, durable, and cost-efficient foundation for storing lots of data. On top of that, the cloud provides an opportunity to provision elastic on-demand performance when needed with elastic compute nodes and SSD caches. With a potential 90% cost savings, this 2-Layer design is here to stay and propel other new data solution ideas.
Want to learn more? Contact us.