As I’ve grown older, I’ve found that I’m not quite as curious as I used to be. Perhaps it’s because I’m busier. Perhaps it’s because I’ve already asked many of the fundamental questions. Perhaps it’s just genetics. I simply don’t know why I don’t ask “why” as much as I once did. Ironic, right?
I’ve seen it with my three daughters as well. There was a period in their lives, peaking around the age of three, where it seemed that half of their conversation consisted of one word — why. I knew it was a great question even though it was tough to hear for the zillionth time in one day. (Hey, who knew zillionth was actually a word?) I knew some of the “why” questions were attention seeking, but I also knew they were, at least in part, seeking to understand cause and effect. New research indicates that children may be able to understand causation at a much earlier age than previously believed. The research shows that children can differentiate cause from effect as early as three years old. Perhaps my children were seeking a real answer all that time.
What does that have to do with Micron? Well, we announced that we are contributing a heterogeneous-memory storage engine, we call it HSE, to the open source community. This software provides a database storage engine that can accelerate the performance of various storage applications, like databases and object stores. You can learn more about the software at our Micron site or at GitHub.
This just might have you asking “why?” Frankly, that’s a fair question. For over 40 years, Micron has been a leader in innovation. We’ve introduced breakthroughs in DRAM, NAND, SSDs, GDDR, NVDIMM and 3D XPoint technology — quite a resume in the world of storage and memory! You may be thinking that’s a great hardware background, but what about software? So that leaves us with a few “why” questions of our own.
Why Micron?
Let’s start at a high level. Micron is the only company in the world that develops and produces DRAM, 3D XPoint technology and SSDs. That’s a comprehensive stack of storage and memory technologies. As we build this portfolio, we are strategically moving up the stack from silicon to solutions. We practice continuous innovation to make it easy for our customers to get the best performance from the systems we sell, which is why we developed, and have now contributed, HSE to the open source community. We know that software runs on our hardware and can also be used to optimize for it. If you look at some of our other work in software, you’ll see we’ve recently built security solutions with Authenta and artificial intelligence (AI) solutions with FWDNXT’s AI software technology. This vertical integration allows us to build solutions by adding software to our systems expertise.
Why HSE?
Micron has a portfolio and a bright future built on DRAM, storage-class memory (SCM) and flash-based storage usually delivered as SSDs. The latter of these two technologies, SCM and SSDs, offers capabilities that were not being fully optimized in other storage engines. Specifically, we found that throughput, latency and SSD endurance could all be improved with a better storage engine — all of which are critically important to our data center customers.
When compared to RocksDB — one of the most successful and widely deployed storage engines in the market today — in various YCSB workloads, HSE delivers up to six times the performance, 11 times lower latency and seven times greater write endurance running on SCM and SSDs. Additional, detailed performance data can be found in the GitHub site for this project.
Why MongoDB?
In case you aren’t aware, not only have we built a storage engine, but we’ve also provided an interface to connect it to MongoDB. We used this as a proof point for the storage engine technology. MongoDB is the most popular NoSQL database in the world, according to DB-Engines, and the fifth most popular database overall as of April 21. Why did we build for MongoDB? Because it’s popular, provides an open API for development and, in our opinion, can be better optimized for SCM and SSDs.
Our integration allowed us to compare our storage engine to its native storage engine. The results were surprising. We saw performance improvements ranging from two to six times. MongoDB has a great native storage engine with WiredTiger. We simply worked to optimize ours for SCM and SSDs. We have more performance testing details and the MongoDB connector available on our GitHub site.
HSE can support many other storage applications. We built an example and look forward to seeing what other use cases our colleagues in the open source community will find and develop for.
Why Linux?
In fairness, you might not have asked this “why.” When you consider that we’re focused on improving application performance in the data center, it seems obvious why we would write this for Linux. When I was at Cumulus Networks, company co-founder JR Rivers often said, “Linux is the lingua franca of the data center.” We have built this for Linux workloads to accelerate and optimize them since that is the preferred operating system of data centers across the globe.
The Right Question
They say the right questions lead to the best answers. We started with a simple question. Why? Why aren’t most storage engines optimized for storage-class memory and flash-based technologies? Our solution? HSE.
