September 17, 2013, Storage Developer Conference, Santa Clara, CA—Andy Rudoff from Intel talked about the impact of the NVM programming model and persistent memory.
Two movements are driving changes into the decades-old basic programming model. Block mode innovations include atomics, access hints, and NVM oriented operations. In addition, the emerging NVM technologies provide much higher performance with a cost structure that is slowly approaching rotating media.
Storage networking industry Association (SNIA) has opened an NVM programming TWG to address these changes. The TWG is chartered to develop specifications for new software programming models which will specify the exact technical behavior up to but not including the OS specific API semantics. They will also identify the critical characteristics of those APIs.
The reason for a programming model versus an API is that the operating system vendors own their own kernel APIs. A committee cannot define one API for multiple operating system platforms, so it is better for innovation and OS vendor buy-in to agree on an overall model and leave the API details to the vendors.
Another reason for forming the working group is the eventual emergence of next-generation NVM technologies which will offer three orders of magnitude speedup over flash. With the next generation nonvolatile memories, the memory is no longer the bottleneck and systems will need an optimized storage platform interconnect and software storage access methods that take advantage of the memory performance.
The next generation technologies will also enable a new storage concept, persistent memory. These memories have many characteristics of RAM--byte addressable, load store access, and possibly DMA-able, similar to a battery backed DRAM. Some key attributes for persistent memory include no unexpected latencies or other inconsistent behaviors. There is a question about persistent memory stores not being durable until data is flushed, and may not always stay the same physical or virtual address.
Some types of persistent memory include battery backed DRAM. Another is an architecture where DRAM is saved on power failure but requires significant additional logic. Nonvolatile memory with significant caching is another possibility that relies on workload assumptions to size cache appropriately. The new generation nonvolatile memories are mostly unknown and the emerging technologies may not be viable for some time.
Persistent memory bridges volatile system memory and SST caching. The system memory uses DRAM DIMMs and offers high throughput and high cost, while the SSD caching is slower but significantly less expensive. Persistent memory use cases include RAID cache, persistent RAM disk, and checkpointing in a DIMM form factor. Another use case is to copy to flash, where sections of DRAM are used for persistent writeback cache which is supported with a UPS and transferred to flash in the event of power loss. And finally, NVM is used as memory in an extended Virtual memory or autocommit memory.
The value of persistent memory is that data sets are addressable with no DRAM footprint. Typically DMA and RDMA works as expected with no buffer copy required for the RDMA. It can also work as a warm cache which is byte addressable and offers direct user mode access.
The challenges of persistent memory are that they require new programming models and APIs which can be built on decades old memory mapped file models. There are some programming challenges which file systems engineers and database engineers have always worked through, but now other apps engineers need to learn how to program these new memory types.
So far, there are two main persistent memory models, NVM volumes and files. The volume model works with a list of physical ranges and performs various operations within those ranges. These uses are most likely in kernel modules, file systems, memory management, and other storage stack components such as raid, cash, and clustered I/O.
The final model works through native file APIs and management to perform its operations. The application memory allocation is the same interface has been used for decades where the memory management is part of the kernel space and the application is in the user space. The memory is gone when the application exits or when machine goes down.
This interface must change for NVM application allocation. Because memory is persistent, apps remain attached to a region of memory. As a result, there is a need to manage permissions for regions and to resize, remove, etc., and back up those data. The applications for NVM.PM.File are persistent data sets requiring addressability without impacting DRAM footprint and persistent caches. Meaning and permissions are uses that must reconnect with blobs of persistence and the kernel modules may require some of those features also.
Memory mapped data structures are usually demand paged and either shared or copy on write (private). The basic persistent memory file model requires APM aware filesystem that is built up with additional libraries which eventually will turn into language extensions. The advent of persistent memory requires resilient data structures.
New and interesting issues associated with persistent memory. A malloc-like interfaces required, but what if the program crashes before anything links to ptr? Memory leak become persistent memory leaks. What if program is half-way through linking it in? Transactions required? How complex? How to test issues that didn’t happen w/volatile memory? Some of these issues will require fault-injection testing, and these issues are not just for malloc, but true for all PM data structures.