JumboMem provides a low-effort solution to the problem of running memory-hungry programs on memory-starved computers. The JumboMem software gives programs access to memory spread across multiple computers, providing the illusion that all of the memory resides within a single computer. When a program exceeds the memory in one computer, it automatically spills over into the memory of the next computer.
Consider running a program that needs 10 GB of memory to run effectively. If you run that program on a computer that has only 4 GB of RAM (Figure 1), the program will will spend most of its time paging to disk, which is tens of thousands of times slower than RAM. With JumboMem, that same program can be spread automatically across multiple computers that are networked together, as shown in Figure 2. In this case, the parts of the program that don't fit in local RAM are sent over a high-speed network to remote RAM, thereby bypassing the slow disk altogether.
Behind the scenes, JumboMem handles all of the network communication required to make this work; the user's program does not need to be modified—not even recompiled—to take advantage of JumboMem. Furthermore, JumboMem does not need administrator privileges to install. Any ordinary user with an account on a workstation cluster has sufficient privileges to install and run JumboMem.
The jumbomem manual page explains how to run a large-memory program across multiple machines using JumboMem.
The following conference paper describes JumboMem's implementation and presents some performance measurements taken on a 256-node cluster with 1 TB of RAM (PDF, 525 KB):
Scott Pakin, Greg Johnson. Performance Analysis of a User-level Memory Server. In Proceedings of the 2007 IEEE International Conference on Cluster Computing (Cluster 2007), Austin, Texas, pp. 249–258, September 2007.
If you don't want to read the entire paper, here's a link to the corresponding presentation (PDF, 1.2 MB):
If you don't want to read the entire presentation, here's a poster displayed at LANL's booth at SC2007 (PDF, 11 MB):
JumboMem is open-source software released under the GNU
General Public License, version 2 with an additional
modifications must be clearly marked
clause.
For more information see the JumboMem project page on SourceForge.net.
Scott Pakin, pakin@lanl.gov