JumboMem
Feed your memory-hungry programs

Description

JumboMem provides a low-effort solution to the problem of running memory-hungry programs on memory-starved computers. The JumboMem software gives programs access to memory spread across multiple computers, providing the illusion that all of the memory resides within a single computer. When a program exceeds the memory in one computer, it automatically spills over into the memory of the next computer.

Running a program that pages to disk
Figure 1: Running a 10 GB program without JumboMem

Consider running a program that needs 10 GB of memory to run effectively. If you run that program on a computer that has only 4 GB of RAM (Figure 1), the program will will spend most of its time paging to disk, which is tens of thousands of times slower than RAM. With JumboMem, that same program can be spread automatically across multiple computers that are networked together, as shown in Figure 2. In this case, the parts of the program that don't fit in local RAM are sent over a high-speed network to remote RAM, thereby bypassing the slow disk altogether.

Running a program across multiple computers
Figure 2: Using JumboMem to run a 10 GB program across multiple computers

Behind the scenes, JumboMem handles all of the network communication required to make this work; the user's program does not need to be modified—not even recompiled—to take advantage of JumboMem. Furthermore, JumboMem does not need administrator privileges to install. Any ordinary user with an account on a workstation cluster has sufficient privileges to install and run JumboMem.

Documentation

The jumbomem manual page explains how to run a large-memory program across multiple machines using JumboMem.

The following conference paper describes JumboMem's implementation and presents some performance measurements taken on a 256-node cluster with 1 TB of RAM (PDF, 525 KB):

Scott Pakin, Greg Johnson. Performance Analysis of a User-level Memory Server. In Proceedings of the 2007 IEEE International Conference on Cluster Computing (Cluster 2007), Austin, Texas, pp. 249–258, September 2007.

If you don't want to read the entire paper, here's a link to the corresponding presentation (PDF, 1.2 MB):

Performance Analysis of a User-level Memory Server (slides)

If you don't want to read the entire presentation, here's a poster displayed at LANL's booth at SC2007 (PDF, 11 MB):

JumboMem (poster)

Download

JumboMem is open-source software released under the GNU General Public License, version 2 with an additional modifications must be clearly marked clause.

Download JumboMem

More information

For more information see the JumboMem project page on SourceForge.net.

Scott Pakin, pakin@lanl.gov