8.3.3 Extremely noisy measurements

Sometimes a coNCePTuaL program runs to completion but the data written to the log files exhibit high levels of variability across runs or even across trials within a single run. A possible source of this variability—especially for long-running programs running on a large number of processors—is the run-time library’s log-file checkpointing mechanism. Because each process in a coNCePTuaL programs writes its own log file, poorly scalable shared filesystems, limited spare network bandwidth, and asynchronous operating-system buffer flushes may each impact program performance in an unpredictable manner and at unpredictable times.

Setting the NCPTL_CHECKPOINT environment variable to ‘0’ disables log-file checkpointing and may thereby reduce some of the data variability. The caveat is that a program that aborts abnormally will leave behind no data in its log files. See Environment Variables, for more information about NCPTL_CHECKPOINT.