Blog: Many Task Computing [MTC]: Bridging the Performance-Throughput Gap

Many Task Computing: Bridging the Performance-Throughput Gap
International Science Grid This Week (01/28/09) Raicu, Ioan; Foster, Ian; Zhao, Yong

Researchers from the University of Chicago, Argonne National Laboratory, and Microsoft have conceived of Many Task Computing (MTC), a methodology designed to tackle the kinds of applications not easily supported by clustered high-performance computing or high-throughput computing (HTC). MTC "involves applications with tasks that may be small or large, single or multiprocessor, compute-intensive or data-intensive," the researchers write. "The set of tasks may be static or dynamic, homogeneous or heterogeneous, and loosely- or tightly-coupled." MTC's distinction from HTC lies in the timescale of task completion and the fact that the nature of the applications is frequently data-intensive. Many resources are utilized over short intervals to perform many computational jobs, both dependent and independent. Loosely-coupled applications involved in MTC are communication-intensive but are not naturally represented through the use of the standard message-passing interface. Applications that run on or generate large data volumes cannot scale without sophisticated data management, which makes them organically complementary to MTC, the researchers say. They conclude that MTC's impact on science will be profound, noting that "we have demonstrated good support for MTC on a variety of resources from clusters, grids, and supercomputers through our work on Swift, a highly scalable scripting language/engine to manage procedures composed of many loosely-coupled components, and Falkon, a novel job management system designed to handle data-intensive applications with up to billions of jobs."

View Full Article