Tools for Parallel I/O and Compute Intensive Applications

V. Messerli

Ph.D. dissertation by Vincent Messerli, under the supervision of Prof. Roger D. Hersch, Thesis No 1915 presented at the Computer Science Department, Ecole Polytechnique Fédérale de Lausanne, July 2001

Most parallel file systems provide applications with conventional sequential views of files that stripe data transparently across multiple disks thus reducing the bottleneck of relatively slow disk access throughput. The Unix-like interface increases the ease of application portability, but conceals the underlying parallelism within the file system precluding any optimization of the I/O access pattern from different processes. Although recent multiprocessor file systems incorporate more sophisticated I/O interfaces, e.g. collective I/O interfaces, enabling the I/O access-pattern information to flow from an application program down to the data management system, they still lack flexibility and they do not enable application-specific codes to run on I/O nodes. Therefore application-specific processing operations cannot be directly executed where data resides. Moreover, with the conventional approach of developing parallel I/O- and compute- intensive applications on distributed memory architectures, programmers are faced with two different systems: a parallel file system for parallel I/O and a message passing system for parallel computation. This separation between storage and processing makes it difficult for application programmers to combine I/O and computation in an efficient manner, i.e. overlap I/O requests with computations.

In this thesis we propose a new approach for developing parallel- I/O and compute- intensive applications on distributed memory PC’s. Using the CAP Computer-Aided Parallelization tool, application programmers create separately the serial program parts and express the parallel behaviour of the program at a high level of abstraction. This high-level parallel program description is preprocessed into a compilable and executable C++ source parallel program. Thanks to a configuration file specifying the layout of the application threads onto the parallel architecture, a same program can run without recompilation on different hardware configurations.

In the context of this thesis we have designed the runtime system of CAP incorporating among others a message-passing system, and, using the CAP tool, a Parallel Storage and Processing Server (PS2) comprising a library of reusable low-level parallel file system components. Thanks to the CAP formalism, these low-level parallel file system components can be combined with processing operations in order to yield efficient pipelined parallel I/O and compute intensive programs. These programs may run on multi-PC servers offering their storage and processing services to clients located over the network.

In this work, we present the CAP computer-aided parallelization tool and its underlying message-passing system along with a performance analysis. We introduce the PS2 framework and discuss its design and implementation using the CAP tool. We describe the new two-dimensional extent-oriented structure of a parallel file and show how the PS2 server can be customized by developers in order to yield efficient parallel I/O and compute intensive applications. Several examples are provided, among them parallel imaging operations. Finally, the applicability and the performance of the CAP tool and the PS2 framework on real applications is demonstrated with a parallel 3D tomographic image server application enabling clients to specify and access parallel slices having any desired position and orientation. This application has been successfully interfaced to a Web server and it is known as the Visible Human Slice WEB server (http://visiblehuman.epfl.ch).

Download the full paper: PDF 29.9 MB