``Read in Distributed, Compute in Distributed, Statistics in Distributed, Output in Distributed.''

What Is This?

This web page introduces a simple computing framework for "Big Data" called single program multiple data (SPMD), and many statistical methodology can be fairly easily redesigned in the same way. We aim to introduce ideas in the sense of STATISTICS, and provide Cookbook to illustrate the framework covering from fundamental statistics to advance methodology. Tentatively, the pages will cover basic ideas of parallel computing, statistical computing, and R programming, and they will be illustrated in a simple manner. "Have a Big dream of Bigger than Big."

About Computing Environment

By default, all examples of this website are illustrated in the Unix/Linux system with pbdMPI. pbdMPI is mainly developed and tested under OpenMPI in xubuntu system. Also, all examples are assumed running under the single program multiple data (SPMD) framework.

For Mac users, OpenMPI is suggested for pbdMPI. For MS Windows users, MPICH2 is suggested and working very well with pbdMPI.

If you don't have many machines/processors, the easier way you can test and learn is to install VirtualBox with Unix/Linux system. The VirtualBox allows to generate simultaneously multiple virtual computers in most common systems. You can duplicate the virtual machines/processors inside VirtualBox as many as you want. Therefore, a parallel computing environment can be done in a single machine. Regardless of computing performance, it is helpful for testing programs and for building projects in a consistent environment.

Citation

When mentioning the HPSC, please cite:

@MISC{hpsc2011, author = {Chen, W.-C. and Ostrouchov, G.}, title = {HPSC -- High Performance Statistical Computing for Data Intensive Research}, year = {2011}, note = {URL http://thirteen-01.stat.iastate.edu/snoweye/hpsc/} }

Need Help

• Bugs Report: wccsnow @ gmail.com

Wei-Chen Chen and George Ostrouchov.

Acknowledgment

Wei-Chen thanks Dr. George Ostrouchov of Oak Ridge National Laboratory for helpful discussion, and provide insightful suggestions and materials about general parallel computing. The contents are outcomes part of the project "Visual Data Exploration and Analysis of Ultra-large Climate Data" supported by U.S. DOE Office of Sience.

Wei-Chen also thanks Dr. Hao Yu, the author of Rmpi, for great discussion about Rmpi design and parallel programming in Rmpi. Also, Wei-Chen thanks Stephen Weston, one author of Parallel R Data Analysis in the Distributed World, for sharing MPI and snow information in R.

This website is built on a machine located in the Department of Statistics at Iowa State University in Ames Iowa, USA.