HDF stands for ``Hierarchical Data Format''. It is a library of functions that let you structure your data files. The library provides functions for writing and reading HDF files and there are also stand-alone utilities for viewing them, analyzing them, for converting GIF images to HDF5 images and for compilation and linking of programs that use HDF.
The version of HDF we are going to work with is HDF5. Although conceptually very similar to HDF4, HDF5 is quite different. HDF5 functions have different names, the file format is different, the data model is different, and, most importantly for us, HDF5 binds to MPI-IO so that you can write HDF5 files from MPI programs in parallel.
At the time of this writing HDF5 is not installed on the AVIDD cluster officially yet. I have it currently in my home directory and you will have to put
/N/B/gustav/binin front of your command search
/N/B/gustav/libin front of your library search path, in order to run HDF5 examples. The reason for this is that there are still some problems with parallel HDF5 files and I want to have these resolved with NCSA folks before I tell AVIDD administrators that HDF5 is ready to install. But we are going to work with sequential HDF5 initially, so these problems should not affect us. And by the time we get to the parallel IO part, they may well be fixed.
HDF5 is a relatively new product. I've been told that its parallel component has not been tested on IA32 yet (so we may as well be the first to use it). But conceptually HDF5 goes back to 1998, when HDF4 developers got a grant to develop HDF further in order to address shortcomings of HDF4 and to make HDF work with MPI-IO. Since its early release in 1998, HDF5 has been used in numerous software packages such as Cactus , Chombo , Flash , Swarm , Vis5D , Intel Array Visualizer , LabVIEW , and some others. HDF5 is also used by some 90 organizations or so, which is a lot for a highly specialized software package of this type.
HDF development was paid for by the Department of Defense, the Department of Energy, NASA, Hughes Information Technology Corporation, Microsoft, NSF, Phillips Petroleum, the State of Illinois, Boeing, NCSA, and other partners. It was developed in Urbana-Champaign by researchers associated with NCSA. The current head of the project is Michael J. Folk, firstname.lastname@example.org, but if you have any queries regarding HDF5 you should send mail to HDF User Support at NCSA, email@example.com, since HDF problems are now handled by the whole army of programmers. Naturally, within this course, I590, you should probably contact me first, and if I can't answer your question, we will then go to NCSA.
NCSA provides quite nice on-line documentation that covers HDF5. The central point of it is http://hdf.ncsa.uiuc.edu/HDF5/, from where you can go to