Parallel netCDF Q&A

• Striping count is the number of object storage targets (OST), i.e. the number of file servers storing the files in round robin.
• Striping size is the size of the block. 1 MB is a good size.
• Striping offset is the starting OST index (default -1). In most of the case, OSTs are selected by the system, not configurable by users.

% lfs getstripe filename
stripe_count: 12 stripe_size: 1048576 stripe_offset: -1

% lfs setstripe -s stripe_size -c stripe_count -o start_ost_index directory|filename

1. Use command "lfs" to set the striping count and size for the output directory and create your output files there.
2. Use collective APIs. Collective I/O coordinates application processes and reorganizes their requests into an access pattern that fits better for the underlying file system.
3. Use nonblocking APIs for multiple small requests. Nonblocking APIs aggregate small requests into large ones and hence have a better chance to achieve higher performance.

To use PnetCDF for parallel I/O, users must add the NC_MPIIO (or NC_PNETCDF) flag to the create mode argument when calling nc_create_par. Without this flag, netCDF-4 programs can only perform sequential I/O on the classic CDF-1 and CDF-2 files. Starting from version 4.4.0, netCDF 4 supports the CDF-5 format which allows larger sized variables and 8-byte integers.

Example netCDF-4 programs that use PnetCDF for parallel I/O are available here.

The "data shift penalty" happens when the new file header grows bigger than the space reserved for the original header and forces PnetCDF to "shift" the entire data section to a location with higher file offset. The header of a netCDF file can grow if a program opens an existing file and enters the redefine mode to add more metadata (e.g. new attributes, dimensions, or variables). PnetCDF provides an I/O hint, nc_header_align_size, to allow user to preserve a larger space for file header if it is expected to grow. We refer to the space allocated for the file header as "file header extent".

The default file header extent is of size 512 bytes, if the file system's striping size cannot be obtained from the underneath MPI-IO library. If the file striping size can be obtained (such as on Lustre) and the total size of all defined fix-sized variables is larger than 4 times the file striping size, then PnetCDF aligns the file header extent to the file striping size. Note PnetCDF always sets a header space of size equal to a multiple of nc_header_align_size.

Below is an example code fragment that sets the file header hint to 1MB and pass it to PnetCDF when creating a file.

MPI_Info_create(&info);
MPI_Info_set(info, "nc_header_align_size", "1048576");
ncmpi_create(MPI_COMM_WORLD, "filename.nc", NC_CLOBBER|NC_64BIT_DATA, info, &ncid);

Note all I/O hints in PnetCDF and MPI-IO are advisory. The actual values used by PnetCDF and MPI-IO may be different from the ones set by the user programs. Users are encouraged to print the actual values used by both libraries. See I/O hints for how to print the hint values.

MPI_Info_create(&info);
MPI_Info_set(info, "nc_var_align_size", "1048576");
ncmpi_create(MPI_COMM_WORLD, "filename.nc", NC_CLOBBER|NC_64BIT_DATA, info, &ncid);

To disable the alignment, set the hint value of nc_var_align_size to 1. If you are using nonblocking APIs to write data, we recommend to disable the alignment.

Note all I/O hints in PnetCDF and MPI-IO are advisory. The actual values used by PnetCDF and MPI-IO may be different from the ones set by the user programs. Users are encouraged to print the actual values used by both libraries. See I/O hints for how to print the hint values.

• PNETCDF_HINTS allows users to pass I/O hints to PnetCDF library. Hints include both PnetCDF and MPI-IO hints. The value is a string of hints separated by ";" and each hint is in the form of "keyword=value". E.g. under csh/tcsh environment, use command:

setenv PNETCDF_HINTS "romio_ds_write=disable;nc_header_align_size=1048576"

• PNETCDF_VERBOSE_DEBUG_MODE is used to print the location in the source code where the error code is originated, no matter the error is intended or not. This run-time environment variable only takes effect when PnetCDF is configure with debug mode, i.e. --enable-debug is used at the configure command line. Users are warned that enabling this mode may result in a lot of debugging messages printed in stderr. Set this variable to 1 to enable. Set it to 0 or keep it unset disables this mode. Default is 0, i.e. disabled.
• PNETCDF_SAFE_MODE is used to enable/disable the internal checking for attribute/argument consistency across all processes. Set it to 1 to enable the checking. Default is 0, i.e. disabled.

Note the environment variables precede the (hint) values set in the application program.

Example programs that print PnetCDF hints only can be found in the "examples" directory of the PnetCDF release: hints.c, hints.f, and hints.f90. Below is a code fragment in C that prints all I/O hints, including PnetCDF and MPI-IO.

    err = ncmpi_get_file_info(ncid, &info_used);
    MPI_Info_get_nkeys(info_used, &nkeys);
    for (i=0; i<nkeys; i++) {
        char key[MPI_MAX_INFO_KEY], value[MPI_MAX_INFO_VAL];
        int  valuelen, flag;
        MPI_Info_get_nthkey(info_used, i, key);
        MPI_Info_get_valuelen(info_used, key, &valuelen, &flag);
        MPI_Info_get(info_used, key, valuelen+1, value, &flag);
        printf("I/O hint: key = %21s, value = %s\n", key,value);
    }
    MPI_Info_free(&info_used);
    

int ncmpi_inq_header_size(int ncid, MPI_Offset *hdr_size);
int ncmpi_inq_header_extent(int ncid, MPI_Offset *hdr_extent);


• There are many variables defined in the netCDF file. MPI processes read/write a list of one variables in a sequence. (PnetCDF aggregation can handle requests across variables.)
• MPI processes read/write a sequence of subarrays of the same variable. (PnetCDF aggregation can also handle multiple requests to a single variable.)
• The numbers of read/write requests are different among processes.

• First, tell PnetCDF how much space can be allocated to be used by the APIs.
• Make calls to the buffered put APIs.
• Make calls to the (collective) wait APIs.
• Free the space allocated by the internal buffer.

On the contrary, independent APIs (also referred as non-collective) impose no such requirement.

All PnetCDF collective APIs (except create, open, and close) have a suffix of "_all", corresponding to their independent counterparts. To switch from collective data mode to independent mode, users must call ncmpi_begin_indep_data. API ncmpi_begin_indep_data is to exit the independent mode.

• Use nonblocking APIs. Individual processes can make any number of calls to nonblocking APIs independently from other processes. At the end, a collective wait API, ncmpi_wait_all, is recommended to used to allow all nonblocking requests to commit to the file system.
• Force all the processes participate the collective calls. When a process has nothing to request, users can still call a collective API with zero-length request. This is achieved by setting the contents of argument count to zero.

    ncmpi_get_varn_<type>_all
    ncmpi_get_varn_<type>
    ncmpi_put_varn_<type>_all
    ncmpi_put_varn_<type>
    

    err = ncmpi_create(comm, path, cmode, info, &ncid);
    if (err != NC_NOERR) {
        int rank;
        MPI_Comm_rank(comm, &rank);
        printf("Error at rank %d: %s\n", rank, ncmpi_strerror(err));
    }
    

API ncmpi_sync does nothing but simply calls MPI_File_sync. It is expected to have a very high cost when files are stored on a parallel file system, as MPI_File_sync (internally most likely calls the POSIX sync function) is designed to ensure the data is safely stored on the persistent storage hardware, before the function returns. POSIX sync usually incurs a huge performance penalty. This API is to be used for extremely cautious behavior.

The term "sync" should not be confused with "flush". The PnetCDF API ncmpi_flush is also available, which is used to flush data cached in memory to the file system. This API does not call MPI_File_sync internally. POSIX sync guarantees the write data is safely stored on the persistent storage devices, such as hard disks, before the function call returns to users. Safe means even if file servers crashed right after the function returns, data is safely stored on the disks. On parallel file systems, calling MPI_File_sync will result in all MPI processes calling sync, which causes all file servers to carry out the sync operations at the same time.

When using a POSIX compliant file system, such as Lustre and GPFS, applications usually need not call ncmpi_sync before closing the file.

Starting from version 1.6.1, PnetCDF supports fill mode in a slightly different way from netCDF. The API ncmpi_set_fill() sets the fill mode for all the non-record variables defined in the file and can only be called in the define mode. For record variables, users are required to call ncmpi_fill_var() explicitly to fill one record of a variable at a time.

Similar to nc_def_var_fill() in netCDF4, API ncmpi_def_var_fill() can be used to set the fill mode for individual variables.

     int ncmpi_inq_get_size(int ncid, MPI_Offset *size);
     int ncmpi_inq_put_size(int ncid, MPI_Offset *size);
    

      External type  No. Bits  Intent of use
      -------------  --------  ---------------------------------
      NC_CHAR         8        text data (the only non-numerical type in NetCDF)
      NC_BYTE         8        1-byte integer
      NC_SHORT       16        2-byte signed integer
      NC_INT         32        4-byte signed integer
      NC_FLOAT       32        4-byte floating point number
      NC_DOUBLE      64        8-byte real number in double precision
      NC_UBYTE        8        unsigned 1-byte integer
      NC_USHORT      16        unsigned 2-byte integer
      NC_UINT        32        unsigned 4-byte integer
      NC_INT64       64        signed 8-byte integer
      NC_UINT64      64        unsigned 8-byte integer

      Data type of                                  Corresponding
      internal I/O buffer   Example API             MPI datatype
      -------------         ----------------------  -----------------
      text                  ncmpi_put_var_text      MPI_CHAR
      schar                 ncmpi_put_var_schar     MPI_SIGNED_CHAR
      uchar                 ncmpi_put_var_uchar     MPI_UNSIGNED_CHAR
      short                 ncmpi_put_var_short     MPI_SHORT
      ushort                ncmpi_put_var_ushort    MPI_UNSIGNED_SHORT
      int                   ncmpi_put_var_int       MPI_INT
      uint                  ncmpi_put_var_uint      MPI_UNSIGNED
      long                  ncmpi_put_var_long      MPI_LONG
      float                 ncmpi_put_var_float     MPI_FLOAT
      double                ncmpi_put_var_double    MPI_DOUBLE
      longlong              ncmpi_put_var_longlong  MPI_LONG_LONG_INT
      ulonglong             ncmpi_put_var_ulonglong MPI_UNSIGNED_LONG_LONG

% pnetcdf_version
PnetCDF Version:    	1.7.0
PnetCDF Release date:	03 Mar 2016
PnetCDF configure: 	--prefix=/usr/local/PnetCDF --with-mpi=/usr/local
MPICC:  /usr/local/bin/mpicc -g -O2
MPICXX: /usr/local/bin/mpicxx -g -O2
MPIF77: /usr/local/bin/mpif77 -g -O2
MPIF90: /usr/local/bin/mpif90 -g -O2
    

% ident libpnetcdf.a
src/lib/libpnetcdf.a:
     $Id: @(#) PnetCDF library version 1.7.0 of 03 Mar 2016 $

% strings libpnetcdf.a |grep "PnetCDF library version"
PnetCDF library version 1.7.0 of 03 Mar 2016
$Id: @(#) PnetCDF library version 1.7.0 of 03 Mar 2016 $