PNETCDF_HINTS

Starting from release of 1.4.0 version, this environment variable allows users to pass I/O hints to PnetCDF library. Hints can be a combination of both PnetCDF and MPI-IO hints.

The value is a string of hints separated by ";" and each hint is in the form of "hint=value". E.g. under csh/tcsh environment, use command:

setenv PNETCDF_HINTS "romio_ds_write=disable;nc_header_align_size=1048576"

Under bash environment, use command:

export PNETCDF_HINTS="romio_ds_write=disable;nc_header_align_size=1048576"

Note the environment variables precede the (hint) values set in the application program, i.e. through MPI info object passed to See ncmpi_create and See ncmpi_open. Hints nc_header_align_size, nc_var_align_size, and nc_record_align_size precede the arguments v_align and r_align passed to See ncmpi__enddef.

Available PnetCDF hints

nc_header_align_size

(Before version 1.4.0) It aligns the size of header extent of a newly created file. This hint was first introduced in version 1.1.0, but can only be passed to PnetCDF through an MPI info object, rather than setting the environment variable PNETCDF_HINTS.

(Between version 1.4.0 and 1.13.0) It aligns the size of header extent of a newly created file. The default is 512 bytes. If the file striping size is known and the total size of all fixed-size variables is larger than 4 times the file striping size, then this hint will be set to the file striping size, i.e. the size of header extent will be the smallest multiple of striping size that is larger than the file header size. Set the value to 1 to disable the alignment.

(Version 1.13.0 and later) Starting from 1.13.0, the file striping size is no longer used to set the file header extent. This is because it can result in a large header extent than wanted, especially when the file striping size is large, e.g. 16 MB, and the header size is much smaller, e.g. less than 1 MB.

nc_var_align_size

(Before version 1.4.0) It aligns the size of header extent of a newly created file. This hint was first introduced in version 1.1.0, but can only be passed to PnetCDF through an MPI info object, rather than setting the environment variable PNETCDF_HINTS.

(Between version 1.4.0 and 1.13.0) This hint aligns the starting file offsets of individual fixed-size variables. In PnetCDF version 1.10.0 and prior, the default is 512 bytes if file striping size is not obtainable by PnetCDF. If the file striping size is known and the total size of all fixed-size variables is larger than 4 times the file striping size, then the starting file offsets of all individual fixed-size variables are aligned to the file striping size, i.e. the smallest multiple of stripe size that is larger than the end offset of the variable defined immediately prior to this variable. Set the value to 1 to disable the alignment. Starting from version 1.11.0, the default has changed to 1, i.e. no alignment. If applications use PnetCDF nonblocking APIs to aggregate write requests to multiple variables, then the best practice is to disable the alignment. Disabling this alignment will prevent creating gaps in file space between two consecutive variables and thus the writes to file system can be contiguous.

(Version 1.13.0 and later) Starting from version 1.13.0, this hint will only align the starting offset of the entire data section, rather than individual fixed-size variables.

nc_header_read_chunk_size

(version 1.4.0 and later) PnetCDF reads the file headers in chunks. This hint indicates the chunk size (in bytes). The default is 256 KB.

nc_record_align_size

(version 1.5.0 and later) This hint aligns the starting file offset of the record variable section. Note this is for the entire section, not individual records.

pnetcdf_subfiling

(version 1.6.0 and later) This hint enables or disables the subfiling feature. Its string value is either "enable" or "disable".

nc_in_place_swap

(version 1.10.0 and later) This hint enables or disables in-place byte swap on Little Endian architectures. When the hint is set to "enable", PnetCDF performs byte swap on user I/O buffers whenever possible. This option results in the least amount of internal memory usage. However, if an immutable user buffer is used, segmentation fault may occur when byte swap is performed on user buffer in place. When the hint is set to "disabled", PnetCDF allocates an internal buffer for byte swap without altering the contents of user buffer, when byte swap is required. This option will increase internal memory footprint. When the hint is set to "auto", PnetCDF allocates an internal buffer for byte swap only when the I/O request size is less than 4 KiB. For larger requests, user buffers are used for in-place byte swap. Note this hint takes effect only on Little Endian machines, as no byte swap is necessary on Big Endian machines.

nc_ibuf_size

(version 1.11.0 and later) It sets the size of a temporal buffer to be allocated by PnetCDF internally to pack noncontiguous user write buffers supplied to the nonblocking requests into a contiguous space. Similarly for read case to unpack the temporal buffer to user read buffers if they are noncontiguous. This affects both blocking and nonblocking APIs. On some systems, using noncontiguous user buffers in MPI collective read/write functions performs significantly worse than using contiguous buffers. Note if the size of aggregated user buffers is larger than nc_ibuf_size, packing/unpacking will be disabled to save memory footprint. The default value is 16 MiB.

nc_num_aggrs_per_node

(version 1.14.0 and later) The intra-node aggregation is a new feature introduced in version 1.14.0, which is designed for the I/O patterns that contain many noncontiguous requests interleaved among processes, and spreading across a wide range of file space. It is particularly useful when the number of MPI processes allocated to a compute node is large. Setting this hint to a positive integral value to indicate the desired number of processes per compute node to be selected as the intra-node I/O aggregators. Each process is assigned a unique aggregator. The non-aggregators send their requests to the assigned aggregators, and then the aggregators make MPI-IO requests to the file. Such strategy can effectively reduce communication congestion due to many pending asynchronous messages produced in the collective write inside of MPI-IO. Setting this hint to 0 disables the aggregation, the default mode.