Release Note of 1.6.0.pre1

• Release date: January 3 2015.
• The source tar balls can be obtained from the ​Download Page

Format conformation updates:

• Conform with netCDF4 on CDF-1 and CDF-2 formats. The only difference now between the two formats are the OFFSET item in the format spec (32 vs. 64 bit integers.) All names (variable, dimension, attribute) now allow extended characters (eg. special2 and MUTF8).
• Refer to ​CDF-5 format specification for the differences among CDF-1, 2, and 5.

New APIs

• Nonblocking buffered varn API family.
  • For C, ncmpi_bput_varn_int() etc.
  • For F77, nfmpi_bput_varn_int() etc.
  • For F90, nf90mpi_bput_varn() overloading
  • For C++, NcmpiVar::bputVarn() overloading
• Nonblocking varn API family.
  • For C, ncmpi_iput_varn_int() ncmpi_iget_varn_int() etc.
  • For F77, nfmpi_iput_varn_int() nfmpi_iget_varn_int() etc.
  • For F90, nf90mpi_iput_varn() nf90mpi_iget_varn() overloading
  • For C++, NcmpiVar::iputVarn() NcmpiVar::igetVarn() overloading
• Blocking vard API family
  • takes an argument of MPI derived data type that describes the file access layout, as opposed to vara and vars APIs that use start[] and count[].
  • For C, ncmpi_put_vard() ncmpi_get_vard()
  • For F77, nfmpi_put_vard() nfmpi_get_vard()
  • For F90, nf90mpi_put_vard() nf90mpi_get_vard()
  • For C++, NcmpiVar::putVard() NcmpiVar::getVard()
• Collective var1 API family
  • For C, ncmpi_put_var1_all() ncmpi_get_var1_all()
  • For F77, nfmpi_put_var1_all() nfmpi_get_var1_all()
  • For F90, nf90mpi_put_var() overloading
  • For C++, NcmpiVar::putVar_all() overloading
• ncmpi_inq_buffer_size()
  • returns the size of buffer previously attached for use of bput APIs. With ncmpi_inq_buffer_usage(), one can calculate the space remaining for additional bput requests.
• ncmpi_inq_recsize()
  • returns the size of record block, i.e. the sum of single records of all record variables.
  • For F77, nfmpi_inq_recsize()
  • For F90, nf90mpi_inq_recsize()
  • For C++, NcmpiGroup::getRecSize()
• ncmpi_inq_num_rec_vars() and ncmpi_inq_num_fix_vars()
  • report the number of record and fixed-size variables, respectively.
  • For F77, nfmpi_inq_num_rec_vars() and nfmpi_inq_num_fix_vars()
  • For F90, nf90mpi_inq_num_rec_vars() and nf90mpi_inq_num_fix_vars()
  • For C++, NcmpiGroup::getRecVarCount() and NcmpiGroup::getFixVarCount()

New PnetCDF hint

• pnetcdf_subfiling -- it can be set in an MPI info object or in the environment variable PNETCDF_HINTS to enable/disable subfiling. The value is either "enable" or "disable".

PnetCDF hint priority

• The alignment hints set in the environment variable "PNETCDF_HINTS" have the highest priority, which overwrite the alignment hints set in ncmpi__enddef(), which overwrite the alignment hints set in the MPI_Info object used in the call of ncmpi_create() and ncmpi_open().

New error code

• NC_ESTRICTCDF2 for attempting CDF-5 operation on CDF-2 file. For example, define a variable of type NC_INT64 in a CDF-2 file.
• NC_ETYPESIZE when filetype size is bigger than the variable size
• NC_ETYPE_MISMATCH when the element type of filetype mismatches the variable type
• NC_ETYPESIZE_MISMATCH when filetype size mismatches buffer type size
• NC_ENULLSTART when argument start is a NULL pointer
• NC_ENULLCOUNT when argument count is a NULL pointer
• NC_EINVAL_CMODE when invalid file create mode is set, (e.g. cannot have both NC_64BIT_OFFSET & NC_64BIT_DATA. In PnetCDF 1.5.0 and earlier versions, if both flags were set, then NC_64BIT_DATA triumphs NC_64BIT_OFFSET.)

New example programs

• C/bput_varn_uint.c and F77/bput_varn_int8.f show the use of nonblocking bput_varn APIs
• C/i_varn_int64.c and F77/i_varn_real.f show the use of nonblocking iput_varn and iget_varn APIs
• C/vard_int.c, F77/vard_int.f, F90/vard_int.f90, and CXX/vard_int.cpp show the use of vard API to write/read record and fixed-size variables.
• C/transpose.c shows the use of ncmpi_put_vara_int_all to write a 3D array that is dimensional-transposed from the one stored in memory. Six transposed 3D arrays are saved whose dimensions are organized as ZYX, ZXY, YZX, YXZ, XZY, and XYZ. The C++, Fortran 77, and Fortran 90 versions are also provided.

New test program

• nonblocking/wait_after_indep.c tests if ncmpi_end_indep_data() returns properly when nonblocking APIs are called in independent data mode and the wait call is made after exiting the independent mode.
• nonblocking/flexible_bput.c tests flexible bput APIs that use noncontiguous buffer type, noncontiguous imap and require type conversion
• testcases/flexible2.c tests flexible APIs that perform type conversion
• testcases/flexible_varm.c tests flexible varm APIs
• testcases/varn_contig.c tests the case when the fileview is actually a contiguous chunk. PnetCDF should be able to merge all.
• nonblocking/bput_varn_uint.c tests nonblocking bput_varn APIs
• nonblocking/i_varn_int64.c tests nonblocking iput_varn and iget_varn APIs
• test/testcases/test_vard.c test/testcases/test_vardf.f test/testcases/test_vardf90.f90 test the new vard APIs.
• test/testcases/inq_recsize.c tests if one can correctly inquire the size of record block from in a netCDF file. A similar program in F90, named inq_recsizef.f90, is also added.
• In test/nc_test, the test programs below are borrowed from netCDF test programs: t_nc.c tst_misc.c tst_norm.c tst_small.c tst_names.c tst_atts3.c tst_atts.c
• test/testcases/one_record.c tests the special case defined in CDF-1 and CDF-2 specification that "A special case: Where there is exactly one record variable, we drop the requirement that each record be four-byte aligned, so in this case there is no record padding."
• test/testcases/modes.c tests if correct error codes are returned when various file create/open modes are used.
• Under test/testcases, varn_int.c varn_intf.f varn_real.f90 test varn APIs
• test/testcases/inq_num_vars.c test if one can correctly inquire the numbers of record and fixed-size variables defined in a netCDF file. A similar program in F90, named inq_num_varsf.f90, is also added.
• test/nonblocking/interleaved.c tests a combination of interleaved file types. This is to test the bug fix in r1758.

New optimization

• Prior to this release, PnetCDF wraps each MPI read/write call in put/get APIs with two MPI_File_set_view(). One is before the MPI read/write call to take advantage of MPI's fileview feature for accessing non-contiguous file locations. The other is after the MPI read/write call to make the whole file visible, as the root process may write to file header later in the data mode and it alone cannot make a call to MPI_File_set_view() because the function is collective.

In this release, the second MPI_File_set_view() has been removed. The root process's fileview is changed to always keep the whole file header visible. Saving a collective call to MPI_File_set_view() is expected to improve some performance.

Semantics updates

• Header consistency mechanism has been updated. See README.consistency for details.
• The use of NC_SHARE is also revised. See README.consistency for details.

Other updates:

• The subfiling feature is now disabled in ncmpidump and ncmpidiff until a bug fix to allow reading the master file with the number of processes that is smaller than the number of subfiles.
• The attribute names reserved for subfiling feature are now changed to use a prefix of "_PnetCDF_SubFiling". The leading "_" underscore is a CDL convention.
• The flexible APIs now allow argument buftype to be MPI_DATATYPE_NULL. In this case, argument bufcount is ignored and argument buf's data type must match the data type of variable defined in the file - no data conversion will be done. This extension makes the flexible APIs be able to correspond to the netCDF APIs whose names do not contain a data type, e.g. nc_put_vara().
• Type conversion between NC_BYTE and unsigned char no longer checks for out of range error (NC_ERANGE). CDF file format specification make a special case for this kind of data type conversion. See: "Note on byte data" in the format specification.
• Conform with the CDF-2 and CDF-5 formats that names are normalized according to Unicode NFC normalization rules during encoding as UTF-8 for storing in the file header.
• A new configure option, --enable-large-file-test, to enable testing I/O on large files and large variables. Note the testing prgrams will run very slowly.

Bug fixes

• fix the bug for flexible get_varn API. When buftype is noncontiguous, the bug forgot to "unpacks" the temporary buffer that reads data from file to the user buffer using buftype.
• fix the bug in blocking flexible get APIs when buftype is noncontiguous, swap is needed, type conversion is not. The bug sets a NULL pointer to the read buffer and passes it to MPI_File_read functions. See r1815.
• fix the bug for the NetCDF special case when there is only one record variable and the record size is not four-byte aligned. In this case, NetCDF spec says the alignment must dropped (no padding). See r1791.
• fix the bug in nonblocking APIs when requests are resorted into nonoverlapping groups and the first group of file types are interleaved, it was mistakenly identified as non-interleaved. See r1758.
• fix the setting for PNETCDF_RELEASE_DATE in configure.in. Using read command together with IFS does not work as expected in bash 4.3.11. See ​http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2014-July/001586.html for further detailed discussion.