Corrupted mesh and solution files when using MPI and NFS (Network File System)

[TRPrasanna]

Has anyone experienced file corruption of the mesh and solution files when using more than 1 computing node on a computing cluster with a network file system (NFS) storage? This seems to happen to me when using approximately 6 or more computing nodes. The corrupted file (either the mesh file, or solution file, or both) invokes the error "Array found in file dimensions does not match that of the introduced variable. File dimensions:" in subroutine getSolutionFileArrayDimensions in Solver/src/libs/io/SolutionFile.f90. The bug seems to be very similar to what I faced on a different code some years ago. In short, the corrupted files have some zeros in them instead of having data, prompting the program (horses3d.ns while attempting to resume the simulation from the file or horses2plt when attempting to convert it to a post file) to crash at the aforementioned subroutine (on second thoughts, resuming from the corrupted file to different polynomial order may also result in "Error reading restart file: wrong dimension for element xx", in addition to above error). Is anyone facing this bug?

[loganoz]

I asked the horses developers at UPM and we don't have experience working with horses and NFS storage

[TRPrasanna]

May I know what file system do you use? If not NFS, it is likely a parallel file system (OrangeFS/PVFS, Lustre, GPFS etc).

[loganoz]

We typically run in MN5 (https://www.bsc.es/marenostrum/marenostrum/technical-information), Magerit-3 (https://www.cesvima.upm.es/infrastructure/magerit). I am not sure about the details of the file system used. May be is detailed in the webpages?

[TRPrasanna]

Thank you. Based on the links that you sent me, it seems MareNostrum4 and Magerit-3 use GPFS and BeeGFS, respectively. Both of these are parallel file systems. Unfortunately, the HPC clusters (Elgato, Ocelote, Puma) in my host university, the University of Arizona, only have NFS, which is not suitable for simultaneous writes to the same file when using MPI processes. There are no plans to upgrade to a parallel file system in the near-future either. However, recently, I have found tremendous success using MPI-IO collective write calls, which to my knowledge, coalesces multiple small write calls into pieces of larger write calls. This is working great so far on NFS, with reasonable performance penalty, on openmpi version 3 and 4 (with MCA parameter fs_ufs_lock_algorithm of 1) and should also work with MPICH (that uses ROMIO).

I have uploaded a draft version of the code on my fork here (on the mpiIOcollective branch). If I were to refine this code such that it works with MPI-IO when say, the code is compiled with WITH_MPIIO=YES, would you consider merging it with the main repository? At the moment my additional code is not feature complete (it writes the mesh and the solution fine, but without the additional gradients/sensors/LES data) but I am working on it.

Edit: MareNostrum5 also uses GPFS as noted here.

[loganoz]

Yes sure, we can consider merging with the main branch to improve horses functionality. Apart from the code itself, please update the documentation (now in web format) so anyone can use the new writing system.
When you have something ready, please let us know so we can take a look.
Thanks!