Unable to import various VTK and VTU file types

[2pt0]

vtkio is unable to parse VTK and VTU files generated by pygmsh which internally uses meshio for generation of VTK and VTU files.

I have tested the following file types:

• VTK ASCII (Legacy)
• VTK XML Binary
• VTU ASCII
• VTU XML Binary - No compression
• VTU XML Binary - LZMA compression
• VTU XML Binary - zlib compression

Paraview is able to visualize each file without problem. The spreadsheet view for each file shows the appropriate data.

Here is the script used to generate the VTK and VTU files:

import pygmsh

with pygmsh.geo.Geometry() as geom:
    p = geom.add_polygon(
        [
            [0.0, 0.0],
            [1.0, -0.2],
            [1.1, 1.2],
            [0.1, 0.7],
        ],
        mesh_size=0.4,
    )
    geom.add_physical(p.lines[0], label="bottom")
    geom.add_physical(p.lines[1], label="right")
    geom.add_physical(p.lines[2], label="top")
    geom.add_physical(p.lines[3], label="left")

    mesh = geom.generate_mesh()

mesh.write("no-compression.vtu", compression=None)
mesh.write("lzma.vtu", compression="lzma")
mesh.write("zlib.vtu", compression="zlib")
mesh.write("ascii.vtu", binary=False)
mesh.write("binary.vtk")
mesh.write("ascii.vtk", binary=False)

Here is the file I am using to test the import of these files:

use std::path::Path;
use vtkio::model::Vtk;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let inputs = &[
        "ascii.vtk",
        "binary.vtk",
        "ascii.vtu",
        "no-compression.vtu",
        "lzma.vtu",
        "zlib.vtu",
    ];

    let vtks = inputs
        .iter()
        .map(|input| Vtk::import(input));

    for (input, vtk) in inputs.iter().zip(vtks) {
        print!("{}.. ", input);

        match vtk {
            Ok(_) => println!("parsed!"),
            Err(e) => println!("{}", e),
        };
    }

    Ok(())
}

And the output

ascii.vtk.. Parse error: Alt
binary.vtk.. Parse error: Alt
ascii.vtu.. XML error: Validation error: InvalidDataFormat
no-compression.vtu.. XML error: Validation error: DataArraySizeMismatch { name: "types", expected: 38, actual: 37 }
lzma.vtu.. XML error: Validation error: Base64Decode(InvalidByte(22, 61))
zlib.vtu.. XML error: Validation error: Base64Decode(InvalidByte(22, 61))

I'm not sure if the problem is with vtkio or mehsio. I have opened an issue here since Kitware's Paraview (which also develops VTK) reports no problems with these files.

Thanks for your help!

[elrnv]

Thank you for the issue! It is definitely a goal to support files accepted by VTK or ParaView as much as possible.

[2pt0]

Thanks for the library! Is there a path forward to fix this or any obvious reasons why the parsing is failing?

[elrnv]

I took a quick look at the ascii.vtk file, and noticed there are two additional fields titled "CONNECTIVITY" and "OFFSETS", which aren't documented (https://kitware.github.io/vtk-examples/site/VTKFileFormats/) unfortunately.
I think these use the same format as the xml files use, which means vtkio can already represent this data in a straightforward way, we just need to add some parsing functions for those extra fields.

I haven't yet looked at the other files. It's possible they are failing for different reasons.

I have been adding these undocumented fields/extensions to the vtk formats previously. I think these are inevitable, but I simply don't know what they are until someone finds files which don't work with vtkio, so thank you for bringing it up. I think these can serve as great test examples that we can add to assets and add more tests for these files in particular.

It might be a while until I have time to dig into these myself, but I'm willing to help out if you have time trying to fix some of these failures :)

[w1th0utnam3]

@elrnv I'm not sure if I should open a new issue for this but it seems to be related. A user of my surface reconstruction tool had problems loading a file that they converted from VTU to VTK using Paraview.
I attached two sample files which both result in a nom error "Parse error: Alt" with the latest commit in your release-0.7 branch.
For me it's a bit unclear how to debug with the macro based interface of nom, so I wasn't able to find out where the error actually occurs.

Here is the content of the simpler file that causes the error:

# vtk DataFile Version 5.1
vtk output
ASCII
DATASET UNSTRUCTURED_GRID

POINTS 8 double
0.5 0.4995095 0.5 0.5 0.4995095 1 0.5 0.9995095 0.5 
0.5 0.9995095 1 1 0.4995095 0.5 1 0.4995095 1 
1 0.9995095 0.5 1 0.9995095 1 

CELLS 9 8

OFFSETS vtktypeint64
0 1 2 3 4 5 6 7 8 

CONNECTIVITY vtktypeint64
0 1 2 3 4 5 6 7 

CELL_TYPES 8
1
1
1
1
1
1
1
1

Files to reproduce: legacy.zip

[elrnv]

Thank you @w1th0utnam3 for the flagging this! As far as I can tell the issue is the with DOS line endings. In the short term, feel free to swap to using Unix line endings, but please feel free to open another issue requesting support for dos line endings. It should be a simple fix in the parser.

[w1th0utnam3]

Thanks for looking into this. I formatted the files to use LF instead of CRLF for the line endings, but for me it did not seem to fix the issue, I still get Parse error: Alt. Did it work for you?

[elrnv]

I only tested the simplified vtk file in the zip. Now looking at legacy_full.vtk, it makes sense that it doesn't work, there are whole bunch of new fields like INFORMATION, METADATA, NAME, and DATA. This is a different issue than supporting CRLF line endings. To unblock you, you should be able to just remove those fields, but it would be good to add automatic support for them in the parser.

[w1th0utnam3]

Ah, sorry, I think when I tried to load the file with LF line endings I forgot to use the version 0.7 branch. In this branch changing from CRLF to LF actually works as you said. I'll open a separate issue regarding the line endings.
Even the legacy_full.vtk works, to me it looks like it silently ignores these METADATA lines.

However it also doesn't load the attributes in the file. I played a bit around and it seems like the usage of the datatype vtktypeint64 (which is not documented for the VTK legacy format) of the index attribute causes it to fail to parse all the attributes. When I change this datatype to int, the entire file is loaded as I would expect (minus the missing metadata, where I anyway don't know what that is). Maybe we could consider supporting this datatype as this is apparently supported by Paraview and also used for OFFSETS and CONNECTIVITY?

[elrnv]

Yes absolutely I agree we should add support for vtktypeinit64 and whatever else new versions of VTK are spitting out even if undocumented. I think we want vtkio to be compatible with other tooling people use in the wild.