VTK parsing error

[efaulhaber]

I have particle data in XML VTU format. I've tried

1. loading the file into ParaView and exporting it as legacy .vtk files,
2. converting the file with ParaView's Python interface.

Both yield perfectly fine .vtk files that can be opened correctly by ParaView.
However, both produce parsing errors when I try to run them through splashsurf:

[2023-04-05T16:30:16.737039+02:00][splashsurf][ERROR]   caused by: Failed to load VTK file "test1.vtk"
[2023-04-05T16:30:16.737046+02:00][splashsurf][ERROR]   caused by: Parse error: Alt

[w1th0utnam3]

It's a bit surprising to me that the legacy output of Paraview does not seem to be supported. Can you upload a VTK file so I can reproduce the issue? I can look into it next week.

I could also consider adding support for the XML format, however I don't have any input files I could use for testing at the moment. So it would also be appreciated if you could provide me some files.

For now you can consider converting the file with the Python tool meshio convert, though you may have to manually specify --output-format vtk42 but I'm not sure about that.

[efaulhaber]

Thanks for looking into this.

I've experimented with meshio earlier, but I couldn't get the files into splashsurf either. The --output-format vtk42 did the trick, thanks!

It seems that vtkio supports XML VTU files, so I thought I might only need to change a few lines to create a PR for this feature. I might look into this when I have some time, but I would greatly appreciate if you would add that feature, since I've never worked with Rust before.

Here is an example XML file and the exported legacy VTK from ParaView that can't be read by splashsurf.
archive.zip

[w1th0utnam3]

Thanks! I'll have a look.

[w1th0utnam3]

It seems that vtkio supports XML VTU files, so I thought I might only need to change a few lines to create a PR for this feature. I might look into this when I have some time, but I would greatly appreciate if you would add that feature, since I've never worked with Rust before.

I enabled loading of VTU files, it was indeed just a few lines. You can try it out by installing splashsurf from this repository using

cargo install --git https://github.com/w1th0utnam3/splashsurf.git

I'll release a new version soon, but maybe you can already try out if everything works as expected. I guess you also want to interpolate attributes? So maybe you can try if that works correctly with VTU files.

Unfortunately, I didn't get the legacy VTK loading to work. It seems like depending on the input file Paraview generates legacy VTK files with a syntax that doesn't match what vtkio expects (similar to elrnv/vtkio#21). I'll try to report this in the issue.

[efaulhaber]

Thanks a lot for looking into this.
I now get this error:

ltt027:out_falling_spheres1 erik$ ~/.cargo/bin/splashsurf reconstruct -i fluid_1_93.vtu --cube-size 0.4 --particle-radius 0.014 --smoothing-length 1.2
[2023-04-18T14:01:41.489403+02:00][splashsurf][INFO] splashsurf v0.9.0 (splashsurf)
[2023-04-18T14:01:41.489830+02:00][splashsurf][INFO] Called with command line: /Users/erik/.cargo/bin/splashsurf reconstruct -i fluid_1_93.vtu --cube-size 0.4 --particle-radius 0.014 --smoothing-length 1.2
[2023-04-18T14:01:41.489955+02:00][splashsurf::reconstruction][INFO] Using single precision (f32) for surface reconstruction.
[2023-04-18T14:01:41.490142+02:00][splashsurf::io][INFO] Reading particle dataset from "fluid_1_93.vtu"...
[2023-04-18T14:01:41.491365+02:00][splashsurf][ERROR] Error occurred: Failed to load particle positions from file "fluid_1_93.vtu"
[2023-04-18T14:01:41.491372+02:00][splashsurf][ERROR]   caused by: Failed to load VTK file "fluid_1_93.vtu"
[2023-04-18T14:01:41.491375+02:00][splashsurf][ERROR]   caused by: XML error: Deserialization error: Xml(UnexpectedEof("XmlDecl"))
[2023-04-18T14:01:41.491379+02:00][splashsurf][ERROR]   caused by: Deserialization error: Xml(UnexpectedEof("XmlDecl"))
[2023-04-18T14:01:41.491382+02:00][splashsurf][ERROR]   caused by: Unexpected EOF during reading XmlDecl.
[2023-04-18T14:01:41.491385+02:00][splashsurf][ERROR]   caused by: Unexpected EOF during reading XmlDecl.

[w1th0utnam3]

Hm, is it possible that the file is partially written or is missing some closing tag for whatever reason?
Unfortunately I only have the XML file you provided so I cannot really reproduce it.

[efaulhaber]

Interesting, both were generated by the same code. I'll try to reproduce that with a smaller file that I can send to you.

[efaulhaber]

fluid_1_91.vtu.zip
This file also fails with

ltt027:out erik$ ~/.cargo/bin/splashsurf reconstruct -i fluid_1_91.vtu --cube-size 0.4 --particle-radius 0.28 --smoothing-length 1.2
[2023-04-18T16:08:10.075408+02:00][splashsurf][INFO] splashsurf v0.9.0 (splashsurf)
[2023-04-18T16:08:10.075498+02:00][splashsurf][INFO] Called with command line: /Users/erik/.cargo/bin/splashsurf reconstruct -i fluid_1_91.vtu --cube-size 0.4 --particle-radius 0.28 --smoothing-length 1.2
[2023-04-18T16:08:10.075515+02:00][splashsurf::reconstruction][INFO] Using single precision (f32) for surface reconstruction.
[2023-04-18T16:08:10.075582+02:00][splashsurf::io][INFO] Reading particle dataset from "fluid_1_91.vtu"...
[2023-04-18T16:08:10.075739+02:00][splashsurf][ERROR] Error occurred: Failed to load particle positions from file "fluid_1_91.vtu"
[2023-04-18T16:08:10.075754+02:00][splashsurf][ERROR]   caused by: Failed to load VTK file "fluid_1_91.vtu"
[2023-04-18T16:08:10.075756+02:00][splashsurf][ERROR]   caused by: XML error: Deserialization error: Xml(UnexpectedEof("</Err(Utf8Error { valid_up_to: 0, error_len: Some(1) })>"))
[2023-04-18T16:08:10.075762+02:00][splashsurf][ERROR]   caused by: Deserialization error: Xml(UnexpectedEof("</Err(Utf8Error { valid_up_to: 0, error_len: Some(1) })>"))
[2023-04-18T16:08:10.075764+02:00][splashsurf][ERROR]   caused by: Unexpected EOF during reading </Err(Utf8Error { valid_up_to: 0, error_len: Some(1) })>.
[2023-04-18T16:08:10.075767+02:00][splashsurf][ERROR]   caused by: Unexpected EOF during reading </Err(Utf8Error { valid_up_to: 0, error_len: Some(1) })>.

It's been generated with the same Julia code (using WriteVTK.jl) as the "source.vtu" file earlier.

Edit: Note that neither ParaView nor meshio have any issues with this file.

[w1th0utnam3]

It works if I re-export the file as VTU with Paraview and either select to write the data arrays as ASCII, (encoded) binary or appended & encoded. I think the problem is that the XML parser used by vtkio cannot handle the binary encoding in the <AppendedData encoding="raw">...</AppendedData> binary blob. The "encoding" option in Paraview writes it using base64 encoding or something like that which works.
I think for now I cannot do much about that except for reporting this to vtkio. Can I re-post this file to an issue in the vtkio repo?

Maybe you can see if you can set an option in the Julia library to change the encoding for now as a workaround?

[efaulhaber]

Thanks! Feel free to repost the file.
Unfortunately, there doesn't seem to be any options like that for the Julia package. But it's interesting that the first file worked, which was exported by the same package.

For now, I'll use meshio to convert to legacy XML with the option you gave me earlier.

[w1th0utnam3]

Thanks! Feel free to repost the file.
Unfortunately, there doesn't seem to be any options like that for the Julia package. But it's interesting that the first file worked, which was exported by the same package.

I think for the first XML file the binary blob just happened to not include end of file characters or similar so the XML parser library was able to parse it. Maybe this way of storing raw binary data is just not supported by the library.

Ok, I'll report this with vtkio then.

[w1th0utnam3]

Thanks! Feel free to repost the file. Unfortunately, there doesn't seem to be any options like that for the Julia package. But it's interesting that the first file worked, which was exported by the same package.

@efaulhaber I just found this page: https://juliavtk.github.io/WriteVTK.jl/stable/grids/syntax/#Data-formatting-options
It seems like you can set the keyword argument append=false to use the inline base64 encoded format that should be supported.

[efaulhaber]

Ah, right. I completely forgot about this.
With this option, I get

ltt027:out erik$ ~/.cargo/bin/splashsurf reconstruct -i fluid_1_0.vtu --cube-size 0.4 --particle-radius 0.014 --smoothing-length 1.2
[2023-04-26T13:09:45.799643+02:00][splashsurf][INFO] splashsurf v0.9.0 (splashsurf)
[2023-04-26T13:09:45.799794+02:00][splashsurf][INFO] Called with command line: /Users/erik/.cargo/bin/splashsurf reconstruct -i fluid_1_0.vtu --cube-size 0.4 --particle-radius 0.014 --smoothing-length 1.2
[2023-04-26T13:09:45.799819+02:00][splashsurf::reconstruction][INFO] Using single precision (f32) for surface reconstruction.
[2023-04-26T13:09:45.799875+02:00][splashsurf::io][INFO] Reading particle dataset from "fluid_1_0.vtu"...
[2023-04-26T13:09:45.800027+02:00][splashsurf][ERROR] Error occurred: Failed to load particle positions from file "fluid_1_0.vtu"
[2023-04-26T13:09:45.800035+02:00][splashsurf][ERROR]   caused by: Failed to load VTK file "fluid_1_0.vtu"
[2023-04-26T13:09:45.800038+02:00][splashsurf][ERROR]   caused by: XML error: Validation error: Base64Decode(InvalidByte(43, 61))
[2023-04-26T13:09:45.800045+02:00][splashsurf][ERROR]   caused by: Validation error: Base64Decode(InvalidByte(43, 61))
[2023-04-26T13:09:45.800048+02:00][splashsurf][ERROR]   caused by: Base64 decode error: Invalid byte 61, offset 43.
[2023-04-26T13:09:45.800052+02:00][splashsurf][ERROR]   caused by: Invalid byte 61, offset 43.

Here's the file:
fluid_1_0.vtu.zip