Run Windows CI and publish artifacts

[ComFreek]

Is your feature request related to a problem? Please describe.

Currently, Curv has the working Travis CI file (at least working in my fork) that I contributed to months ago, but

• I cannot find curv3d/curv on Travis CI, neither on travis-ci.org nor travis-ci.com. I suspect it's not configured, so the CI does not even run
• the Travis build is slow since it redownloads and reinstalls all the Mingw64 packages we need
• Travis builds do not allow to easily store artifacts

Describe the solution you'd like
Use GitHub actions. I have already come up with a working build which also uploads Curv Windows binaries: https://github.com/ComFreek/curv/runs/886676572

Advantages:

• look at how small and concise the workflow file is in comparison to the Travis file this repo currently has.

• the subaction msys2/setup-msys has built-in ability to cache all Mingw64 packages!

• artifact upload is a breeze

Right now the uploaded Windows binaries claim libHalf-2_5.dll,tbb.dll,libopenvdb.dll,libboost_filesystem-mt.dll are missing. A straight-forward copy procedure in the CMake configuration should be able to copy those DLLs automatically to ./release from which the GitHub Actions aggregates the artifact files and uploads them.

Describe alternatives you've considered

• Travis CI: old solution, see disadvantages above
• AppVeyor: GitHub Actions just worked surprisingly well that I didn't look into AppVeyor anymore

[doug-moen]

Thanks for telling me about Github Actions. I will read the documentation and see if I can get it working.

[doug-moen]

I copied your main.yml file into the Curv repo, and it ran automatically. Very cool.

There was a bug in the step "Copy runtime dependencies to build dir": it was using incorrect pathnames for the DLLs. After fixing the pathnames, it ran to completion. (No need to mess with the CMake configuration.)

[doug-moen]

The CI integration is very useful. I get email if the windows build breaks. Thanks for that.

[doug-moen]

What problem are you trying to solve with the "publish artifacts" part of this script?

curv.exe is not usable by itself. It needs many DLLs from MSYS2, of which only 4 are included in the current set of artifacts. It also needs the lib/curv directory (which needs to be kept in sync with curv.exe), and g++, and the glm header files. So a user cannot download the present build artifacts and get a runnable Curv program.

[ComFreek]

There was a bug in the step "Copy runtime dependencies to build dir": it was using incorrect pathnames for the DLLs. After fixing the pathnames, it ran to completion.

Ah, wonderful! Running curv.exe from the artifact publisher in your last action build in this repo still reports those DLLs as missing, unfortunately. Not sure why as they are in the same directory.

(No need to mess with the CMake configuration.)

It seemed a bit saner to be to let CMake do the copying - but that turned out to be non-trivial to get working. However, there was also another reason I messed with the CMake configuration: to add ccache.
As of now, the build always starts from zero and hence takes >= 10min. If we instead cache compilation of files (via ccache within the VM and then via @actions/cache to GitHub), build times may drastically be reduced.

What problem are you trying to solve with the "publish artifacts" part of this script?

Ideally, it would package all DLLs needed and be instantly runnable on Windows. It only needs g++ for this specific optimization that is non-critical to miss, right?
I think a self-contained Windows build that is downloadable as a ZIP and showcases what curv can do would be great. I would have loved it at least before I started messing with building curv 😄

[ComFreek]

I opened curv.exe from the published artifact with the program Dependencies. It shows the required DLLs and how they are resolved:

It seems we should also add copy instructions for libwinpthread-1.dll,libgcc_s_seh-1.dll,libstdc++-6.dll,libboost_iostreams-mt.dll.

[doug-moen]

The ldd command in MSYS2 shows a larger set of DLLs, because it is also finding indirect dependencies:

$ ldd curv.exe|grep -e /mingw64/bin|sort -u|awk '{print $1}'
libblosc.dll
libboost_filesystem-mt.dll
libboost_iostreams-mt.dll
libbz2-1.dll
libgcc_s_seh-1.dll
libHalf-2_5.dll
liblz4.dll
liblzma-5.dll
libopenvdb.dll
libsnappy.dll
libstdc++-6.dll
libwinpthread-1.dll
libzstd.dll
tbb.dll
zlib1.dll

Hard coding the list of libraries would be fragile. Fortunately, we can put the above shell command into the build script to compute the list dynamically.

[ComFreek]

Do all those files reside in /usr/bin, too? I am not sure. Is there a *nix way to copy from wherever the library can be found on PATH?

[doug-moen]

I want the simplified Windows installation to contain g++; that's needed for generating STL files and 3D printing.

My current idea is to use the following procedure:

• install MSYS2
• download windows.sh from https://raw.githubusercontent.com/curv3d/curv/master/windows.sh
• run windows.sh in a MinGW64 shell window

windows.sh is a shell script that contains the current install procedure. It is tested by running it in the Github Actions CI server.

This approach will give us a simple upgrade procedure:

• cd /curv; make upgrade; make

It will allow Curv to be uninstalled using the uninstall GUI in the Windows Start menu (you uninstall MSYS2).

[ComFreek]

While that sounds totally doable, it still has the problem that the user has to spend time compiling Curv. Even if that is done by windows.sh in an unsupervised fashion, it's time-consuming.
I don't know about you, but many Windows users (myself included) instantly throw software/websites* away if they don't install/compile/load within seconds :-)
*) meaning things we are interested in only casually

How big is MSYS2? Would it be feasible to offer a reduced MSYS2 (incl. g++) + pre-built Curv therein?

[doug-moen]

I just tested this, and you are correct. The Curv build is the most time consuming part. Maybe a hybrid approach would work.

I have no idea how to create my own MSYS2 distribution. Sounds like a lot of work (possibly with ongoing maintenance)? I'm not inclined to pour a large amount of work into this.

IMO a more reasonable approach with MSYS is to add a Curv package. Then you install MSYS2 and:

• pacman -S curv

All the needed dependencies are pulled in automatically, so it's just this one command.

To satisfy people who want to try out Curv "within seconds", the appropriate response is to make it run inside a web browser.
I have an idea for this:

• create a single-window, 3 pane GUI version of Curv, with the technology I'm using today: glfw and imgui.
• use the Zep project to provide the text editor and console panes.
• use emscripten to compile this to webassembly
• embed the WASM module in a web site

Exporting a shape to a file will not be initially supported. But you can install the native client to get the full capabilities.

[doug-moen]

Correction, I just did a full Windows install from scratch, using the new script. Unfortunately I didn't time it, but the MSYS2 installation and pacman package installation took significantly longer than the Curv build. There's no way to do an install in "seconds" if MSYS2 is part of the installation. Either a web app or a stripped down GUI app (which I'm calling "gcurv") are required for a user-friendly instant-gratification experience.

[ComFreek]

the appropriate response is to make it run inside a web browser.

This sounds very cool if it works out. I've never compiled C(++) to JS via any of the tools, so can neither judge feasibility nor if end result would be usable perfomance-wise.

IMO a more reasonable approach with MSYS is to add a Curv package. Then you install MSYS2 and: [...]

All the needed dependencies are pulled in automatically, so it's just this one command.

That I would find reasonable enough for Windows users. At least here you know as an end user that the probability of an error is much lower than compared to self-compilation.
But can MSYS2 packages come bundled with binaries? Or are they just abstraction layers over compilation commands?

scratch plan:

• improve GitHub action serving as CI:
  • switch to msys2/setup-msys@v2 (they fixed a bug)
  • eventually employ their upcoming caching functionality to avoid reinstalling packages
• create MINGW64 Curv package (?)

[eine]

@doug-moen, @ComFreek, maintainer of msys2/setup-msys here. Just some hints that you might find useful:

• eine/tip is an Action that, used as a last step in a workflow, allows to upload artifacts as assets of a (pre-)release. This makes it easier for users to find latest successful build assets, instead of searching the list of CI runs. See, for example, https://github.com/ghdl/ghdl/blob/master/.github/workflows/push.yml#L104-L114 (see also Port nightly build to github actions msys2/msys2-installer#9).
• In ci: add PKGBUILD scripts and GitHub Actions workflow gtkwave/gtkwave#20 GtkWave is built in MSYS2, "regular packages" are generated and, for gtk3 builds "standalone zipfiles" are generated too. I believe that these "standalone" packages are what you are talking about. Note the following references:

I'd like to compare the "manual" procedure with https://github.com/achadwick/styrene (or some other reference from https://www.msys2.org/wiki/Distributing/) and/or https://www.msys2.org/wiki/Using-packages/#finding-dependencies-of-a-package.

• In GHDL, PKGBUILD files are used for internal testing (e.g. https://github.com/ghdl/ghdl/blob/master/dist/msys2-mingw/llvm/PKGBUILD). These are similar to the upstreamed PKGBUILD files, but they are slightly different because project sources are already available (not need to retrieve them explicitly during the build). I think that using a PKGBUILD file internally is a good starting point before submitting a PR to msys2/MINGW-packages.

[doug-moen]

MSYS2 packages are binary. The reason for slow installation is the large number of dependencies. C:\msys64 on my system has 86,700 files. That's a lot, just to install Curv.

I endorse the idea of a "light" version of Curv that installs quickly, and doesn't require a C++ compiler or an MSYS2 installation. Styrene (thanks @eine) looks promising as a tool for creating a Windows installer .exe and/or zipfile for Curv.

Curv is not a typical Windows app. It is a command line utility. The curv command is run from Command Prompt or PowerShell.

• Instead of creating a start menu item, the installer needs to add the directory containing the Curv executable to the Path variable in the registry.
• We don't want the DLLs that Curv uses to be visible in the Path, because these DLLs will conflict with an MSYS2 installation. The DLLs will have the same names as MSYS2 DLLs but may have different versions. The DLLs must be installed in a different directory than curv.exe. Therefore, the Curv install tree will contain bin/curv.exe and lib/*.dll. The curv.exe file will, after installation, use absolute path names to load its DLLs, or a relative pathname ../lib relative to the .exe file. it will not load the DLLs from the Path.
• The Curv standard library files (std.curv, etc) need to be installed somewhere that the Curv executable can find them: in lib/curv within the Curv install tree.
• The examples directory needs to be installed somewhere that the User can conveniently access from Command Prompt or Powershell, probably in a curv directory under the user's home directory.

If Styrene can satisfy these requirements by reading a config file, that would be great.

[ComFreek]

Styrene indeed sounds super useful for packaging a light Curv here.

Roadmap:

1. Create a (local, not necessarily published) MINGW64 package for Curv
2. Test Styrene on it
3. Modify GitHub actions to run Styrene and publish the artifact via eine/tip (not looked into it; but if it does what it promises, that'd be great!)

I probably won't have time this week to look into anything listed above 😄

[doug-moen]

It would be nice to statically link Curv.exe, so that it doesn't depend on any DLLs from the MSYS2 distribution. I just noticed that the Windows distribution of OpenSCAD does this: they distribution openscad.exe without any DLLs.

That would be another solution to the problem of how do you make curv.exe visible in the PATH without causing DLL conflicts.

[doug-moen]

Static linking is easy: just add -static to the MSYS2 link line for curv in the CMakeLists file.

It almost works, except that MSYS2 does not provide static libraries for libHalf and libtbb. These libraries are only needed as dependencies for libopenvdb.

Libopenvdb has been a "nightmare" dependency for a long time. It has created more trouble than all the others. I only use it for exporting a shape to a mesh file. I am going to consider replacing openvdb with an alternative open source meshing library. This has been on my TODO list for some time. There are a number to choose from on github, and the benefit is that I may be able to upgrade to a better algorithm that does sharp edge detection.