Improve the deployment script #598

wilzbach · 2018-04-05T02:24:57Z

Testing dlang-community/discussions#9 (comment)

As I don't have OSX, I simply added make release to the Travis script for now. Let's see if this works out of the box.

wilzbach · 2018-04-05T05:09:35Z

release.sh

+unameOut="$(uname -s)"
+case "$unameOut" in
+    Linux*) OS=linux; ;;
+    Darwin*) OS=osx; LDC_FLAGS+=("-macosx_version_min 10.7" "-lcrt1.o"); ;;


ldc2: Unknown command line argument '-macosx_version_min'. Try: 'ldc2 -help' ldc2: Did you mean '-dwarf-version'? ldc2: Unknown command line argument '-lcrt1.o'. Try: 'ldc2 -help'

:/

These are linker flags. You can check the DStep Dub file: https://github.com/jacob-carlborg/dstep/blob/19a5ddb8ad3f44b3445c89840155bb7cd1ee44b7/dub.json#L20

wilzbach · 2018-04-05T07:33:12Z

makefile

@@ -76,26 +76,5 @@ report: all
 	dscanner --report src > src/dscanner-report.json
 	sonar-runner

-.ONESHELL:


ONESHELL is only available in Make 3.82.
Travis uses 3.81 and there's no easy way to upgrade.

wilzbach · 2018-04-05T07:35:13Z

.travis.yml

@@ -16,10 +16,10 @@ script: "./.travis.sh"
 jobs:
  include:
    - stage: GitHub Release
-      if: tag IS present
-      d: ldc
+      #if: tag IS present


Apparently building the binaries on Travis is vastly more complicated than thought, so let's better constantly build them to avoid bad surprises on releases. Imho the ~1.5min are worth the mind peace.

wilzbach · 2018-04-05T07:36:20Z

.travis.yml

-      if: tag IS present
-      d: ldc
+      #if: tag IS present
+      d: ldc-1.8.0


Locking the release compiler. Just in case there are any regressions coming up (I experienced some with the dlang-tour and it was a hell to figure out that the sudden failures were due to an automatic compiler upgrade)

wilzbach · 2018-04-05T08:06:23Z

@bbasile FYI: this is ready now. Would be nice if we can merge this before tagging 0.5.1 - thanks!

jacob-carlborg · 2018-04-05T08:10:23Z

release.sh

+    *) echo "Unknown ARCH: $ARCH"; exit 1
+esac
+
+archiveName="dscanner-$VERSION-$OS-$ARCH_SUFFIX.tar.gz"


Nitpick, binaries/packages for macOS usually don't contain the architecture. If multiple architectures are provided it's expected to be provided in a single universal binary.

Hmm, we do the same for dub too (https://github.com/dlang/dub/releases, http://code.dlang.org/download) as it makes automated downloads easier.
I just saw that LDC has the same pattern: https://github.com/ldc-developers/ldc/releases

jacob-carlborg · 2018-04-05T08:11:14Z

release.sh

+unameOut="$(uname -s)"
+case "$unameOut" in
+    Linux*) OS=linux; LDC_FLAGS=("-flto=full" "-linker=gold" "-static") ;;
+    Darwin*) OS=osx; LDC_FLAGS+=("-L-macosx_version_min" "-L10.7" "-L-lcrt1.o"); ;;


Nitpick, I would prefer "macOS" over osx, since this is what the OS is called now.

The naming is consistent with Dub's and LDC's releases:

https://github.com/dlang/dub/releases
https://github.com/ldc-developers/ldc/releases

Yes, please @wilzbach. He's right about the terminology. osx is a serie of Mac OS versions.

But that's true that even for pre-defined D version identifiers osx is used....

Well, to be honest I don't feel strongly about it as long as we finally get the automated release builds working. We have been pushing this off for far too long ;-)

The naming is consistent with Dub's and LDC's releases:
But that's true that even for pre-defined D version identifiers osx is used....

That doesn't mean it's correct 😃. The predefined version identifier was arbitrary chosen by Walter even though darwin already existed in other compilers. Back then the official name was "Mac OS X", so naming it "OSX" was neither correct.

ghost · 2018-04-05T08:17:34Z

@wilzbach there's still #575 that's planned for 0.5.1. It's simple, will do it today, the tag will follow then.

wilzbach · 2018-04-05T10:03:24Z

@bbasile I will merge this then, s.t. we can test it for 0.5.1
@jacob-carlborg If you feel strongly about the package name, we can always change it for 0.5.2 ;-)

wilzbach · 2018-04-05T10:58:01Z

Looks like this finally worked :)

I tested the Linux binary and it works fine - even on a 2.6.X kernel

So now we just need to figure out how to painlessly compile the Windows binaries.
Did someone try the Wine-built binary of DScanner that I posted here? If that works, it might be a feasible approach.
Otherwise I LDC supports cross-compiling since a while though it might require some work to get that working flawlessly in Docker (I saw on Slack that @jacob-carlborg gave this a shot recently) + there's the licensing problem.

jacob-carlborg · 2018-04-05T12:59:20Z

Otherwise I LDC supports cross-compiling since a while though it might require some work to get that working flawlessly in Docker (I saw on Slack that jacob-carlborg gave this a shot recently) + there's the licensing problem.

You don't need to run it in Docker. It simple enough to setup everything that is needed before building. This Docker image, compared with the one for macOS, did not require the multiarch/crossbuild base Docker image since LDC contains the cross-linker.

jacob-carlborg · 2018-04-05T13:06:43Z

If you feel strongly about the package name

I don't.

jacob-carlborg · 2018-04-06T06:32:56Z

Adding -L-dead_strip on macOS will reduce the binary size to almost a third. This corresponds to the --gc-sections flag that LDC uses by default on Linux. Not sure why the macOS flag is not enabled by default.

Adding -flto=full and running strip on the binary will get the size down to 3MB from the original 14MB.

wilzbach · 2018-04-06T10:15:41Z

You don't need to run it in Docker.

I know, but I was referring to downloading the VS libraries from Dropbox and copying the libs over.
That seems a bit fragile and like something I don't want to do for all X repositories as it seems like it might require small tweaks in the future.
I'm already not very well pleased about the duplication of the build script between D-Scanner, DCD and dfmt and thinking about moving the build script to tools ;-)

jacob-carlborg · 2018-04-06T11:02:31Z

Right. Perhaps a pre-packaged cross-compiler? Download and unpack. Event better if it can be upstreamed.

jacob-carlborg · 2018-04-13T06:18:24Z

So now we just need to figure out how to painlessly compile the Windows binaries.

@wilzbach If you don't want to learn PowerShell, you can use AppVeyor and install msys2 which bundles bash. I did some experiments and it's pretty straight forward [1]:

- cinst msys2 --params "/NoUpdate"
- bash -lic "echo 'asd'"

I'm not sure which tools are bundled but you can install additional tools using chocolatey [2] [3]. AppVeyor also has native support for GitHub releases [4].

[1] https://github.com/jacob-carlborg/dstep/blob/ea24360e5dbbdb8eebfbba869e4af1f42e9d18cc/appveyor.yml#L31-L38

[2] https://chocolatey.org/packages/curl
[3] https://www.appveyor.com/docs/build-configuration/#chocolatey
[4] https://www.appveyor.com/docs/deployment/github/

wilzbach · 2018-04-13T11:53:37Z

@wilzbach If you don't want to learn PowerShell, you can use AppVeyor and install msys2 which bundles bash. I did some experiments and it's pretty straight forward [1]:

Thanks a lot for your pointers, but we already went through this and while annoying PowerShell isn't too hard. Even this repo has a appveyor.yml that builds and uploads the binary:

https://github.com/dlang-community/D-Scanner/blob/master/appveyor.yml

But AppVeyor was disabled, because I think (?) there have been too many random failures.
For now, we went with using Travis and Wine to build the Windows binaries:

#603

wilzbach mentioned this pull request Apr 5, 2018

Binary releases for tools dlang-community/discussions#9

Open

wilzbach force-pushed the osx-minimal-deployment branch 2 times, most recently from 9485a53 to 12927ea Compare April 5, 2018 04:39

wilzbach commented Apr 5, 2018

View reviewed changes

wilzbach force-pushed the osx-minimal-deployment branch 9 times, most recently from 959bd99 to 8a8e357 Compare April 5, 2018 07:32

wilzbach commented Apr 5, 2018

View reviewed changes

Reduce the minimal deployment target for OSX

3fc2059

wilzbach force-pushed the osx-minimal-deployment branch from 8a8e357 to 3fc2059 Compare April 5, 2018 07:34

wilzbach commented Apr 5, 2018

View reviewed changes

wilzbach changed the title ~~Reduce the minimal deployment target for OSX~~ Improve the deployment script Apr 5, 2018

wilzbach added this to the 0.5.1 milestone Apr 5, 2018

jacob-carlborg reviewed Apr 5, 2018

View reviewed changes

wilzbach merged commit 6ca59c7 into dlang-community:master Apr 5, 2018

wilzbach deleted the osx-minimal-deployment branch April 5, 2018 10:03

wilzbach mentioned this pull request Apr 5, 2018

Use full Link-Time Optimization (LTO) for the release binaries dlang/dub#1433

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve the deployment script #598

Improve the deployment script #598

wilzbach commented Apr 5, 2018

wilzbach Apr 5, 2018

jacob-carlborg Apr 5, 2018 •

edited

Loading

wilzbach Apr 5, 2018

wilzbach Apr 5, 2018

wilzbach Apr 5, 2018

wilzbach commented Apr 5, 2018

jacob-carlborg Apr 5, 2018

wilzbach Apr 5, 2018

jacob-carlborg Apr 5, 2018

wilzbach Apr 5, 2018

ghost Apr 5, 2018 •

edited by ghost

Loading

ghost Apr 5, 2018

wilzbach Apr 5, 2018

jacob-carlborg Apr 5, 2018

ghost commented Apr 5, 2018

wilzbach commented Apr 5, 2018

wilzbach commented Apr 5, 2018

jacob-carlborg commented Apr 5, 2018

jacob-carlborg commented Apr 5, 2018

jacob-carlborg commented Apr 6, 2018

wilzbach commented Apr 6, 2018

jacob-carlborg commented Apr 6, 2018

jacob-carlborg commented Apr 13, 2018

wilzbach commented Apr 13, 2018

Improve the deployment script #598

Improve the deployment script #598

Conversation

wilzbach commented Apr 5, 2018

Choose a reason for hiding this comment

jacob-carlborg Apr 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wilzbach commented Apr 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost Apr 5, 2018 • edited by ghost Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Apr 5, 2018

wilzbach commented Apr 5, 2018

wilzbach commented Apr 5, 2018

jacob-carlborg commented Apr 5, 2018

jacob-carlborg commented Apr 5, 2018

jacob-carlborg commented Apr 6, 2018

wilzbach commented Apr 6, 2018

jacob-carlborg commented Apr 6, 2018

jacob-carlborg commented Apr 13, 2018

wilzbach commented Apr 13, 2018

jacob-carlborg Apr 5, 2018 •

edited

Loading

ghost Apr 5, 2018 •

edited by ghost

Loading