Advice on adding macOS support to the nix-ros-overlay

[hegde-atri]

I have an access to macOS(arm64) laptop in which I'd like to use ROS2 Humble.

I want to contribute to this project, but being new to nix & nix flakes I would like pointers to go down or to work on.

If there are any contribution guidelines I missed do let me know :)

[hegde-atri]

Cloning the repository and running nix-shell ros2-basic.nix in the examples directory yields the following output.

error:
       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:9:12:
            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'nix-shell'
         whose name attribute is located at /nix/store/snp5f95xhkz89kdw9703kana3m6xfd0a-source/pkgs/stdenv/generic/make-derivation.nix:352:7

       … while evaluating attribute 'nativeBuildInputs' of derivation 'nix-shell'
         at /nix/store/snp5f95xhkz89kdw9703kana3m6xfd0a-source/pkgs/stdenv/generic/make-derivation.nix:396:7:
          395|       depsBuildBuild              = elemAt (elemAt dependencies 0) 0;
          396|       nativeBuildInputs           = elemAt (elemAt dependencies 0) 1;
             |       ^
          397|       depsBuildTarget             = elemAt (elemAt dependencies 0) 2;

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: Package ‘acl-2.3.1’ in /nix/store/snp5f95xhkz89kdw9703kana3m6xfd0a-source/pkgs/development/libraries/acl/default.nix:40 is not available on the requested hostPlatform:
         hostPlatform.config = "aarch64-apple-darwin"
         package.meta.platforms = [
           "aarch64-linux"
           "armv5tel-linux"
           "armv6l-linux"
           "armv7a-linux"
           "armv7l-linux"
           "i686-linux"
           "loongarch64-linux"
           "m68k-linux"
           "microblaze-linux"
           "microblazeel-linux"
           "mips-linux"
           "mips64-linux"
           "mips64el-linux"
           "mipsel-linux"
           "powerpc64-linux"
           "powerpc64le-linux"
           "riscv32-linux"
           "riscv64-linux"
           "s390-linux"
           "s390x-linux"
           "x86_64-linux"
         ]
         package.meta.badPlatforms = [ ]
       , refusing to evaluate.

       a) To temporarily allow packages that are unsupported for this system, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1

          Note: When using `nix shell`, `nix build`, `nix develop`, etc with a flake,
                then pass `--impure` in order to allow use of environment variables.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowUnsupportedSystem = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowUnsupportedSystem = true; }
       to ~/.config/nixpkgs/config.nix.

[wentasah]

Have you tried what the error message suggests?

Try running:

NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1 nix-shell ros2-basic.nix

The problem seems that something depends on package acl, which is not supported on MacOS (which makes sense, because MacOS probably implements ACLs differently from Linux). On Linux, I see that acl is a dependency of cmake (via libarchive) and coreutils, but all of these are conditional on Linux.

If the above doesn't help, can you try running

nix-build -A rosPackages.humble.ros-core --show-trace

in the root directory of the repo?

[hegde-atri]

Thank you for your explanation on acl, it does clear things up.

I know that on the ROS2 documentation they've listed all the dependencies they need for building the packages. Would it be possible for me to build all the packages using Nix, and then cache it somewhere to avoid building it every single time? I am really new to packaging on nix as I've only used it to create quick developmental shells before this.

Running the nix-build command as you've given errors on acl not being available on the requested host platform - output

Yep I did try using the unsupported system flag. This was where it errored - output

[wentasah]

I know that on the ROS2 documentation they've listed all the dependencies they need for building the packages. Would it be possible for me to build all the packages using Nix, and then cache it somewhere to avoid building it every single time? I am really new to packaging on nix as I've only used it to create quick developmental shells before this.

That's how nix works by default. When you build the package, they stay cached until you run garbage collection (GC) and the built packages are not protected by a GC root.

Whenever you do nix build or nix-build, it creates the ./result symlink, which is the GC root. Unless you delete that link, the compiled packages should stay cached.

If you use development environments (nix-shell, nix develop), the results are not GC roots so things need to be rebuild after every GC. You can prevent this by using nix-direnv for your dev environments.

Running the nix-build command as you've given errors on acl not being available on the requested host platform - output

Ok. So the problem is that iceoryx_hoofs depends on acl unconditionally. This package is a dependency of CycloneDDS, which is not used by default on humble, because humble default to FastDDS. CycloneDDS only only a build-time (not run-time) dependency of rmw-implementation. It should be possible to build rmw-implementation without CycloneDDS. Try changing the definition of rmw-implementation as follows:

diff --git a/distros/humble/rmw-implementation/default.nix b/distros/humble/rmw-implementation/default.nix
index 5957d7baf..e797c9195 100644
--- a/distros/humble/rmw-implementation/default.nix
+++ b/distros/humble/rmw-implementation/default.nix
@@ -14,7 +14,7 @@ buildRosPackage {
   };
 
   buildType = "ament_cmake";
-  buildInputs = [ ament-cmake rmw rmw-connextdds rmw-cyclonedds-cpp rmw-fastrtps-cpp rmw-fastrtps-dynamic-cpp ];
+  buildInputs = [ ament-cmake rmw rmw-connextdds rmw-fastrtps-cpp rmw-fastrtps-dynamic-cpp ];
   checkInputs = [ ament-cmake-gtest ament-lint-auto ament-lint-common performance-test-fixture ];
   propagatedBuildInputs = [ ament-index-cpp rcpputils rcutils rmw-implementation-cmake ];
   nativeBuildInputs = [ ament-cmake rmw-implementation-cmake ];

Then repeat the build and we should get a bit further. If that would work, the question will be how to make the change permanent, because the above is just a hack. If it fails, post the log again.

[hegde-atri]

After applying your patch it errors while building ros-humble-builtin-interfaces I think. Here's the full output.

Your explanations are helping a lot thankyou :)

[wentasah]

It fails because the build looks for /nix/store/03q8gn91mj95y5bqbcl90hyvmpqpz738-python3-3.11.7/lib/libpython3.11.so, whereas the actual file name is probably something like /nix/store/03q8gn91mj95y5bqbcl90hyvmpqpz738-python3-3.11.7/lib/libpython3.11.dylib (different extension).

I can reproduce the same problem on x86 Mac (GitHub runner) when running nix-shell -A rosPackages.humble.builtin-interfaces. However, when I run nix-shell -A rosPackages.rolling.builtin-interfaces, it builds fine. Therefore, this is likely a problem in some ROS package (my guess is some rosidl_* package), which have been fixed but was not backported to humble.

If this the same on your end, can you try finding the fix in ROS repositories? Then, this could be backported to humble, or if that's not possible, we could apply the patch in this overlay.

[hegde-atri]

Both commands nix-shell -A rosPackages.rolling.builtin-interfaces and nix-shell -A rosPackages.humble.builtin-interfaces run without errors and drops me into a nix-shell.

However when I try doing nix-shell -A rosPackages.humble.ros-core it seems to error with this output and running nix log on this gives me this output.

[wentasah]

Ah sorry, my fault. I'm no longer at the computer. Can you try nix-build instead of nix-shell? I'll have closer look in the evening.

[hegde-atri]

Ah yes, rolling seems to build fine, but humble seems to have the same error as you mentioned. I will go try to find a fix in the ROS repositories and get back to you :)

[hegde-atri]

I couldn't find any commits (this repo) that might've had this change. Is this the correct repository to be looking at?

[wentasah]

It seems that that error was caused by python-cmake-module. If I use the version from rolling in humble:

cp ./distros/{rolling,humble}/python-cmake-module/default.nix
nix-build -A rosPackages.humble.builtin-interfaces

I no longer get the error about libpython3.11.so, but about missing Python.h:

../rosidl_generator_py/builtin_interfaces/msg/_duration_s.c:5:10: fatal error: 'Python.h' file not found

Is is the case on your side too?

[hegde-atri]

Yep, I get the same error.

[wentasah]

It seems that my previous hypothesis about a need of a fix on ROS side was not completely correct. There were changes that allow building rolling, but the following change in this repo allows building builtin-interfaces also for humble:

diff --git a/distros/ros2-overlay.nix b/distros/ros2-overlay.nix
index 4f3d45a00..f57c8894b 100644
--- a/distros/ros2-overlay.nix
+++ b/distros/ros2-overlay.nix
@@ -90,7 +90,7 @@ rosSelf: rosSuper: with rosSelf.lib; {
     python = rosSelf.python;
   in {
     pythonExecutable = python.pythonOnBuildForHost.interpreter;
-    pythonLibrary = "${python}/lib/lib${python.libPrefix}.so";
+    pythonLibrary = "${python}/lib/lib${python.libPrefix}.dylib";
     pythonIncludeDir = "${python}/include/${python.libPrefix}";
     setupHook = ./python-cmake-module-setup-hook.sh;
     outputs = [ "out" "dev" ];

With this and the above change, I'm able to build ros-core for humble on an x86 Mac.

[hegde-atri]

Yes, that indeed does work! Now I can use nix-shell on ./examples/ros2-basic.nix, Amazing stuff.

Now I've added rviz2 to the to the ros2-basic.nix file and have errors with ogre. Specifically, ros-humble-rviz-ogre-vendor. Seems like something fails after not finding the CPU target. Here is the output.

I was trying to build rviz manually by cloning it and building it in nix-shell. However it errored since I do not have my ROS2 environment setup correctly with the various cmake flags like in ./distros/ros2-overlay.nix. I wanted to ask how you would setup a reliable/reproducible way to use ros2 on macOS. Would I just create a flake for this? Thanks for all your input and making this possible.

[hegde-atri]

When trying to use ros2 from a flake(as shown in the README). I get an error (output) to do with sros2 - Failed to load entry point 'security'. This does not happen when using nix-shell in the examples directory.

I get this same issue on linux. (On linux I fllow this repo, whereas on the mac I follow my fork of the repo with the macOS patches)

When on the mac I run ros2 node list which runs completely fine on linux, but errors at foonathan::memory - here the output. This also occurs when in nix-shell in the examples directory.

Have you seen this before?

[wentasah]

I wanted to ask how you would setup a reliable/reproducible way to use ros2 on macOS. Would I just create a flake for this?

Yes. I'm creating a flake for every project I work with. Try modifying your flake according to #402. It might help. I've never seen the error with the foonathan memory allocator. If #402 doesn't help, we'll have to go deeper.

I've tried to prepare the fixes we discussed above in a (hopefully) mergable form in #421. Can you test them?

With respect to rviz (and other problems) I'll have more time in a week or two, so I'll try to look at it then.

[hegde-atri]

#421 - This works perfectly fine for me. I tested it changing this line in the inputs of my flake.

nix-ros-overlay.url = "github:wentasah/nix-ros-overlay";

Modifying my flake according to #402 did fix my issues!

Thats completely fine! Thank you for the great support :)

[hegde-atri]

Tried a few hacks to try and fix the error with foonathan memory allocator, looks like a deeper look is required.

[hegde-atri]

Hey @wentasah , How would I go about adding a patch to ros-humble-rcl to make this work?
I'm puzzled as to where I would even try to fix it 😅

[stevalkr]

When on the mac I run ros2 node list which runs completely fine on linux, but errors at foonathan::memory - here the output. This also occurs when in nix-shell in the examples directory.

@hegde-atri @wentasah Hello, I found this issue fixed in foonathan-memory 0.7-3, please test this PR #442

[hegde-atri]

Thanks @etherswangel! It fixes the errors.

Do you have any idea how to solve the rviz2 error?