Remote build proxying

[LisannaAtHome]

I think the user story for many people sharing a cluster of remote nix builders is currently limited by the fact that remote building only works one level deep. i.e., if you have a cluster of build machines that you want to share with many Nix users, each user has to maintain their own copy of the list of each machine, and each user has to have an account on each of the machines (or otherwise share an account and SSH key).

This could be improved with allowing remote building to forward builds to machines that are themselves configured as remote builders on the currently selected remote builder. This would allow you to create clusters of build machines where in machine A has B, C, and D configured as remote builders, and anyone who wants to use the large cluster for remote building only has to know about and have access to machine A.

[edolstra]

Doesn't this already work? I haven't tried it, but in principle there is nothing stopping a remote builder from forwarding builds itself.

[LisannaAtHome]

There's at least one complication with this today: a proxy build server would have to have all of the supported features of anything it forwards to, even though that might not be the case. It must also be of the same system type.

An ideal user story for this would be something like a very lightweight proxy server which technically has a nix build environment and is setup to forward builds, but won't ever be used for any builds itself. I don't really see a way for a machine to forward a build it received without it also having some chance of being the thing that builds it. For build clusters with heterogeneous supported / mandatory features and systems, this won't work.

[stale]

I marked this as stale due to inactivity. → More info

[bryanhonof]

@edolstra I just tried to achieve sort-of the same thing, and I'm running into an issue.

Building for different architectures on the proxy server works just fine when executing nix commands on there directly.
However, when I try to do a build from my local machine through that proxy, it fails with the following message.
Note, my local machine is of type aarch64-darwin in this instance.

$ nix build -L --impure --expr '(with import <nixpkgs> { system = "aarch64-linux"; }; runCommand "foo" {} "uname -mo > $out")'
error: build of '/nix/store/nly18j4i88n7ppbfk9pkqy26j0qvvhxs-foo.drv' on 'ssh-ng://nixbld@proxyhost' failed: error: unable to start any build; remote machines may not have all required system features.
              https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html
error: builder for '/nix/store/nly18j4i88n7ppbfk9pkqy26j0qvvhxs-foo.drv' failed with exit code 1

Local machine config

My local nix.conf looks as follows.

allowed-users = root bryanhonof @staff
trusted-users = bryanhonof @staff
auto-optimise-store = false
builders = @/etc/nix/machines
builders-use-substitutes = true
download-attempts = 2
experimental-features = flakes ca-derivations impure-derivations nix-command recursive-nix repl-flake
log-lines = 25
max-jobs = auto
build-users-group = nixbld

Where /etc/nix/machines stores the following.

ssh-ng://nixbld@proxyhost x86_64-linux,aarch64-linux,x86_64-darwin,aarch64-darwin /etc/ssh/ssh_host_ed25519_key 100 1 kvm,big-parallel,nixos-test,benchmark - c3…

Proxy machine config

# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!
allowed-users = *
auto-optimise-store = false
connect-timeout = 10
cores = 0
extra-sandbox-paths = 
max-free = 1000000000
max-jobs = 0
min-free = 128000000
require-sigs = true
sandbox = true
sandbox-fallback = false
stalled-download-timeout = 30
substituters = https://cache.nixos.org/
system-features = kvm big-parallel nixos-test benchmark
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
trusted-substituters = 
trusted-users = @wheel
experimental-features = nix-command flakes

extra-trusted-users = nixbld
builders-use-substitutes = true
extra-platforms = x86_64-linux aarch64-linux x86_64-darwin aarch64-darwin

And it's /etc/nix/machines file.

ssh-ng://nixbld@x86-intel-builder x86_64-linux /etc/ssh/ssh_host_ed25519_key 2 1 kvm,big-parallel,nixos-test,benchmark - c3…
ssh-ng://nixbld@aarch64-intel-builders aarch64-linux /etc/ssh/ssh_host_ed25519_key 2 1 kvm,big-parallel,nixos-test,benchmark - c3…
ssh-ng://nixbld@x86-darwin-builder x86_64-darwin /etc/ssh/ssh_host_ed25519_key 6 1 kvm,big-parallel,nixos-test,benchmark - c3…
ssh-ng://nixbld@aarch64-darwin-builder aarch64-darwin /etc/ssh/ssh_host_ed25519_key 6 1 kvm,big-parallel,nixos-test,benchmark - c3…

I already tried setting the cores and max-jobs config to a higher value than 0, which allows me to build for x86_64-linux (the machine's architecture), but it doesn't proxy the build, it just builds it itself.

The nixbld user has some ssh config to it, so it can only execute the nix-daemon command. That's happening as follows.

{
  services.openssh.extraConfig = let
    command = pkgs.writeShellApplication {
      name = "command";
      runtimeInputs = [ pkgs.nix pkgs.coreutils ];
      text = builtins.readFile ./nixbld-force-command.sh;
    };
  in ''
      Match User nixbld
        AllowAgentForwarding no
        AllowTcpForwarding no
        PermitTTY no
        PermitTunnel no
        X11Forwarding no
        ForceCommand ${command}/bin/command
      Match All
  '';
  }

The ForceCommand is a simple bash script.

#!/usr/bin/env bash

if [[ "${SSH_ORIGINAL_COMMAND:=''}" != 'nix-daemon --stdio' ]]; then
    echo "Hi ${USER}! You've successfully authenticated!";
    echo "However, we do not provide shell access :( Sorry!";
    exit 0;
fi

nix-daemon --stdio; 

I believe I'm running into the following piece of code.

nix/src/libstore/build/worker.cc

Lines 307 to 325 in d00fe5f

	/* Wait for input. */
	if (!children.empty() || !waitingForAWhile.empty())
	waitForInput();
	else {
	if (awake.empty() && 0U == settings.maxBuildJobs)
	{
	if (getMachines().empty())
	throw Error("unable to start any build; either increase '--max-jobs' "
	"or enable remote builds."
	"\nhttps://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html");
	else
	throw Error("unable to start any build; remote machines may not have "
	"all required system features."
	"\nhttps://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html");
	}
	assert(!awake.empty());
	}
	}