Table of Contents
- ssh-uuid, scp-uuid
- Usage examples
- Remote command at the end of the command line, like standard
ssh
- Remote command execution with
--service
- Remote command execution preserves the command's exit status code
- Piping stdin/stdout/stderr works just like standard
ssh
- File transfer with
scp
- File transfer with
cat
- File transfer with
tar
- File transfer with
rsync
- Port forwarding and SOCKS proxy with ssh's
-D
,-L
and-R
options
- Remote command at the end of the command line, like standard
- Installation
- Authentication
- Troubleshooting
- Why?
This project is a proof of concept implementation of the "thinnest useful wrapper" around
the standard ssh
and scp
tools to allow them to connect to the ssh
server of a
remote balenaOS device, using the balenaCloud backend as a "raw bytestream pipe" between
the ssh client and the ssh server.
The thin wrapper does a light-touch editing of the ssh
or scp
command line options
provided by the user, before passing them to the actual ssh
or scp
tools. It
automatically adds a '-o ProxyCommand=…'
option that tunnels the connection through the
balenaCloud backend. It also adds a --service
command line option that allows specifying
the name of a balena fleet service (application container running in balenaOS) in order to
target a service instead of the host OS.
With this approach, all standard ssh
and scp
command line options, standard
environment variables and standard configuration files are supported, plus remote command
execution as provided by standard ssh
-- preserving the remote command exit status code
and allowing the plumbing of stdin/stdout/stderr.
This project is also an alternative vision of what the balena ssh
command could be.
For more background information, check the Why section.
The device's ssh/scp hostname has the format '<device-UUID>.balena'
, using the device's
full UUID (not a shortened form).
The --service
option is used to target a balena fleet service name. When --service
is
used, behind the scenes a balena-engine exec
command takes care of executing the remote
command in the service container (similar to balena ssh
), but the command syntax and
argument escaping are identical to standard ssh
.
$ ssh-uuid a123456abcdef123456abcdef1234567.balena cat /etc/issue
balenaOS 2.85.2 \n \l
$ ssh-uuid --service my-service a123456abcdef123456abcdef1234567.balena cat /etc/issue
Debian GNU/Linux 11 \n \l
$ ssh-uuid --service my-service a123456abcdef123456abcdef1234567.balena true; echo $?
0
$ ssh-uuid --service my-service a123456abcdef123456abcdef1234567.balena false; echo $?
1
... because it is standard ssh
!
$ cat local.txt | ssh-uuid --service my-service a123456abcdef123456abcdef1234567.balena cat '>' /tmp/local-copy.txt
$ ssh-uuid --service my-service a123456abcdef123456abcdef1234567.balena cat /tmp/remote.txt > remote-copy.txt
Note how '>'
was quoted in the first command line above, so that the stdout redirection
is interpreted by the remote shell rather than the local shell (standard ssh
syntax). In
the second command line, '>'
is not quoted because we actually want the local shell to
interpret it as stdout redirection.
Host OS:
# local -> remote
$ scp-uuid local.txt a123456abcdef123456abcdef1234567.balena:/mnt/data/
local.txt 100% 6 0.0KB/s 00:00
# remote -> local
$ scp-uuid a123456abcdef123456abcdef1234567.balena:/tmp/remote.txt .
remote.txt 100% 17 0.0KB/s 00:01
Service container:
# local -> remote
$ scp-uuid --service my-service local.txt a123456abcdef123456abcdef1234567.balena:
local.txt 100% 6 0.0KB/s 00:00
# remote -> local
$ scp-uuid --service my-service a123456abcdef123456abcdef1234567.balena:remote.txt .
remote.txt 100% 17 0.0KB/s 00:01
All scp
command line options are supported. The -r
option is used to copy a whole
folder:
# local -> remote
$ scp-uuid -r local-folder a123456abcdef123456abcdef1234567.balena:/mnt/data/
local.txt 100% 6 0.0KB/s 00:00
# remote -> local
$ scp-uuid -r a123456abcdef123456abcdef1234567.balena:/mnt/data/remote-folder .
remote.txt 100% 17 0.0KB/s 00:01
In order to use the --service
option, the scp
client program (not a SSH server!) must
be installed in the remote service container (as well as the local workstation), as
follows:
$ apt-get install -y openssh-client # Debian, Ubuntu, etc
$ apk add openssh # Alpine
If you would rather not install scp
in the service container, check the following
sections for file copy with cat
, tar
or rsync
. Also, if you only need transfer files
to/from a service's named volume (often at the '/data' mount point in service containers),
it is possible to scp to/from named volumes without using --service
, by directly
targeting the host OS folder that holds named volumes such as
'/mnt/data/docker/volumes/<fleet-id>_data/_data/'
:
# local -> remote
$ scp-uuid -r local-folder UUID.balena:/mnt/data/docker/volumes/<fleet-id>_data/_data/
# remote -> local
$ scp-uuid -r UUID.balena:/mnt/data/docker/volumes/<fleet-id>_data/_data/remote-folder .
# local -> remote
$ cat local.txt | ssh-uuid --service my-service UUID.balena cat \> /data/remote.txt
# remote -> local
$ ssh-uuid --service my-service UUID.balena cat /data/remote.txt > local.txt
# local -> remote
$ tar cz local-folder | ssh-uuid --service my-service UUID.balena tar xzvC /data/
# remote -> local
$ ssh-uuid --service my-service UUID.balena tar czC /data remote-folder | tar xvz
tar
must be installed both on the local workstation and on the remote service container:
$ apt-get install -y tar # Debian, Ubuntu, etc
$ apk add tar # Alpine
# local -> remote
$ rsync -av -e 'ssh-uuid --service my-service' local-folder UUID.balena:/data/
# remote -> local
$ rsync -av -e 'ssh-uuid --service my-service' UUID.balena:/data/remote-folder .
rsync
must be installed both on the local workstation and on the remote service
container:
$ apt-get install -y rsync # Debian, Ubuntu, etc
$ apk add rsync # Alpine
The wrapper does not interfere, and the full power of standard ssh is made available. This
is a full replacement for the balena tunnel
command, with additional capabilities such a
dynamic port forwarding (SOCKS proxy) and the benefit of the full range of ssh
configuration options and files.
The --service
option does not apply because port forwarding terminates in the scope of
the ssh
server, which is the host OS. To access service container ports, expose them to
the host OS through the docker-compose.yml
file.
$ ssh-uuid -NL 8000:127.0.0.1:80 a123456abcdef123456abcdef1234567.balena
Point the web browser at http://127.0.0.1:8000
Any port numbers lower than 1024 normally require administrator privileges (sudo
).
$ sudo ssh-uuid -NL 80:127.0.0.1:80 a123456abcdef123456abcdef1234567.balena
Point the web browser at http://127.0.0.1
This is a rudimentary replacement for the balena-proxy "hostvia" functionality. It may be
useful as a diagnostics operation if one of the remote devices gets in trouble and loses
access to balena's VPN service, but can still access its local network. Both remote
devices should be on the same local network, e.g. connected to the same WiFi access point.
For example, assuming two devices on the 192.168.1.xxx
subnet, run both the following
commands on two command prompt windows on your workstation:
ssh-uuid -NL 22222:192.168.1.86:22222 2eb94bd6ea9a3b9b4c0442aebf7bdb18.balena
where the UUID is for the good/online/gateway device, and192.168.1.86
is the IP address of the device in trouble (reportedly offline).ssh -p 22222 [email protected]
where the username is your balenaCloud account username ('root' may also work if the second device is running a development image of balenaOS).
Use the -R
option to expose a server running on your workstation or on the workstation's
local network to a remote device. For example, a local netcat chat server may be setup as
follows:
- Run a netcat server on your workstation (syntax may vary, check the
nc
manual page):
$ nc -l 1300
- Forward remote port 1300 to local port 1300:
ssh-uuid -tR 1300:127.0.0.1:1300 2eb94bd6ea9a3b9b4c0442aebf7bdb18.balena
- Run a netcat client on the remote device, balenaOS host OS prompt:
$ nc 127.0.0.1 1300
- Type words and hit Enter on each shell (local and remote).
Use the -D
option to run a SOCKS proxy server on the local workstation (standard ssh
functionality - dynamic port forwarding) and then configure the Firefox web browser (on
the workstation) to use it, thus using the remote device as an Internet "point of
presence". For example, if you are in Europe and the device is in the USA and you open
'whatsmyip.com' on the Firefox browser, you will be reported as being in the USA.
$ ssh-uuid -vND 8888 a123456abcdef123456abcdef1234567.balena
Configure Firefox (Network Settings, Connection Settings) as per screenshot below.
NOTE: This screenshot is just an advanced usage example for the
-D
option, which you do not have to use! You do not need to use Firefox or change any proxy settings in order to use thessh-uuid
orscp-uuid
tools.
Clone this repo and create a couple of soft links as follows:
$ git clone https://github.com/pdcastro/ssh-uuid.git
$ cd ssh-uuid
$ sudo ln -sf "${PWD}/ssh-uuid.sh" /usr/local/bin/ssh-uuid
$ sudo ln -sf "${PWD}/ssh-uuid.sh" /usr/local/bin/scp-uuid
$ which scp-uuid ssh-uuid
/usr/local/bin/scp-uuid
/usr/local/bin/ssh-uuid
The soft links (ln -s
) are important. Both ssh-uuid
and scp-uuid
are soft links
to the same ssh-uuid.sh
script. The script inspects how it was invoked at runtime
in order to decide what functionality to provide.
You will also need to install the dependencies below.
Follow the steps in your system-specific section below to install:
bash
v4.4 or latersocat
v1.7.4 or laterjq
,sed
,ssh
,scp
The balena CLI is not strictly required, but it can make authentication easier. See Authentication section.
Install Homebrew (https://brew.sh/), and then:
brew update && brew install bash git jq socat ssh
At the time of this writing, the latest stable distributions of Debian and Ubuntu provide an
outdated version of socat
with apt-get install
(they provide socat v1.7.3, but we need
socat v1.7.4 or later). Here's how to install socat
v1.7.4 (tested with Debian 9, Debian
10 and Ubuntu 20.04):
$ sudo apt-get update
$ sudo apt-get install -y curl git jq ssh build-essential libreadline-dev libssl-dev libwrap0-dev
$ curl -LO http://www.dest-unreach.org/socat/download/socat-1.7.4.2.tar.gz
$ tar xzf socat-1.7.4.2.tar.gz
$ cd socat-1.7.4.2
$ ./configure
$ make
$ sudo make install
$ which socat
/usr/local/bin/socat
If needed, more details about the compilation of socat can be found at:
http://www.dest-unreach.org/socat/doc/README
Native PowerShell or cmd.exe are not supported by this proof-of-concept implementation, but you can use Microsoft's WSL - Windows Subsystem for Linux (e.g. Ubuntu), and then follow the instructions for Linux.
Like balena ssh
, authentication involves both ssh public key authentication (for
the ssh server) and balenaCloud authentication (balenaCloud username and session token or
API key). See SSH Access
documentation.
Using the balena CLI is NOT a requirement for using ssh-uuid
or scp-uuid
.
However, if you happen to have the balena CLI installed, then all you need to do for
authentication is to log in with the balena CLI by running the following commands:
balena login
balena whoami
Running both commands will ensure that file ~/.balena/cachedUsername
exists and
contains a valid (not expired) session token. The file stores both a balenaCloud username
and a session token. The ssh-uuid
or scp-uuid
commands will check whether that file
exists (optional), to use the details in there for convenience.
Note: The
ssh-uuid
orscp-uuid
script does not check whether a session token has expired. Expired tokens will cause authentication errors. If the'balena whoami'
command succeeds, the token is good.
Alternatively, if you would rather not use the balena CLI for balenaCloud authentication, perhaps for non-interactive use, you can set the following two environment variables instead:
BALENA_USERNAME
: your balenaCloud usernameBALENA_TOKEN
: your balenaCloud session token or API key
Both the username and the session token (or API key) can be found in the balenaCloud web dashboard, Preferences page.
The authentication instructions above should be sufficient to get you started. The following section gets into more details, in case you are interested or for troubleshooting.
Two levels of authentication are involved:
-
balenaCloud username and session token (or API key): These are sent to the balenaCloud proxy backend (over HTTPS), which uses the details to ensure that the tunneling service is only provided to registered users, and for activity logging.
-
SSH username and keys: Used by the balenaOS ssh server (on the device) to authenticate the user against their public ssh key. The username may be
'root'
or a balenaCloud username, with different behaviors depending on whether the remote device is running a development or production image of balenaOS, as per table below.
Username | Production Image | Development Image |
---|---|---|
root | Requires adding a ssh public key to the sshKeys section of config.json | ⚠ Allows unauthenticated access |
balenaCloud username | ssh server authenticates against user's public SSH key stored in balenaCloud (requires balenaOS v2.44.0 or later) | ssh server authenticates against user's public SSH key stored in balenaCloud (requires balenaOS v2.44.0 or later) |
Reminder: public key authentication involves a pair of private and public keys:
- Both the private and public keys are stored in the user's workstation, typically in the
~/.ssh/
folder in the user's home directory on the machine wheressh
(orssh-uuid
) is executed. - A copy of the public key is additionally stored remotely, either on the machine where
the ssh server is running (e.g. the
config.json
file of a balenaOS device), or in the cloud (balenaCloud dashboard / API database).
- Set the DEBUG=1 environment variable to enable some debugging output produced by
the
ssh-uuid.sh
script itself. - Add the
-v
,-vv
or-vvv
command line option, e.g.:
DEBUG=1 ssh-uuid -v a123456abcdef123456abcdef1234567.balena cat /etc/issue
Common errors:
-
scp: not found
orscp: command not found
Typically indicates that thescp
program (client) is not installed in the remote service container. Thescp
program (client) must be installed both on the local workstation and in remote service containers. This can be easily done withapt-get install -y openssh-client
on Ubuntu or Debian, orapk add openssh
on Alpine. Note: A SSH server does NOT need to be installed in either the local workstation or remote service containers. The only SSH server involved is the one that runs by default on balenaOS (host OS). -
socat[1355] E parseopts(): unknown option "proxy-authorization-file"
This error means that your system is using an outdate version of 'socat'. Updatesocat
to version 1.7.4 or later as per Dependencies section. -
socat[25336] E CONNECT a123456abcdef123456abcdef1234567.balena:22222: Proxy Authorization Required
Double check that the authentication session or API token is correct and has not expired. See Authentication section. -
socat[25072] E CONNECT a123456abcdef123456abcdef1234567.balena:22222: Internal Server Error
Double check that the UUID is correct. Ensure you're using the full long UUID, not the 7-char short form.
Why are ssh-uuid
and scp-uuid
needed? What's wrong with the existing balena ssh
and
balena tunnel
implementations?
Currently, balena ssh
does not aim at command line compatibility with ssh
or scp
and
offers only a small subset of the "Swiss army knife" functionality provided by standard
ssh
and scp
. The manual page of ssh lists more than
40 command line options and tens of further configuration
options, most of which consist of functionality
not matched by balena ssh
.
The most commonly reported issues/requests are probably the lack of file copy
functionality (no balena scp
command exists), and the limited support for running remote
commands with balena ssh
while preserving command exit code and allowing plumbing of
stdin / stdout / stderr, as in the Usage Examples section.
balena ssh
and the balenaCloud proxy service offer their own "custom interface"
(incompatible with standard ssh) for remote command execution:
$ echo "uptime; exit;" | balena ssh 8dc8a1b
This custom interface takes a command line via stdin. It is problematic because:
- It does not allow plumbing stdin/stderr/stdout, as stdin is used to specify the remote commands.
- It does not preserve the remote command exit code (not event explicit exit codes such as
exit 3
). - Unlike standard
ssh
, it requires explicitly runningexit
in order to end the remote session, which is prone to "hanging session" bugs. - Being named
balena ssh
, it suggests the provision ofssh
's functionality, while being incompatible with basicssh
command line usage.
These limitations are fundamentally related to how the balenaCloud proxy currently sits as a "man in the middle" ssh server that provides its own API to clients and makes its own connections to the ssh server on devices.
The balena tunnel
command was created in part to work around these limitations, by
tunneling a TCP connection (raw bytestream) between a standard ssh
client and the ssh
server on a balenaOS device. However, to that end, balena tunnel
adds complexity and a
point of failure by requiring users to explicitly run two different processes in different
shell prompts: balena tunnel
in a prompt, and standard ssh
in another. It also falls
on the user to choose a free TCP port number to provide as argument to both processes.
Also, balena tunnel
offers only a small subset of features/options of standard ssh
,
and indeed balena tunnel
could be completely replaced with the standard port forwarding
options of standard ssh
(-L
, -R
and -D
options).
While balena ssh
offers a custom, incompatible interface to ssh
functionality, it
still uses the standard ssh
tool behind the scenes, and having the ssh
tool
pre-installed is already a requirement of the balena CLI. In the past, there was
discussion about removing the need of pre-installing ssh
by using a Javascript
implementation of ssh
such as the ssh2
npm module, but the motivation for that effort
was weakened when Microsoft Windows 10 started shipping with the ssh
tool pre-installed.
Nowadays, Apple, Microsoft and all Linux distributions offer a standard ssh
tool in
their operating systems.
The ssh-uuid
implementation provided in this project is able to replace both the balena ssh
and the balena tunnel
commands (and potentially, eventually, part of the
balena-proxy backend) with a single revamped balena ssh
command implemented as a thin
wrapper around ssh
. To end users, it offers a lot more functionality, full compatibility
with the ssh
and scp
command lines that many users are familiar with, and removes the
complexity of managing separate processes (balena tunnel
+ ssh
) in common usage
scenarios. To balena developers, ssh-uuid
promises fewer lines of code and a simpler
architecture by minimally wrapping the standard ssh
tool and avoiding the "man in the
middle" backend ssh server that terminates and initiates ssh
sessions on behalf of the
balena CLI.