Foo over SSH

Using SSH as a transport for your protocol

A comparison of the different solutions for using SSH2 as a secured transport for protocols/services/applications.

SSH-2 Protocol

Overview

The SSH-2 protocol uses its Transport Layer Protocol to provide encryption, confidentiality, server authentication and integrity over a (potentially) unsafe reliable bidirectional data stream (usually TCP port 22):

Protocol stack:

                                       [ session | forwarding ]
[ SSH: Transport | SSH: Authentication | SSH: Connection      ]
[ SSH: Binary Packet Protocol                                 ]
[ SSH: Encryption                                             ]
[ Underlying stream (eg. TCP)                                 ]

Connection Protocol

The Connection Protocol is used to manage channels and transfer data over them. Each channel is (roughly) a bidirectionnal data stream:

  • they are reliable, flow-controled1, do not preserve message boundaries:

  • multiple channels can be multiplexed over the same SSH connection4;

  • a single channel can transport extended data streams (SSH_MSG_CHANNEL_EXTENDED_DATA) in addition of the main data stream (SSH_MSG_CHANNEL_DATA) 3;

  • both client and server can request the creation of channels (SSH_MSG_CHANNEL_OPEN) which may be accepted (SSH_MSG_CHANNEL_OPEN_CONFIRMATION) or rejected (SSH_MSG_CHANNEL_OPEN_FAILURE) by the other side;

  • the protocol has support for signaling the “end-of-file” when one side does not have anything to send anymore on the channel (SSH_MSG_CHANNEL_EOF);

  • conversely, there is an OpenSSH extensions for signaling “end-of-write”, when one side is not willing to accept any data from the channel anymore.

Multiple channels can be multiplexed over the same SSH connection:

C → S CHANNEL_DATA(1, "whoami\n")
C → S CHANNEL_DATA(2, "GET / HTTP/1.1\r\nHost: foo.example.com\r\n\r\n")
C ← S CHANNEL_DATA(5, "root\n")
C ← S CHANNEL_DATA(6, "HTTP/1.1 200 OK\r\nContent-Type:text/plain\r\n")
C ← S CHANNEL_DATA(6, "Content-Length: 11\r\n\r\nHello World!")

Channels

Session Channel

A session channel is used to start:

  • a shell session (SSH_MSG_CHANNEL_REQUEST(chan, "shell", …);

  • or the execution of a (string) command (SSH_MSG_CHANNEL_REQUEST(chan, "exec", …, command)) which is usually passed to the user shell;

  • or the execution of a subsystem (SSH_MSG_CHANNEL_REQUEST(chan, "subsystem", …, subsystem_name)).

For session channels, the protocol has support for setting environment variables, allocating a server-side TTY, enabling X11 forwarding, notifying of the terminal size modification (see SIGWINCH), sending signals, reporting the exit status or exit signal.

C → S CHANNEL_OPEN("session", 2, …)
C ← S CHANNEL_OPEN_CONFIRMATION(3, 6)
C → S CHANNEL_REQUEST(6, "pty-req", TRUE, "xterm", 80, 120, …)
C ← S CHANNEL_SUCCESS(3)
C → S CHANNEL_REQUEST(6, "env", TRUE, "LANG", "fr_FR.utf8")
C ← S CHANNEL_SUCCESS(3)
C → S CHANNEL_REQUEST(6, "exec", TRUE, "ls /usr/")
C ← S CHANNEL_SUCCESS(3)
C ← S CHANNEL_DATA(3, "bin\ngames\ninclude\nlib\nlocal\sbin\nshare\nsrc\n")
C ← S CHANNEL_EOF(3)
C ← S CHANNEL_REQUEST(3, "exit-status", FALSE, 0)
C ← S CHANNEL_CLOSE(3)
C → S CHANNEL_CLOSE(6)

Shell

Shell session channels are used for interactive session are not really useful for protocol encapsulation.

Commands

In SSH, a command is a single string. This is not an array of strings (argv). On a UNIX-ish system, the command is usually expected to be called by the user's shell ("$SHELL" -c "$command"): variable expansions, globbing are applied by the server-side shell.

ssh foo.example.com 'ls *'
ssh foo.example.com 'echo $LANG'
ssh foo.example.com 'while true; do uptime ; sleep 60 ; done'

Subsystems

A subsystem is a “well-known” service running on top of SSH. It is identified by a string which makes it system independent: it does not depend on the user/system shell, environment (PATH), etc.

With the OpenSSH client, a subsystem can be invoked with ssh -S $subsystem_name.

Subsystem names come in two forms:

  • unregistered subsystem names are expected to be in the form service_name@domain;

  • other subsystem names are to be registered to the IANA and included in the SSH parameters registry.

Well-known subsystem names include:

When using a subsystem:

  • both the OpenSSH client and paramiko do not seem to handle any stderr data: I don't think this is mandated by the protocol however;

  • return status is handled;

  • TTY allocation is possible;

  • environment variables can be passed.

With the OpenSSH server, a command can be associated with a given subsystem name with a configuration entry such as:

Subsystem sftp /usr/lib/openssh/sftp-server

The command is run under the identity of the user with its own shell ("$SHELL" -c "$command").

If you want to connect to a socket you might use:

Subsystem http socat STDIO TCP:localhost:80
Subsystem hello@example.com socat STDIO UNIX:/var/run/hello

It is possible to use exec to avoid keeping a shell process6:

Subsystem http exec socat STDIO TCP:localhost:80
Subsystem hello@example.com exec socat STDIO UNIX:/var/run/hello

This works but OpenSSH complains because it checks for the existence of an exec executable file.

Forwarding channels

TCP/IP Forwarding

The SSH has support for forwarding (either incoming or outgoing) TCP connections.

Local forwarding is used to forward a local connection (or any other local stream) to a remote TCP endpoint. A channel of type forwarded-tcpip is opened to initiate a TCP connection on the remote side. This is used by ssh -L, ssh -W and ssh -D

C → S CHANNEL_OPEN("direct-tcpip", chan, …, "foo.example.com", 9000, "", 0);
C ← S CHANNEL_OPEN_CONFIRMATION(chan, chan2, …)
C → S CHANNEL_DATA(chan2, "aaa")

Remote forwarding is used to request to forward all incoming connections on a remote port over the SSH connection. The remote side then opens a new forwarded-tcpip channel for each connection. This is used by ssh -R.

C → S      GLOBAL_REQUEST("tcpip-forward", remote_addr, remote_port)
C ← S      REQUEST_SUCCESS(remote_port)
    S ← X  Incoming connection
C ← S      CHANNEL_OPEN("forwarded-tcpip", chan, …, address, port, peer_address, peer_port)
C → S      CHANNEL_OPEN_CONFIRMATION(chan, chan2, …)
    S ← X  TCP Payload "aaa"
S ← X      CHANNEL_DATA(chan2, "aaa")

Unix socket forwarding

Since OpenSSH 6.7, it is possible to involve (either local or remote) UNIX sockets in forwards (ssh -L, ssh -R, ssh -W):

Client support is needed when the UNIX socket is on the client-side but server-side support is not needed.

When the UNIX socket is on the server-side, both client and server support is needed. This is using a protocol extension which works similarly to the TCP/IP forwarding:

  • SSH_MSG_CHANNEL_OPEN("direct-streamlocal@openssh.com", …, path) for initiating a connection to a remote UNIX stream socket (local forwarding);

  • SSH2_MSG_GLOBAL_REQUEST("streamlocal-forward@openssh.com", TRUE, path) is used to request a remote forwarding and each new connection opens a channel with SSH_MSG_CHANNEL_OPEN("forwarded-streamlocal@openssh.com", …, path, …).

TUN/TAP Forwarding

As an extension, OpenSSH has support for tunnel forwarding. A tunnel can be either Ethernet-based (TUN devices) or IP based (TAP devices). As channels do not preserve message boundaries, a header is prepended to each message (Ethernet frame or IP packet respectively): this header contains the message length (and for IP based tunnels, the address family).

This is used by ssh -w.

Messages for an IP tunnel:

C → S CHANNEL_OPEN("tun@openssh.com", chan, …, POINTOPOINT, …)
C ← S CHANNEL_OPEN_CONFIRMATION(chan, chan2)
C → S CHANNEL_DATA(chan2, encapsulation + ip_packet)

and the packets use the form:

4B  packet length
4B  address family (SSH_TUN_AF_INET or SSH_TUN_AF_INET6)
var data

Messages for an Ethernet tunnel:

C → S CHANNEL_OPEN("tun@openssh.com", chan, …, ETHERNET, …)
C ← S CHANNEL_OPEN_CONFIRMATION(chan, chan2)
C → S CHANNEL_DATA(chan2, encapsulation + ethernet_frame)

and the packets use the form:

4B  packet length
var data

X11 forwarding

The x11 channel type is used for X11 forwarding.

Examples of applications working over SSH

SCP

scp uses SSH to spawn a remote-side scp process. This remote scp process communicates with the local instance using its stdin and stdout.

When the local scp sends data, it spawns:

scp -t /some_path/

When the local scp receives data, it spawns:

scp -f /some_path/some_file

rsync

rsync can work over SSH. In this mode of operation, it uses SSH to spawn a server rsync process which communicates with its stdin and stdout.

The local rsync spawns something like in the remote side:

rsync --server -e.Lsfx . /some_path/

SFTP

SFTP is a file transfer protocol. It is expected to to work on top of SSH using the sftp subsystem. However it can work on top of other streams (see sftp -S $program and sftp -D $program).

This is not FTP running over SSH.

FISH

FISH is another solution for file system operation over a remote shell (such as rsh or ssh): it uses exec sessions to execute standard UNIX commands on the remote side in order to do the operations. This first approach will not work if the remote side is not a UNIXish system: in order to have support for non UNIX, it encodes the same requests as special comments at the beginning of the command.

Git

Git spawns a remote git-upload-pack /some_repo/ which communicates with the local instance using its standard I/O.

Systemd

Many systemd *ctl tools (hostnamectl, busctl, localectl, timedatectl, loginctl, systemctl) have builtin support for connecting to a remote host. They use a ssh -xT $user@$host systemd-stdio-bridge. This tool connects to the D-Bus system bus (i.e. ${DBUS_SYSTEM_BUS_ADDRESS:-/var/run/dbus/system_bus_socket}).

Summary

Program Solution
scp Command (scp)
rsync Command (rsync)
sftp Subsystem (sftp)
FISH Commands / special comments
git Command (git-upload-pack)
systemd Command (systemd-stdio-bridge)

Comparison of the different solutions for protocol transport

Which solution should be used to export your own protocol over SSH? The shell, X11 forwarding and TUN/TAP forwarding are not really relevant in this context so we're left with:

  • command;

  • subsystem;

  • TCP forwarding;

  • Unix socket forwarding.

Convenience

Using a dedicated subsystem is the cleaner solution. The subsystem feature of SSH has been designed for this kind of application: it's supposed to hide implementation details such as the shell, PATH, whether the service is exposed as a socket or a command, what is the location of the socket, whether socat is installed on the system, etc. However with OpenSSH, installing a new subsystem is done by adding a new entry in the /etc/ssh/sshd_config file which is not so convenient for packaging and not necessarily ideal for configuration management. An Include directive has been included for ssh_config (client configuration) in OpenSSH 7.3: the same directive for sshd_config would probably be useful in this context. In practice, the subsystem feature seems to be mostly used by sftp.

Using a command is the simpler solution: the only requirement is to add a suitable executable, preferably in the PATH. Moreover, the user can add his/her own commands (or override the system ones) for his/her own purpose by adding executables in its own PATH.

These two solutions have a few extra features which are not really necessary when used as a pure stream transport protocol but might be handy:

  • environment variables passing which can be used to get get i18n support (think HTTP Accept-Language) for free with LANG, and LC_*;

  • terminal and terminal size support;

  • exit status;

  • signal delivery.

  • the command approach gives you a stderr which is not really useful for encapsulating protocols.

The two forwarding solutions have fewer features which are more in line with what's expected of a stream transport but:

  • The Unix socket redirection is an OpenSSH extensions and may not be available in other SSH implementations.

  • The TCP socket requires the user to use TCP port number instead of a service name to access the service. This is OK if there's a well known port associated with the service but it's not as convenient as myservice@example.com for custom non-standard protocols.

Authentication and authorization

The command and subsystem solutions run code with the user's identity and will by default run with the user permissions. The setuid and setgid bits might be used if this is not suitable.

Another solution is to use socat or netcat to connect to a socket and get the same behavior as socket forwarding (security-wise).

For Unix socket forwarding, OpenSSH uses the user identity to connect to the socket. The daemon can use SO_PEERCRED (on Linux, OpenBSD), getpeereid() (on BSD), getpeerucred() (Solaris) to get the user UID, GID in order to avoid a second authentication. On Linux, file-system permissions can be used to restrict the access to the socket as well.

For TCP socket forwarding, OpenSSH uses the user identity to connect to the socket and ident (on localhost) might be used in order to get the user identity but this solution is not very pretty.

Conclusion

I kind-of like the subsystem feature even if it's not used that much.

The addition of an Include directive in sshd_config might help deploying such services. Another interesting feature would be an option to associate a subsystem with a Unix socket (without having to rely on socat).

References


  1. The receiver uses the SSH_MSG_CHANNEL_WINDOW_ADJUST message to request more data. 

  2. The random padding is used to make the whole Binary Packet Protocol message a multiple of the cipher block size (or 8 if the block size is smaller). 

  3. This is used to transport both stdout (SSH_MSG_CHANNEL_DATA(channel, data)) and stderr (SSH_MSG_CHANNEL_EXTENDED_DATA(channel, SSH_EXTENDED_DATA_STDERR, data)) over the same session channel. 

  4. Each channel is associated with two integer IDs, one for each side of the connection. 

  5. It is currently not yet registered but it is described in the SFTP drafts and widely deployed. 

  6. bash already does an implicit exec when bash -c "$a_single_command" is used.