Foo over SSH

A comparison of the different solutions for using SSH2 as a secured transport for protocols/services/applications.

Table of content

SSH-2 Protocol

Overview

The SSH-2 protocol uses its Transport Layer Protocol to provide encryption, confidentiality, server authentication and integrity over a (potentially) unsafe reliable bidirectional data stream (usually TCP port 22):

The transport layer transports SSH packets. It handles:

• MAC and encryption (MAC-and-encrypt)
• compression;
• random padding^[1];
• compression;
• packet framing;
• algorithm negotiations, key exchange, etc.

Each packet starts with a message number and can belong to:

• the Transport Layer Protocol itself (algorithm negotiation, key exchange, etc.);
• the user Authentication Protocol;
• the Connection Protocol) manages channels and transfers data over them.

Typical protocol stack (assuming TCP/IP):

            [Session | Forwarding]
[SSH Authn. |SSH Connection      ]
[SSH Transport                   ]
[TCP                             ]
[IP                              ]

Connection Protocol

The Connection Protocol is used to manage channels and transfers data over them. Each channel is (roughly) a bidirectionnal data stream:

• they are reliable, flow-controled^[2], do not preserve message boundaries:
• multiple channels can be multiplexed over the same SSH connection^[3];
• a single channel can transport extended data streams (SSH_MSG_CHANNEL_EXTENDED_DATA) in addition of the main data stream (SSH_MSG_CHANNEL_DATA) ^[4];
• both client and server can request the creation of channels (SSH_MSG_CHANNEL_OPEN) which may be accepted (SSH_MSG_CHANNEL_OPEN_CONFIRMATION) or rejected (SSH_MSG_CHANNEL_OPEN_FAILURE) by the other side;
• the protocol has support for signaling the “end-of-file” when one side does not have anything to send anymore on the channel (SSH_MSG_CHANNEL_EOF);
• conversely, there is an OpenSSH extensions for signaling “end-of-write”, when one side is not willing to accept any data from the channel anymore.

Multiple channels can be multiplexed over the same SSH connection:

C → S SSH CHANNEL_DATA(1, "whoami\n")
C → S SSH CHANNEL_DATA(2, "GET / HTTP/1.1\r\nHost: foo.example.com\r\n\r\n")
C ← S SSH CHANNEL_DATA(5, "root\n")
C ← S SSH CHANNEL_DATA(6, "HTTP/1.1 200 OK\r\nContent-Type:text/plain\r\n")
C ← S SSH CHANNEL_DATA(6, "Content-Length: 11\r\n\r\nHello World!")

Channels

Session Channel

A session channel is used to start:

• a shell session (SSH_MSG_CHANNEL_REQUEST(chan, "shell", …);
• or the execution of a (string) command (SSH_MSG_CHANNEL_REQUEST(chan, "exec", …, command)) which is usually passed to the user shell;
• or the execution of a subsystem (SSH_MSG_CHANNEL_REQUEST(chan, "subsystem", …, subsystem_name)).

For session channels, the protocol has support for setting environment variables, allocating a server-side TTY, enabling X11 forwarding, notifying of the terminal size modification (see SIGWINCH), sending signals, reporting the exit status or exit signal.

C → S SSH CHANNEL_OPEN("session", 2, …)
C ← S SSH CHANNEL_OPEN_CONFIRMATION(3, 6)
C → S SSH CHANNEL_REQUEST(6, "pty-req", TRUE, "xterm", 80, 120, …)
C ← S SSH CHANNEL_SUCCESS(3)
C → S SSH CHANNEL_REQUEST(6, "env", TRUE, "LANG", "fr_FR.utf8")
C ← S SSH CHANNEL_SUCCESS(3)
C → S SSH CHANNEL_REQUEST(6, "exec", TRUE, "ls /usr/")
C ← S SSH CHANNEL_SUCCESS(3)
C ← S SSH CHANNEL_DATA(3, "bin\ngames\ninclude\nlib\nlocal\sbin\nshare\nsrc\n")
C ← S SSH CHANNEL_EOF(3)
C ← S SSH CHANNEL_REQUEST(3, "exit-status", FALSE, 0)
C ← S SSH CHANNEL_CLOSE(3)
C → S SSH CHANNEL_CLOSE(6)

Shell

Shell session channels are used for interactive session are not really useful for protocol encapsulation.

Commands

In SSH, a command is a single string. This is not an array of strings (argv). On a UNIX-ish system, the command is usually expected to be called by the user's shell ("$SHELL" -c "$command"): variable expansions, globbing are applied by the server-side shell.

ssh foo.example.com 'ls *'
ssh foo.example.com 'echo $LANG'
ssh foo.example.com 'while true; do uptime ; sleep 60 ; done'

Subsystems

A subsystem is a “well-known” service running on top of SSH. It is identified by a string which makes it system independent: it does not depend on the user/system shell, environment (PATH), etc.

With the OpenSSH client, a subsystem can be invoked with ssh -S $subsystem_name.

Subsystem names come in two forms:

• unregistered subsystem names are expected to be in the form service_name@domain;
• other subsystem names are to be registered to the IANA and included in the SSH parameters registry.

Well-known subsystem names include:

• sftp is used to connect a local SFTP client to a remote SFTP server^[5];
• publickey is used for the SSH Public Key Substem which can be used by clients to manage their SSH public keys;
• snmp is used for SNMP over SSH;
• netconf for NETCONF over SSH;
• rpki-rtr for RPKI-Router over SSH.

When using a subsystem:

• both the OpenSSH client and paramiko do not seem to handle any stderr data: I don't think this is a requirements of the SSH protocol however;
• return status is handled;
• TTY allocation is possible;
• environment variables can be passed.

With the OpenSSH server, a command can be associated with a given subsystem name with a configuration entry such as:

Subsystem sftp /usr/lib/openssh/sftp-server

The command is run under the identity of the user with its own shell ("$SHELL" -c "$command").

If you want to connect to a socket you might use:

Subsystem http socat STDIO TCP:localhost:80
Subsystem hello@example.com socat STDIO UNIX:/var/run/hello

It is possible to use exec to avoid keeping a shell process^[6]:

Subsystem http exec socat STDIO TCP:localhost:80
Subsystem hello@example.com exec socat STDIO UNIX:/var/run/hello

This works but OpenSSH complains because it checks for the existence of an exec executable file.

Forwarding channels

TCP/IP Forwarding

The SSH has support for forwarding (either incoming or outgoing) TCP connections.

Local forwarding is used to forward a local connection (or any other local stream) to a remote TCP endpoint. A channel of type direct-tcpip is opened to initiate a TCP connection on the remote side. This is used by ssh -L, ssh -W and ssh -D

C → S SSH CHANNEL_OPEN("direct-tcpip", chan, …, "foo.example.com", 9000, "", 0);
C ← S SSH CHANNEL_OPEN_CONFIRMATION(chan, chan2, …)
C → S SSH CHANNEL_DATA(chan2, "aaa")

Remote forwarding is used to request to forward all incoming connections on a remote port over the SSH connection. The remote side then opens a new forwarded-tcpip channel for each connection. This is used by ssh -R.

C → S      SSH GLOBAL_REQUEST("tcpip-forward", remote_addr, remote_port)
C ← S      SSH REQUEST_SUCCESS(remote_port)
    S ← X  Incoming connection
C ← S      SSH CHANNEL_OPEN("forwarded-tcpip", chan, …, address, port, peer_address, peer_port)
C → S      SSH CHANNEL_OPEN_CONFIRMATION(chan, chan2, …)
    S ← X  TCP Payload "aaa"
S ← X      SSH CHANNEL_DATA(chan2, "aaa")

Unix socket forwarding

Since OpenSSH 6.7, it is possible to involve (either local or remote) UNIX sockets in forwards (ssh -L, ssh -R, ssh -W):

When the UNIX socket is on the client-side, client support is needed but server-side support is not needed.

When the UNIX socket is on the server-side, both client and server support is needed. This is using an (OpenSSH) protocol extension which works similarly to the TCP/IP forwarding:

• SSH_MSG_CHANNEL_OPEN("direct-streamlocal@openssh.com", …, path) for initiating a connection to a remote UNIX stream socket (local forwarding);
• SSH2_MSG_GLOBAL_REQUEST("streamlocal-forward@openssh.com", TRUE, path) is used to request a remote forwarding and each new connection opens a channel with SSH_MSG_CHANNEL_OPEN("forwarded-streamlocal@openssh.com", …, path, …).

TUN/TAP Forwarding

As an extension, OpenSSH has support for tunnel forwarding. A tunnel can be either Ethernet-based (TUN devices) or IP based (TAP devices). As channels do not preserve message boundaries, a header is prepended to each message (Ethernet frame or IP packet respectively): this header contains the message length (and for IP based tunnels, the address family i.e. IPv4 or IPv6).

This is used by ssh -w.

Messages for an IP tunnel:

C → S SSH CHANNEL_OPEN("tun@openssh.com", chan, …, POINTOPOINT, …)
C ← S SSH CHANNEL_OPEN_CONFIRMATION(chan, chan2)
C → S SSH CHANNEL_DATA(chan2, encapsulation + ip_packet)

and the packets use the form:

4B  packet length
4B  address family (SSH_TUN_AF_INET or SSH_TUN_AF_INET6)
var data

Messages for an Ethernet tunnel:

C → S SSH CHANNEL_OPEN("tun@openssh.com", chan, …, ETHERNET, …)
C ← S SSH CHANNEL_OPEN_CONFIRMATION(chan, chan2)
C → S SSH CHANNEL_DATA(chan2, encapsulation + ethernet_frame)

and the packets use the form:

4B  packet length
var data

X11 forwarding

The x11 channel type is used for X11 forwarding.

Examples of applications working over SSH

SCP

scp uses SSH to spawn a remote-side scp process. This remote scp process communicates with the local instance using its stdin and stdout.

When the local scp sends data, it spawns:

scp -t /some_path/

When the local scp receives data, it spawns:

scp -f /some_path/some_file

rsync

rsync can work over SSH. In this mode of operation, it uses SSH to spawn a server rsync process which communicates with its stdin and stdout.

The local rsync spawns something like in the remote side:

rsync --server -e.Lsfx . /some_path/

SFTP

SFTP is a file transfer protocol. It is expected to to work on top of SSH using the sftp subsystem. However it can work on top of other streams (see sftp -S $program and sftp -D $program).

This is not FTP running over SSH.

FISH

FISH (Files transferred over Shell) is another solution for file system operation over a remote shell (such as rsh or ssh): it uses exec sessions to execute standard UNIX commands on the remote side in order to do the operations. This first approach will not work if the remote side is not a UNIXish system: in order to have support for non UNIX, it encodes the same requests as special comments at the beginning of the command.

Git

Git spawns a remote git-upload-pack /some_repo/ which communicates with the local instance using its standard I/O.

Systemd

Many systemd *ctl tools (hostnamectl, busctl, localectl, timedatectl, loginctl, systemctl) have builtin support for connecting to a remote host. They use a ssh -xT $user@$host systemd-stdio-bridge. This tool connects to the D-Bus system bus (i.e. ${DBUS_SYSTEM_BUS_ADDRESS:-/var/run/dbus/system_bus_socket}).

Summary

Comparison of the different solutions for protocol transport

Which solution should be used to export your own protocol over SSH? The shell, X11 forwarding and TUN/TAP forwarding are not really relevant in this context so we are left with:

• command;
• subsystem;
• TCP forwarding;
• Unix socket forwarding.

Convenience

Using a dedicated subsystem is the cleaner solution. The subsystem feature of SSH has been designed for this kind of application: it is supposed to hide implementation details such as the shell, PATH, whether the service is exposed as a socket or a command, what is the location of the socket, whether socat is installed on the system, etc. However with OpenSSH, installing a new subsystem is done by adding a new entry in the /etc/ssh/sshd_config file which is not so convenient for packaging and not necessarily ideal for configuration management. An Include directive has been included for ssh_config (client configuration) in OpenSSH 7.3: the same directive for sshd_config would probably be useful in this context. In practice, the subsystem feature seems to be mostly used by sftp.

Using a command is the simpler solution: the only requirement is to add a suitable executable, preferably in the PATH. Moreover, the user can add their own commands (or override the system ones) for their own purpose by adding executables in its own PATH.

These two solutions have a few extra features which are not really necessary when used as a pure stream transport protocol but might be handy:

• environment variables passing which can be used to get get i18n support (think HTTP Accept-Language) for free with LANG, and LC_*;
• terminal and terminal size support;
• exit status;
• signal delivery.
• the command approach gives you a stderr which is not really useful for encapsulating protocols.

The two forwarding solutions have fewer features which are more in line with what is expected of a stream transport but:

• The Unix socket redirection is an OpenSSH extensions and may not be available in other SSH implementations.
• The TCP socket requires the user to use TCP port number instead of a service name to access the service. This is OK if there is a well known port associated with the service but it is not as convenient as myservice@example.com for custom non-standard protocols.

Authentication and authorization

The command and subsystem solutions run code with the user's identity and will by default run with the user permissions. The setuid and setgid bits might be used if this is not suitable.

Another solution is to use socat or netcat to connect to a socket and get the same behavior as socket forwarding (security-wise).

For Unix socket forwarding, OpenSSH uses the user identity to connect to the socket. The daemon can use SO_PEERCRED (on Linux, OpenBSD), getpeereid() (on BSD), getpeerucred() (Solaris) to get the user UID, GID in order to avoid a second authentication. On Linux, file-system permissions can be used to restrict the access to the socket as well.

For TCP socket forwarding, OpenSSH uses the user identity to connect to the socket and ident (on localhost) might be used in order to get the user identity but this solution is not very pretty.

Conclusion

I kind-of like the subsystem feature even if it is not used that much.

The addition of an Include directive in sshd_config might help deploying such services. Another interesting feature would be an option to associate a subsystem with a Unix socket (without having to rely on socat).

References

• SSH, overview of the remote secure shell protocol for secure remote sessions and port forwarding
• The SSH Protocol Architecture, RFC4251
• The SSH Authentication Protocol, RFC4252
• The SSH Transport Layer Protocol, RFC4253
• The SSH Connection Protocol, RFC4254