Foo over SSH

Using SSH as a transport for your protocol



A comparison of the different solutions for using SSH2 as a secured transport for protocols/services/applications.

Table of content

SSH-2 Protocol


The SSH-2 protocol uses its Transport Layer Protocol to provide encryption, confidentiality, server authentication and integrity over a (potentially) unsafe reliable bidirectional data stream (usually TCP port 22):

The transport layer transports SSH packets. It handles:

Each packet starts with a message number and can belong to:

Typical protocol stack (assuming TCP/IP):

            [Session | Forwarding]
[SSH Authn. |SSH Connection      ]
[SSH Transport                   ]
[TCP                             ]
[IP                              ]

Connection Protocol

The Connection Protocol is used to manage channels and transfers data over them. Each channel is (roughly) a bidirectionnal data stream:

Multiple channels can be multiplexed over the same SSH connection:

C → S SSH CHANNEL_DATA(1, "whoami\n")
C → S SSH CHANNEL_DATA(2, "GET / HTTP/1.1\r\nHost: foo.example.com\r\n\r\n")
C ← S SSH CHANNEL_DATA(5, "root\n")
C ← S SSH CHANNEL_DATA(6, "HTTP/1.1 200 OK\r\nContent-Type:text/plain\r\n")
C ← S SSH CHANNEL_DATA(6, "Content-Length: 11\r\n\r\nHello World!")


Session Channel

A session channel is used to start:

For session channels, the protocol has support for setting environment variables, allocating a server-side TTY, enabling X11 forwarding, notifying of the terminal size modification (see SIGWINCH), sending signals, reporting the exit status or exit signal.

C → S SSH CHANNEL_OPEN("session", 2, …)
C → S SSH CHANNEL_REQUEST(6, "pty-req", TRUE, "xterm", 80, 120, …)
C → S SSH CHANNEL_REQUEST(6, "env", TRUE, "LANG", "fr_FR.utf8")
C → S SSH CHANNEL_REQUEST(6, "exec", TRUE, "ls /usr/")
C ← S SSH CHANNEL_DATA(3, "bin\ngames\ninclude\nlib\nlocal\sbin\nshare\nsrc\n")
C ← S SSH CHANNEL_REQUEST(3, "exit-status", FALSE, 0)


Shell session channels are used for interactive session are not really useful for protocol encapsulation.


In SSH, a command is a single string. This is not an array of strings (argv). On a UNIX-ish system, the command is usually expected to be called by the user's shell ("$SHELL" -c "$command"): variable expansions, globbing are applied by the server-side shell.

ssh foo.example.com 'ls *'
ssh foo.example.com 'echo $LANG'
ssh foo.example.com 'while true; do uptime ; sleep 60 ; done'


A subsystem is a “well-known” service running on top of SSH. It is identified by a string which makes it system independent: it does not depend on the user/system shell, environment (PATH), etc.

With the OpenSSH client, a subsystem can be invoked with ssh -S $subsystem_name.

Subsystem names come in two forms:

Well-known subsystem names include:

When using a subsystem:

With the OpenSSH server, a command can be associated with a given subsystem name with a configuration entry such as:

Subsystem sftp /usr/lib/openssh/sftp-server

The command is run under the identity of the user with its own shell ("$SHELL" -c "$command").

If you want to connect to a socket you might use:

Subsystem http socat STDIO TCP:localhost:80
Subsystem hello@example.com socat STDIO UNIX:/var/run/hello

It is possible to use exec to avoid keeping a shell process[6]:

Subsystem http exec socat STDIO TCP:localhost:80
Subsystem hello@example.com exec socat STDIO UNIX:/var/run/hello

This works but OpenSSH complains because it checks for the existence of an exec executable file.

Forwarding channels

TCP/IP Forwarding

The SSH has support for forwarding (either incoming or outgoing) TCP connections.

Local forwarding is used to forward a local connection (or any other local stream) to a remote TCP endpoint. A channel of type direct-tcpip is opened to initiate a TCP connection on the remote side. This is used by ssh -L, ssh -W and ssh -D

C → S SSH CHANNEL_OPEN("direct-tcpip", chan, …, "foo.example.com", 9000, "", 0);
C → S SSH CHANNEL_DATA(chan2, "aaa")

Remote forwarding is used to request to forward all incoming connections on a remote port over the SSH connection. The remote side then opens a new forwarded-tcpip channel for each connection. This is used by ssh -R.

C → S      SSH GLOBAL_REQUEST("tcpip-forward", remote_addr, remote_port)
C ← S      SSH REQUEST_SUCCESS(remote_port)
    S ← X  Incoming connection
C ← S      SSH CHANNEL_OPEN("forwarded-tcpip", chan, …, address, port, peer_address, peer_port)
C → S      SSH CHANNEL_OPEN_CONFIRMATION(chan, chan2, …)
    S ← X  TCP Payload "aaa"
S ← X      SSH CHANNEL_DATA(chan2, "aaa")

Unix socket forwarding

Since OpenSSH 6.7, it is possible to involve (either local or remote) UNIX sockets in forwards (ssh -L, ssh -R, ssh -W):

When the UNIX socket is on the client-side, client support is needed but server-side support is not needed.

When the UNIX socket is on the server-side, both client and server support is needed. This is using an (OpenSSH) protocol extension which works similarly to the TCP/IP forwarding:

TUN/TAP Forwarding

As an extension, OpenSSH has support for tunnel forwarding. A tunnel can be either Ethernet-based (TUN devices) or IP based (TAP devices). As channels do not preserve message boundaries, a header is prepended to each message (Ethernet frame or IP packet respectively): this header contains the message length (and for IP based tunnels, the address family i.e. IPv4 or IPv6).

This is used by ssh -w.

Messages for an IP tunnel:

C → S SSH CHANNEL_OPEN("tun@openssh.com", chan, …, POINTOPOINT, …)
C → S SSH CHANNEL_DATA(chan2, encapsulation + ip_packet)

and the packets use the form:

4B  packet length
4B  address family (SSH_TUN_AF_INET or SSH_TUN_AF_INET6)
var data

Messages for an Ethernet tunnel:

C → S SSH CHANNEL_OPEN("tun@openssh.com", chan, …, ETHERNET, …)
C → S SSH CHANNEL_DATA(chan2, encapsulation + ethernet_frame)

and the packets use the form:

4B  packet length
var data

X11 forwarding

The x11 channel type is used for X11 forwarding.

Examples of applications working over SSH


scp uses SSH to spawn a remote-side scp process. This remote scp process communicates with the local instance using its stdin and stdout.

When the local scp sends data, it spawns:

scp -t /some_path/

When the local scp receives data, it spawns:

scp -f /some_path/some_file


rsync can work over SSH. In this mode of operation, it uses SSH to spawn a server rsync process which communicates with its stdin and stdout.

The local rsync spawns something like in the remote side:

rsync --server -e.Lsfx . /some_path/


SFTP is a file transfer protocol. It is expected to to work on top of SSH using the sftp subsystem. However it can work on top of other streams (see sftp -S $program and sftp -D $program).

This is not FTP running over SSH.


FISH (Files transferred over Shell) is another solution for file system operation over a remote shell (such as rsh or ssh): it uses exec sessions to execute standard UNIX commands on the remote side in order to do the operations. This first approach will not work if the remote side is not a UNIXish system: in order to have support for non UNIX, it encodes the same requests as special comments at the beginning of the command.


Git spawns a remote git-upload-pack /some_repo/ which communicates with the local instance using its standard I/O.


Many systemd *ctl tools (hostnamectl, busctl, localectl, timedatectl, loginctl, systemctl) have builtin support for connecting to a remote host. They use a ssh -xT $user@$host systemd-stdio-bridge. This tool connects to the D-Bus system bus (i.e. ${DBUS_SYSTEM_BUS_ADDRESS:-/var/run/dbus/system_bus_socket}).


Program Solution
scp Command (scp)
rsync Command (rsync)
sftp Subsystem (sftp)
FISH Commands / special comments
git Command (git-upload-pack)
systemd Command (systemd-stdio-bridge)

Comparison of the different solutions for protocol transport

Which solution should be used to export your own protocol over SSH? The shell, X11 forwarding and TUN/TAP forwarding are not really relevant in this context so we are left with:


Using a dedicated subsystem is the cleaner solution. The subsystem feature of SSH has been designed for this kind of application: it is supposed to hide implementation details such as the shell, PATH, whether the service is exposed as a socket or a command, what is the location of the socket, whether socat is installed on the system, etc. However with OpenSSH, installing a new subsystem is done by adding a new entry in the /etc/ssh/sshd_config file which is not so convenient for packaging and not necessarily ideal for configuration management. An Include directive has been included for ssh_config (client configuration) in OpenSSH 7.3: the same directive for sshd_config would probably be useful in this context. In practice, the subsystem feature seems to be mostly used by sftp.

Using a command is the simpler solution: the only requirement is to add a suitable executable, preferably in the PATH. Moreover, the user can add their own commands (or override the system ones) for their own purpose by adding executables in its own PATH.

These two solutions have a few extra features which are not really necessary when used as a pure stream transport protocol but might be handy:

The two forwarding solutions have fewer features which are more in line with what is expected of a stream transport but:

Authentication and authorization

The command and subsystem solutions run code with the user's identity and will by default run with the user permissions. The setuid and setgid bits might be used if this is not suitable.

Another solution is to use socat or netcat to connect to a socket and get the same behavior as socket forwarding (security-wise).

For Unix socket forwarding, OpenSSH uses the user identity to connect to the socket. The daemon can use SO_PEERCRED (on Linux, OpenBSD), getpeereid() (on BSD), getpeerucred() (Solaris) to get the user UID, GID in order to avoid a second authentication. On Linux, file-system permissions can be used to restrict the access to the socket as well.

For TCP socket forwarding, OpenSSH uses the user identity to connect to the socket and ident (on localhost) might be used in order to get the user identity but this solution is not very pretty.


I kind-of like the subsystem feature even if it is not used that much.

The addition of an Include directive in sshd_config might help deploying such services. Another interesting feature would be an option to associate a subsystem with a Unix socket (without having to rely on socat).


  1. The random padding is used to make the whole Binary Packet Protocol message a multiple of the cipher block size (or 8 if the block size is smaller). ↩︎

  2. The receiver uses the SSH_MSG_CHANNEL_WINDOW_ADJUST message to request more data. ↩︎

  3. Each channel is associated with two integer IDs, one for each side of the connection. ↩︎

  4. This is used to transport both stdout (SSH_MSG_CHANNEL_DATA(channel, data)) and stderr (SSH_MSG_CHANNEL_EXTENDED_DATA(channel, SSH_EXTENDED_DATA_STDERR, data)) over the same session channel. ↩︎

  5. It is currently not yet registered but it is described in the SFTP drafts and widely deployed. ↩︎

  6. bash already does an implicit exec when bash -c "$a_single_command" is used. ↩︎