A comparison of the different solutions for using SSH2 as a secured transport for protocols/services/applications.
The SSH-2 protocol uses its Transport Layer Protocol to provide encryption, confidentiality, server authentication and integrity over a (potentially) unsafe reliable bidirectional data stream (usually TCP port 22):
first an encryption layer is used to create an encrypted stream over the underlying stream;
[ session | forwarding ] [ SSH: Transport | SSH: Authentication | SSH: Connection ] [ SSH: Binary Packet Protocol ] [ SSH: Encryption ] [ Underlying stream (eg. TCP) ]
The Connection Protocol is used to manage channels and transfer data over them. Each channel is (roughly) a bidirectionnal data stream:
they are reliable, flow-controled1, do not preserve message boundaries:
multiple channels can be multiplexed over the same SSH connection4;
a single channel can transport extended data streams (
SSH_MSG_CHANNEL_EXTENDED_DATA) in addition of the main data stream (
both client and server can request the creation of channels (
SSH_MSG_CHANNEL_OPEN) which may be accepted (
SSH_MSG_CHANNEL_OPEN_CONFIRMATION) or rejected (
SSH_MSG_CHANNEL_OPEN_FAILURE) by the other side;
the protocol has support for signaling the “end-of-file” when one side does not have anything to send anymore on the channel (
conversely, there is an OpenSSH extensions for signaling “end-of-write”, when one side is not willing to accept any data from the channel anymore.
Multiple channels can be multiplexed over the same SSH connection:
C → S CHANNEL_DATA(1, "whoami\n") C → S CHANNEL_DATA(2, "GET / HTTP/1.1\r\nHost: foo.example.com\r\n\r\n") C ← S CHANNEL_DATA(5, "root\n") C ← S CHANNEL_DATA(6, "HTTP/1.1 200 OK\r\nContent-Type:text/plain\r\n") C ← S CHANNEL_DATA(6, "Content-Length: 11\r\n\r\nHello World!")
session channel is used to start:
a shell session (
SSH_MSG_CHANNEL_REQUEST(chan, "shell", …);
or the execution of a (string) command (
SSH_MSG_CHANNEL_REQUEST(chan, "exec", …, command)) which is usually passed to the user shell;
or the execution of a subsystem (
SSH_MSG_CHANNEL_REQUEST(chan, "subsystem", …, subsystem_name)).
For session channels, the protocol has support for setting environment variables, allocating a server-side TTY, enabling X11 forwarding, notifying of the terminal size modification (see
SIGWINCH), sending signals, reporting the exit status or exit signal.
C → S CHANNEL_OPEN("session", 2, …) C ← S CHANNEL_OPEN_CONFIRMATION(3, 6) C → S CHANNEL_REQUEST(6, "pty-req", TRUE, "xterm", 80, 120, …) C ← S CHANNEL_SUCCESS(3) C → S CHANNEL_REQUEST(6, "env", TRUE, "LANG", "fr_FR.utf8") C ← S CHANNEL_SUCCESS(3) C → S CHANNEL_REQUEST(6, "exec", TRUE, "ls /usr/") C ← S CHANNEL_SUCCESS(3) C ← S CHANNEL_DATA(3, "bin\ngames\ninclude\nlib\nlocal\sbin\nshare\nsrc\n") C ← S CHANNEL_EOF(3) C ← S CHANNEL_REQUEST(3, "exit-status", FALSE, 0) C ← S CHANNEL_CLOSE(3) C → S CHANNEL_CLOSE(6)
Shell session channels are used for interactive session are not really useful for protocol encapsulation.
In SSH, a command is a single string. This is not an array of strings (
argv). On a UNIX-ish system, the command is usually expected to be called by the user's shell (
"$SHELL" -c "$command"): variable expansions, globbing are applied by the server-side shell.
ssh foo.example.com 'ls *' ssh foo.example.com 'echo $LANG' ssh foo.example.com 'while true; do uptime ; sleep 60 ; done'
A subsystem is a “well-known” service running on top of SSH. It is identified by a string which makes it system independent: it does not depend on the user/system shell, environment (
With the OpenSSH client, a subsystem can be invoked with
ssh -S $subsystem_name.
Subsystem names come in two forms:
unregistered subsystem names are expected to be in the form
other subsystem names are to be registered to the IANA and included in the SSH parameters registry.
Well-known subsystem names include:
sftpis used to connect a local SFTP client to a remote SFTP server5;
publickeyis used for the SSH Public Key Substem which can be used by clients to manage their SSH public keys;
snmpis used for SNMP over SSH;
netconffor NETCONF over SSH;
rpki-rtrfor RPKI-Router over SSH.
When using a subsystem:
both the OpenSSH client and paramiko do not seem to handle any stderr data: I don't think this is mandated by the protocol however;
return status is handled;
TTY allocation is possible;
environment variables can be passed.
With the OpenSSH server, a command can be associated with a given subsystem name with a configuration entry such as:
Subsystem sftp /usr/lib/openssh/sftp-server
The command is run under the identity of the user with its own shell (
"$SHELL" -c "$command").
If you want to connect to a socket you might use:
Subsystem http socat STDIO TCP:localhost:80 Subsystem firstname.lastname@example.org socat STDIO UNIX:/var/run/hello
It is possible to use
exec to avoid keeping a shell process6:
Subsystem http exec socat STDIO TCP:localhost:80 Subsystem email@example.com exec socat STDIO UNIX:/var/run/hello
This works but OpenSSH complains because it checks for the existence of an
exec executable file.
The SSH has support for forwarding (either incoming or outgoing) TCP connections.
Local forwarding is used to forward a local connection (or any other local stream) to a remote TCP endpoint. A channel of type
forwarded-tcpip is opened to initiate a TCP connection on the remote side. This is used by
ssh -W and
C → S CHANNEL_OPEN("direct-tcpip", chan, …, "foo.example.com", 9000, "", 0); C ← S CHANNEL_OPEN_CONFIRMATION(chan, chan2, …) C → S CHANNEL_DATA(chan2, "aaa")
Remote forwarding is used to request to forward all incoming connections on a remote port over the SSH connection. The remote side then opens a new
forwarded-tcpip channel for each connection. This is used by
C → S GLOBAL_REQUEST("tcpip-forward", remote_addr, remote_port) C ← S REQUEST_SUCCESS(remote_port) S ← X Incoming connection C ← S CHANNEL_OPEN("forwarded-tcpip", chan, …, address, port, peer_address, peer_port) C → S CHANNEL_OPEN_CONFIRMATION(chan, chan2, …) S ← X TCP Payload "aaa" S ← X CHANNEL_DATA(chan2, "aaa")
Unix socket forwarding
Since OpenSSH 6.7, it is possible to involve (either local or remote) UNIX sockets in forwards (
Client support is needed when the UNIX socket is on the client-side but server-side support is not needed.
When the UNIX socket is on the server-side, both client and server support is needed. This is using a protocol extension which works similarly to the TCP/IP forwarding:
SSH_MSG_CHANNEL_OPEN("firstname.lastname@example.org", …, path)for initiating a connection to a remote UNIX stream socket (local forwarding);
SSH2_MSG_GLOBAL_REQUEST("email@example.com", TRUE, path)is used to request a remote forwarding and each new connection opens a channel with
SSH_MSG_CHANNEL_OPEN("firstname.lastname@example.org", …, path, …).
As an extension, OpenSSH has support for tunnel forwarding. A tunnel can be either Ethernet-based (TUN devices) or IP based (TAP devices). As channels do not preserve message boundaries, a header is prepended to each message (Ethernet frame or IP packet respectively): this header contains the message length (and for IP based tunnels, the address family).
This is used by
Messages for an IP tunnel:
C → S CHANNEL_OPEN("email@example.com", chan, …, POINTOPOINT, …) C ← S CHANNEL_OPEN_CONFIRMATION(chan, chan2) C → S CHANNEL_DATA(chan2, encapsulation + ip_packet)
and the packets use the form:
4B packet length 4B address family (SSH_TUN_AF_INET or SSH_TUN_AF_INET6) var data
Messages for an Ethernet tunnel:
C → S CHANNEL_OPEN("firstname.lastname@example.org", chan, …, ETHERNET, …) C ← S CHANNEL_OPEN_CONFIRMATION(chan, chan2) C → S CHANNEL_DATA(chan2, encapsulation + ethernet_frame)
and the packets use the form:
4B packet length var data
x11 channel type is used for X11 forwarding.
Examples of applications working over SSH
scp uses SSH to spawn a remote-side
scp process. This remote
scp process communicates with the local instance using its
When the local
scp sends data, it spawns:
scp -t /some_path/
When the local
scp receives data, it spawns:
scp -f /some_path/some_file
rsync can work over SSH. In this mode of operation, it uses SSH to spawn a server rsync process which communicates with its
rsync spawns something like in the remote side:
rsync --server -e.Lsfx . /some_path/
SFTP is a file transfer protocol. It is expected to to work on top of SSH using the
sftp subsystem. However it can work on top of other streams (see
sftp -S $program and
sftp -D $program).
This is not FTP running over SSH.
FISH is another solution for file system operation over a remote shell (such as
ssh): it uses
exec sessions to execute standard UNIX commands on the remote side in order to do the operations. This first approach will not work if the remote side is not a UNIXish system: in order to have support for non UNIX, it encodes the same requests as special comments at the beginning of the command.
Git spawns a remote
git-upload-pack /some_repo/ which communicates with the local instance using its standard I/O.
*ctl tools (
systemctl) have builtin support for connecting to a remote host. They use a
ssh -xT $user@$host systemd-stdio-bridge. This tool connects to the D-Bus system bus (i.e.
| ||Command ( |
| ||Command ( |
| ||Subsystem ( |
|FISH||Commands / special comments|
| ||Command ( |
|systemd||Command ( |
Comparison of the different solutions for protocol transport
Which solution should be used to export your own protocol over SSH? The shell, X11 forwarding and TUN/TAP forwarding are not really relevant in this context so we're left with:
Unix socket forwarding.
Using a dedicated subsystem is the cleaner solution. The subsystem feature of SSH has been designed for this kind of application: it's supposed to hide implementation details such as the shell,
PATH, whether the service is exposed as a socket or a command, what is the location of the socket, whether
socat is installed on the system, etc. However with OpenSSH, installing a new subsystem is done by adding a new entry in the
/etc/ssh/sshd_config file which is not so convenient for packaging and not necessarily ideal for configuration management. An
Include directive has been included for
ssh_config (client configuration) in OpenSSH 7.3: the same directive for
sshd_config would probably be useful in this context. In practice, the subsystem feature seems to be mostly used by
Using a command is the simpler solution: the only requirement is to add a suitable executable, preferably in the
PATH. Moreover, the user can add his/her own commands (or override the system ones) for his/her own purpose by adding executables in its own
These two solutions have a few extra features which are not really necessary when used as a pure stream transport protocol but might be handy:
environment variables passing which can be used to get get i18n support (think HTTP
Accept-Language) for free with
terminal and terminal size support;
the command approach gives you a
stderrwhich is not really useful for encapsulating protocols.
The two forwarding solutions have fewer features which are more in line with what's expected of a stream transport but:
The Unix socket redirection is an OpenSSH extensions and may not be available in other SSH implementations.
The TCP socket requires the user to use TCP port number instead of a service name to access the service. This is OK if there's a well known port associated with the service but it's not as convenient as
email@example.com custom non-standard protocols.
Authentication and authorization
The command and subsystem solutions run code with the user's identity and will by default run with the user permissions. The
setgid bits might be used if this is not suitable.
Another solution is to use
socat or netcat to connect to a socket and get the same behavior as socket forwarding (security-wise).
For Unix socket forwarding, OpenSSH uses the user identity to connect to the socket. The daemon can use
SO_PEERCRED (on Linux, OpenBSD),
getpeereid() (on BSD),
getpeerucred() (Solaris) to get the user UID, GID in order to avoid a second authentication. On Linux, file-system permissions can be used to restrict the access to the socket as well.
For TCP socket forwarding, OpenSSH uses the user identity to connect to the socket and
ident (on localhost) might be used in order to get the user identity but this solution is not very pretty.
I kind-of like the subsystem feature even if it's not used that much.
The addition of an
Include directive in
sshd_config might help deploying such services. Another interesting feature would be an option to associate a subsystem with a Unix socket (without having to rely on
The SSH Protocol Architecture, RFC4251
The SSH Authentication Protocol, RFC4252
The SSH Transport Layer Protocol, RFC4253
The SSH Connection Protocol, RFC4254
The random padding is used to make the whole Binary Packet Protocol message a multiple of the cipher block size (or 8 if the block size is smaller). ↩
This is used to transport both
SSH_MSG_CHANNEL_DATA(channel, data)) and
SSH_MSG_CHANNEL_EXTENDED_DATA(channel, SSH_EXTENDED_DATA_STDERR, data)) over the same session channel. ↩
Each channel is associated with two integer IDs, one for each side of the connection. ↩
bashalready does an implicit
bash -c "$a_single_command"is used. ↩