/dev/posts/

OpenSSH tunneling guide

“Welcome to the Warp Zone!”

Published:

Updated:

The OpenSSH client has a lot of very powerful features for tunneling applications through a SSH connections and is one of my favorite tools for quick-and-dirty network plumbing tasks. It can be very useful for fixing/bypassing connectivity issues (caused by NATs, firewalls), accessing internal networks. This post is an overview of the different tunneling options available in OpenSSH. This is inteded as a reference to use when I am explaining (every so often) how to use SSH for tunneling.

Prerequisites (not explained here):

Table of content

Overview

The main techniques we will see are:

More exotic techniques (more rarely used, at least in my experience) are:

TLDR

For the impatient the main forwarding commands are summarized here.

Summary of some OpenSSH tunneling features

The SSH-based VPNs have been omitted.

Port forwarding

Local port forwarding

Summary: we want to access a remote (TCP-based) service (either be a service running on the same machine as the SSH server or accessible from the SSH server) from the local machine through a SSH tunnel. This can be:

Let's say we want to talk to a remote MySQL server which only listens on localhost. We can forward a local port (eg. 8306) to the remote MySQL port (3306) with either of:

ssh mysql.example.com -o"ExitOnForwardFailure true" -L localhost:8306:localhost:3306 -N

We can now connect to the remote MySQL server by connecting to the local port (8306):

mysql -h 127.0.0.1 -P 8306

The MySQL client is actually connecting to the local machine:

  1. the local MySQL client is connecting to the local SSH client (ssh) on port 8306;
  2. the SSH client forwards the data inside the SSH tunnel to the SSH server;
  3. the SSH server then sends the data to the remote MySQL server (port 3306).
Diagram for OpenSSH local fowarding (localhost)

Note both the application client and the application server both believe they are communicating with a local process.

       TCP/8306                   TCP/3306  
          ↓                          ↓
[App. ]-->|------------------------->[App. ]
                [SSH]==>[SSH ]
[TCP  ]-->[TCP  |TCP]-->[TCP |TCP]-->[TCP  ]
[IP   ]---[     IP  ]---[    IP  ]---[IP   ]
Client    SSH Client    SSH Server   Server
Protocol stack for SSH local port forwarding

Note: protocol stack diagrams

In this post, the direction of the arrows in the protocol stack diagrams represents the direction of the connections and/or the client/server relationships.

Note: communicating with a web servers over SSH

For communicating to web servers over SSH, it is usually better to use a SOCKS proxy (-D) instead (explained below).

When using local port forwarding, the HTTP client thinks it is communicating with the machine of the SSH client. This does not work very well with virtual hosts and TLS because host name (and/or port) used/expected by the HTTP client will not match with the one used/expected by the target HTTP server.

Advanced usage: local port forwarding to another machine

ssh relay.example.com -o"ExitOnForwardFailure true" -L localhost:8306:mysql.example.com:3306 -N

Warning: The communications are not protected by SSH between the remote SSH server (relay.example.com) and the remote targer server (mysql.example.com).

Advanced usage: local port forwarding accessible to other machines

By default, the SSH client only listens on localhost and is thus only accessible from the local machine.

You can bind to all available addresses in order exposes the tunnel to other machines:

ssh relay.example.com -o"ExitOnForwardFailure true" -L [::]:8306:mysql.example.com:3306 \
    -L 0.0.0.0:8306:mysql.example.com:3306 -N

Warning: the communications are not protected by SSH between the client hosts and the local tunnel endpoint.

Warning: the tunnel will be available to every machine which is able to connect to it. This might be everyone on your LAN but this might be everyone on the internet.

Advanced usage: avoiding SSH multiplexing

Inside the tunnel, all the connections are multiplexed over the same SSH connection and the same TCP connection so they might suffer from head-of-line blocking. If you want all connections to use a different SSH connection, you could use something like:

socat TCP-LISTEN:8306,bind=127.0.0.1,fork EXEC:"ssh mysql.example.com -W 127.0.0.1\:80"

I do not really recommend doing this! However, I found this useful one or twice (in particular when using a SSH server which did not allow multiplexing).

Diagram for OpenSSH local fowarding (general case)

Note that the data is protected by the SSH tunnel between the SSH client and the SSH server (denoted by the red arrow) but it is left unprotected between the application client and the SSH client and between the SSH server and the application server. This is not much of a problem when they are on the same machine but can be problematic otherwise.

Remote port forwarding

Summary: I want to serve a local (TCP-based) service (either on the my machine or accessible from my machine) to a remote network through a SSH tunnel.

You can expose a web application running on your machine (eg. on port 8080) to a remote machine (eg. on port 8080) (for example a development web application running on your machine that you want to expose to a remote service):

ssh remote.example.com -o"ExitOnForwardFailure true" -R localhost:8080:localhost:8080 -N

Where your local instance was launched for example using:

flask run --port 8080

Your service is now accessible on the remote machine as:

http://localhost:8080/

The remote HTTP client actually connects to the remote a port bound by the remote SSH server:

  1. the HTTP client connects to the remote SSH server;
  2. the SSH server forwards the data inside the SSH tunnel to the SSH client;
  3. the SSH client then sends the data to the HTTP server.
Diagram for OpenSSH local fowarding (localhost)
  TCP/8080                  TCP/8080
     ↓                          ↓
[App.]<-------------------------|<--[App.]
               [SSH]==>[SSH ]
[TCP ]<--[TCP  |TCP]-->[TCP |TCP]<--[TCP ]
[IP  ]---[     IP  ]---[    IP  ]---[IP  ]
Server   SSH Client    SSH Server   Client
Protocol stack for SSH remote port forwarding

Advanced usage: remote port forwarding to another machine

If the target server is not on your local machine:

ssh relay.example.com -o"ExitOnForwardFailure true" -R localhost:8080:taget.example.com:8080 -N

Warning: The communications are not protected by SSH between the local machine and the target machine (taget.example.com).

Advanced usage: exposing the tunnel to other machines

You can expose the tunnel port to other machines by binding to any address:

ssh remote.example.com -o"ExitOnForwardFailure true" \
    -R [::]:8080:localhost:8080 \
    -R 0.0.0.0:8080:localhost:8080 -N

Your service is now accessible on the remote network as:

http://relay.example.com:8080/

By default, the OpenSSH server won't let us listen on something else than localhost. In order to fix that, the GatewayPorts directive is needed in the OpenSSH server configuration:

GatewayPorts clientspecified

Warning: The communications are not protected by SSH between the machines from the remote end of the tunnel and the SSH server.

Diagram for OpenSSH local fowarding (general case)

Dynamic port forwarding

Motivation: Static (local, remote) port forwarding is nice when you have one or few endpoints that you want to tunnel. Even then, you have to choose on which ports your tunnel is going to listen and call the clients accordingly. Dynamic port forwarding lets you transparently route the TCP traffic of an application through a SSH connection. This is for example useful when you want to use a web browser through a SSH tunnel (for example in order to access a web service which is only available on the network of the SSH server) or when you want to make pip install, npm install over your SSH tunnel.

Dynamic port forwarding (-D) is a feature of the OpenSSH client. This creates a local SOCKS server which forwards TCP connection request inside the SSH tunnel. Using this method, any application which support SOCKS proxy can transparently use a SSH tunnel.

The OpenSSH client implements a SOCKS proxy with:

ssh relay.example.com -o"ExitOnForwardFailure true" -D localhost:1080 -N
Diagram for OpenSSH dynamic fowarding (localhost)

In contrast to static local and remote forwardings, here the client application is aware it is usuing a proxy to talk to a remote machine. The target application server is still unaware that a tunnel is used.

Many applications can be configured to use a SOCKS proxy through environnment variables (see appendix for details on how to use a SOCKS proxy for different applications):

export https_proxy=socks5h://localhost:1080
export http_proxy=socks5h://localhost:1080
       TCP/1080
          ↓
[App. ]----------------------------->[App.]
[SOCKS]-->[SOCKS|SSH]==>[SSH ]
[TCP  ]-->[TCP  |TCP]-->[TCP |TCP]<->[TCP ]
[IP   ]---[IP       ]---[IP      ]---[IP  ]
Client    SSH Client    SSH Server   Server

[App. ]<-----------------------------[App.]
                [SSH]==>[SSH ]
[TCP  ]<--[TCP  |TCP]-->[TCP |TCP]<--[TCP ]
[IP   ]---[IP       ]---[IP      ]---[IP  ]
Client    SSH Client    SSH Server   Server
Protocol stacks for SSH-based SOCKS proxy

Advanced usage: Exposing the SOCKS service

As before you can expose, the SOCKS port to other machines:

ssh relay.example.com -o"ExitOnForwardFailure true" -N \
    -D [::1]:1080 \
    -D 0.0.0.0:1080
Diagram for OpenSSH dynamic fowarding (general case)

Advanced usage: SOCKS proxy with Unix domain socket (UDS)

It is possible to have the SOCKS proxy listen on a Unix domain socket:

ssh relay.example.com -o"ExitOnForwardFailure true" -D "$XDG_RUNTIME_DIR/ssh.socks" -N

On Linux, using a Unix domain socket instead of a TCP socket for the SOCKS service is interesting because it may be used to prevent other local users from accessing the SOCKS service (as explained below).

Firefox supports talking to a SOCKS proxy using a Unix domain socket (see appendix for details). This is not supported by most other clients (such as Chromium-based browsers).

       UDP (…/ssh.socks)
          ↓
[App. ]----------------------------->[App.]
[SOCKS]-->[SOCKS|SSH]==>[SSH ]
                [TCP]-->[TCP |TCP]<->[TCP ]
[UDS  ]---[UDS  |IP ]---[IP      ]---[IP  ]
Client    SSH Client    SSH Server   Server
Protocol stack for SSH-based SOCKS proxy using Unix domain socket

Note: implementation details

The SOCKS proxy is implemented by the SSH client without any special support (no extension) needed from the SSH server. The SSH client translates the SOCKS CONNECT or BIND requests into SSH local or remote port forwarding commands:

Client  SSH client   SSH     Target
    |   (ssh -D)    Server   Server
    |------>|          |        |  SOCKS CONNECT("www.example.com", 80)
    |       |--------->|        |  SSH MSG_CHANNEL_OPEN("direct-tcpip", "www.example.com", 80)
    |       |          |------->|  TCP SYN
    |       |          |<-------|  TCP SYN+ACK
    |       |          |------->|  TCP ACK
    |       |<·········|        |  SSH CHANNEL_OPEN_CONFIRMATION
    |<······|          |        |  SOCKS succeeded
    |       |          |        |
    |-------O==========O------->|  GET / HTTP/1.1
    |<······O==========O········|  HTTP/1.1 200 OK

Unix domain socket forwarding

Since OpenSSH 6.7, it is possible to use a Unix (AF_LOCAL, SOCK_STREAM) socket at either (or both) end of the tunnel. This works both for static port forwarding (-L, -R, -W) as well as for dynamic port forwarding (-D).

To forward towards a remote MySQL Unix domain socket:

ssh mysql.example.com -o"ExitOnForwardFailure true" \
    -L localhost:8306:/var/run/mysqld/mysqld.sock -N &
mysql -h 127.0.0.1 -P 8306
Diagram for OpenSSH local fowarding with Unix domain sockets

In this case, Unix domain sockets are used in both side of the tunnel but you can mix and match TCP and UDS.

       TCP/8306                   UDS (…/mysqld.sock)
          ↓                           ↓
[App. ]-->|-------------------------->[App. ]
                [SSH]==>[SSH ]
[TCP  ]-->[TCP  |TCP]-->[TCP ]
[IP   ]---[     IP  ]---[    |UDS]<-->[UDS]
Client    SSH Client    SSH Server   Server
Protocol stack for SSH local port forwarding to a remote Unix domain socket

To forward from a local Unix domain socket to a remote TCP port:

ssh mysql.example.com -o"ExitOnForwardFailure true" \
    -L "${XDG_RUNTIME_DIR}/mysql-tunnel.sock:localhost:3306" -N &
mysql -h localhost -S "${XDG_RUNTIME_DIR}/mysql-tunnel.sock"

To forward from a local Unix domain socket to a remote Unix domain socket:

ssh mysql.example.com -o"ExitOnForwardFailure true" \
    -L "${XDG_RUNTIME_DIR}/mysql-tunnel.sock:/var/run/mysqld/mysqld.sock" -N &
mysql -h localhost -S "${XDG_RUNTIME_DIR}/mysql-tunnel.sock"

Advanced usage: relaxing access to the local Unix domain socket (Linux)

On Linux systems, the permissions on Unix domain socket are honored: by default, other system users won't be able to connect to the Unix domain socket created by OpenSSH. The StreamLocalBindMask option can be used to change the permissions on the socket and give other system users access to the tunnel. Another option is to change the permissions of the socket manually (using chmod, setfacl).

On Linux systems, I would suggest using a Unix domain socket instead of a TCP socket for your tunnels if possible.

The permissions on Unix domain sockets are not honored on some other OSes: all system users can use the tunnel.

Note: support for Unix domain socket protocol

For Unix domain socket on the remote side, both client and server support are needed because this relies on an OpenSSH specific extension of the SSH protocol.

For Unix domain socket on the local side, client support is needed and server support is not required as it does not rely on an OpenSSH protocol extension.

Tunneling SSH

SSH over SSH

SSH over SSH with ProxyJump

Often a SSH server is not directly reachable but we can/must use another SSH server as a relay to reach it. A common scenario is when many SSH servers are not directly reachable but can only be accessed through a given relay SSH server:

Diagram for OpenSSH over SSH (ProxyJump)
[SSH  ]=================>[SSH   ]
[SSH  ]====>[SSH ]
[TCP  ]---->[TCP ]------>[TCP   ]
[IP   ]-----[IP  ]-------[IP    ]
  SSH       SSH           SSH
client     server        server
          (relay)       (target)
Protocol stack for SSH over SSH

Note that when using SSH over SSH, the relay server does not see the content of the inner SSH traffic.

We could use local port forwarding (-L) for implenting SSH over SSH but this can be cumbersome: there are better (and simpler) ways of doing it. Since OpenSSH 7.3 the ProxyJump configuration option can be used to easily configure SSH over SSH.

The following OpenSSH configuration snippet (~/.ssh/config) says that OpenSSH must use the relay.example.com SSH server as a realy in order to reach target.example.com:

Host target.example.com
ProxyJump relay.example.com

With this configuration, we can connect to the target SSH server with:

ssh target.example.com

The -J flag can be used instead of the ProxyJump configuration:

ssh target.example.com -J relay.example.com

A typical SSH relay setup can be achieved with:

Host relay.example.com
  User john
  ProxyJump none

Host !relay.example.com *.example.com
  User john
  ProxyJump relay.example.com

Advanced usage: mutiple SSH relays

We can specify multiple jump proxies for SSH over SSH over SSH (etc.):

Host target.example.com
ProxyJump relay1.example.com,relay2.example.com

It is usually better to use something like:

Host target.example.com
  User john
  ProxyJump relay2.example.com

Host relay2.example.com
  User john
  ProxyJump relay1.example.com
[SSH  ]==============================>[SSH   ]
[SSH  ]=================>[SSH   ] 
[SSH  ]====>[SSH ]                             
[TCP  ]---->[TCP ]------>[TCP   ]---->[TCP   ] 
[IP   ]-----[IP  ]-------[IP    ]-----[IP    ] 
  SSH        SSH         SSH           SSH     
client      server       server        server  
           (relay1)     (relay2)      (target)
Protocol stack for SSH over SSH over SSH

Note: implementation details

Under the hood, ProxyJump is actually implemented by OpenSSH by spawning a second OpenSSH process using netcat mode (-W). The following command:

ssh target.example.com -J relay.example.com

calls a child process:

ssh -W target.example.com:22 relay.example.com

SSH over SSH with ProxyCommand

Before OpenSSH 7.3, the ProxyCommand configuration option could be used instead of ProxyJump. ProxyCommand let us define a system command which will be called to create the communication channel with the target SSH server: the standard input and standard output of this command will be used by the SSH client (instead of a direct TCP connection) to communicate with the SSH server.

We can combine ProxyCommand and a program such as netcat or socat (installed on the relay server) to get the same effect as ProxyJump. This can be useful if your OpenSH version does not support ProxyJump.

With netcat:

Host *.example.com
User john
ProxyCommand ssh relay.example.com netcat %h %p

where %h and %p are expanded by the SSH client into the target host name and port respectively.

With socat:

Host *.example.com
User john
ProxyCommand ssh relay.example.com socat STDIO TCP:%h:%p

Since OpenSSH client 5.4 has introduced at “netcat mode” (-W) which can be used instead of relying on an external program on the relay server:

Host *.example.com
User john
ProxyCommand ssh relay.example.com -W %h:%p

SSH over a proxy

SSH over SOCKS

You can use ProxyCommand to connect to a SSH server through a SOCKS proxy:

Host foo.example.com
ProxyCommand netcat -x socks.example.com %h %p

Since OpenSSH 6.4, the ProxyUseFdpass can be used to avoid keeping the netcat process around:

Host foo.example.com
ProxyCommand netcat -x socks.example.com -F %h %p
ProxyUseFdpass yes

This can be used for SSH over tor.

[SSH  ]======================>[SSH  ]
[SOCKS]------>[SOCKS]
[TCP  ]------>[TCP      ]---->[TCP  ]
[IP   ]-------[IP       ]-----[IP   ]
SSH Client    SOCKS proxy     SSH server
Protocol stack for SSH over a SOCKS proxy

SSH over HTTP proxy

netcat can use a HTTP proxy (using the HTTP CONNECT method) as well:

Host foo.example.com
ProxyCommand netcat -x socks.example.com -X connect -F %h %p
ProxyUseFdpass yes
[SSH  ]---------------------->[SSH  ]
[HTTP ]------>[HTTP]
[TCP  ]------>[TCP      ]---->[TCP  ]
[IP   ]-------[IP       ]-----[IP   ]
SSH Client    HTTP proxy     SSH server
Protocol stack for SSH over a HTTP proxy

SSH-based VPN

Using SOCKS and HTTP proxies have some limitations:

In this section, we'll see several approaches to run a simple VPN over SSH:

Note: using dynamic port forwarding

In many cases, you can use dynamic port forwarding instead of this. I recommend using dynamic port forwarding if you can.

Dynamic port forwarding is a simpler alternative to a real VPN in many cases: it does not require much configuration to setup and does not require root access.

OpenSSH TUN/TAP forwarding

OpenSSH has an extension for setting up a (TUN or TAP) VPN over a SSH connection. It can either be:

The extension is needed on both client and server-side. It is disabled by default in the OpenSSH server but can be enabled with:

Match User john
PermitTunnel yes

I would suggest sticking to a TUN VPN if you only care about IP connectivity: you will avoid the overhead of the additional Ethernet layer, (and ARP and NDP requests). You might want to use a TAP VPN if you want to bridge the VPN with a LAN.

Note: TCP over TCP

SSH is using TCP so you end up doing TCP over TCP which is not as nice as a VPN over UDP instead. If this is a problem, you might want to use a VPN which works over UDP. OpenVPN or WireGuard can do that for example.

There is a Network Manager plugin for setting up a SSH-based VPN (client-side).

TUN forwarding

[IP   ]---[IP ]---------[IP ]------[IP  ]
          [SSH]========>[SSH]
          [TCP]-------->[TCP]
          [IP ]---------[IP ]
Local    SSH Client    SSH Server  Remote
Program                            Program
Protocol stack for OpenSSH TUN forwarding

You need to be root (both on the SSH client side and on the SSH server side), to be able to create the (TUN) network interfaces. You can run the VPN as root (both on the client side and the server side):

sudo env env SSH_AGENT_PID="$SSH_AGENT_PID" SSH_AUTH_SOCK="$SSH_AUTH_SOCK" \
    ssh root@relay.example.com -w 0:0

If you don't want (or can't) run the VPN as root, you must create (as root) the virtual network interface manually beforehand and grant yourself the right to use them. The following command creates a persistent TUN device and gives $USER the ability to use it:

sudo ip tuntap add dev tun0 mode tun user "$USER"

This has to be done both on the client side and the server side.

You then need to configure IP addresses, routing tables and possibly enable IP forwarding, configure a NAT.

Example configuration script on the server-side:

#!/bin/sh
set -e

user=john
tun=tun0
local_ip=192.168.0.1
remote_ip=192.168.0.2
dev=eth0
gateway_ip=192.0.2.42

# Create and configure the TUN device:
ip tuntap add dev $tun mode tun user "$user"
ip address add "$local_ip" peer "$remote_ip" dev "$tun"

# NAT:
iptables -t nat -A POSTROUTING -s "$remote_ip" -o "$dev" -j SNAT --to "$gateway_ip"

# IP forwarding:
iptables -A FORWARD -i "$dev" -o "$tun" -j ACCEPT
iptables -A FORWARD -i "$tun" -o "$dev" -j ACCEPT
iptables -P FORWARD REJECT
sysctl -w net.ipv4.ip_forward=1

ip link set dev "$tun" up

Example configuration script on the client-side:

#!/bin/sh
set -e

user=john
tun=tun0
local_ip=192.168.0.2
remote_ip=192.168.0.1/32
dev=eth0
gateway_ip=192.0.2.42

ip tuntap add dev "$tun" user "$user" mode tun
ip address add "$local_ip" peer "$remote_ip" dev "$tun"

ip link set dev "$tun" up

# Keep the current routes for the:
# * SSH server;
# * for the private IP addresses.
for addr in "$gateway_ip" 172.16.0.0/12 10.0.0.0/8 ; do
    route="$(ip route get "$addr" | sed 's/^[^ ]*//' | head -n1)"
    ip route add "$addr" "$route"
done

# "Default route trick": this overrides the default route without touching it.
for addr in 0.0.0.0/1 128.0.0.0/1 ; do
    ip route add "$addr" dev "$tun"
done

Now, start the tunnel itself (without root):

ssh -o ServerAliveInterval=10 -o TCPKeepAlive=yes server.example.com -w 0:0 -N

TAP forwarding

The basic command for a TAP VPN is:

ssh relay.example.com -w 0:0 -o"Tunnel ethernet" o"ExitOnForwardFailure true" -N

As for TUN forwarding, you then need to configure IP addresses, routing tables and possibly enable IP forwarding, configure a NAT.

[Eth. ]---[Eth.]---------[Eth.]-----[Eth.]
          [SSH ]========>[SSH ]
          [TCP ]-------->[TCP ]
          [IP  ]---------[IP  ]
Local    SSH Client    SSH Server    Remote
Program                              Program
Protocol stack for OpenSSH TAP forwarding

I won't go into the details of a TAP forwarding. You probably want to bridge the virtual TAP network interface with a real interface.

TUN forwarding in a network namespace

You could want to only expose the VPN to some applications. On Linux systems, a solution is to you use network namespaces (eg. using ip netns).

First we create and configure a TUN interface on the server side:

ip tuntap add dev tun0 mode tun user $user
ip link set dev tun0 up
ip address add 10.1.1.1 peer 10.1.1.2 dev tun0

Client script:

#!/bin/sh

set -e

user=john
tun_no=0
remote_tun_no=0
tun=tun${tun_no}
local_ip=10.1.1.2
remote_ip=10.1.1.1/24
ns=vpn
pid=""
server=john@server.example.com

do_cleanup() {
    echo cleanup
    ip -n $ns link del $tun || true
    ip netns del $ns || true
    ip link del $tun || true
    kill -9 "$pid" || true
}
trap 'do_cleanup' INT QUIT TERM EXIT

ip tuntap add dev $tun mode tun user "$user"

sudo -u $user env SSH_AGENT_PID="$SSH_AGENT_PID" SSH_AUTH_SOCK="$SSH_AUTH_SOCK" \
    ssh "$server" -o"ExitOnForwardFailure true" -w "${tun_no}:${remote_tun_no}" -N &
pid=$!
sleep 2

ip netns add $ns
ip link set $tun netns $ns
ip -n $ns address add "$local_ip" peer "$remote_ip" dev "$tun"
ip -n $ns link set dev "$tun" up
ip -n $ns route add default dev "$tun"

wait

Used with:

sudo env SSH_AGENT_PID="$SSH_AGENT_PID" SSH_AUTH_SOCK="$SSH_AUTH_SOCK" ./runvpn

You can now get a shell in the network namespace:

ip netns exec $ns sudo -u $user -s

From this network namespace you can ping over your tunnel:

ping 10.1.1.1

PPP over SSH

If you cannot use TUN/TAP forwarding[1], you can setup a PPP-based VPN over SSH using by running pppd over SSH.

[IP   ]---[IP     ]------------------------------------[IP      ]---[IP  ]
             [PPP ]------------------------------------[PPP ]
             [PTY ]<-->[PTY |SSH]========>[SSH|PTY ]<->[PTY ]
                            [TCP]-------->[TCP]
                            [IP ]---------[IP ]
Local       Local      SSH Client    SSH Server         Remote      Remote
Program     pppd                                         pppd      Program
Protocol stack for PPP over SSH

First start a PPP daemon listening on a Unix domain socket on the remote SSH server:

ssh ssh.example.com -t \
    'sudo socat UNIX-LISTEN:/run/ssh-tunnel.pppd,user=$USER,mode=700 "SYSTEM:pppd nodetach noipdefault noauth nodeflate 10.1.1.1\:10.1.1.2,pty"'

where 10.1.1.1 and 10.1.1.2 are the private IP addresses that you are going to use on the tunnel for the SSH server side and for the SSH client side respectively.

You can now connect to a local PPP daemon to this socket:

sudo pppd nodetach noipdefault noauth nodeflate pty "
sudo -u $USER env SSH_AGENT_PID=$SSH_AGENT_PID SSH_AUTH_SOCK=$SSH_AUTH_SOCK \
  ssh odin.urdhr.fr -W /run/ssh-tunnel.pppd
"

You can now ping over your tunnel:

ping 10.1.1.1

As for TAP forwarding, you still probably have to configure IP forwarding, NAT and routes if you want the tunnel to be useful.

Other

Netcat mode

Since v5.4, the client OpenSSH supports a netcat mode (-W) which can be used as an alternative to netcat over SSH. The following commands are equivalent:

ssh server.example.com -W     target.example.com:8080
ssh server.example.com netcat target.example.com 8080

However the former, does not need any software to be installed on server.example.com.

Diagram for OpenSSH netcat mode
                      TCP/8080
                         ↓
[App.]------------------>[App. ]
[SSH ]=====>[SSH ]
[TCP ]----->[TCP |TCP]-->[TCP  ]
[IP  ]------[    IP  ]---[IP   ]
SSH Client   SSH Server   Server
Protocol stack for SSH netcat mode

This can be useful to use SSH over SSH without ProxyJump (before OpenSSH 7.3).

Conclusion

Other useful tools for your network plumbing toolkit are:

Appendix, things to avoid

Do not use SSH agent forwarding

TLDR: do not use agent forwarding if you can avoid it!

When combining multiple SSH servers, the recommended approach is to use SSH over SSH (ProxyCommand/ProxyJump).

Another solution is to use SSH agent forwarding (ssh -A):

# Don't use that if you can! See below.
ssh relay.example.com -t -A ssh target.example.com

This avoids the encapsulation of SSH over SSH as demonstrated in the following diagram.

[SSH     ]====>[SSH | SSH]======>[SSH   ]
[TCP     ]---->[   TCP   ]------>[TCP   ]
[IP      ]-----[    IP   ]-------[IP    ]
   SSH             SSH             SSH
 client          server           server
                 (relay)         (target)
Protocol stack when using SSH agent forwarding

SSH agent forwarding works by giving the relay machine the ability to use of your SSH private key: everyone having access to your account (or root) on the relay machine can use your SSH private key for whatever purpose. It is much safer to use SSH over SSH (ProxyCommand/ProxyJump) instead.

Note: OpenSSH agent restriction

This can be somewhat mitigated using OpenSSH agent restriction (available since OpenSSH 8.9).

Do not use X11 forwarding

X11 forwarding is quite dangerous. If the remote server is compromised, it may use the X11 forwarding to attack your local desktop.

Appendix, using a SOCKS proxy

In this appendix, we see some details about how to configure programs to use a SOCKS proxy.

Environment configuration

Some clients can be configured through the environment:

export http_proxy=socks5h://127.0.0.1:1080/
export https_proxy="$http_proxy"
export ftp_proxy="$http_proxy"
export rsync_proxy=$http_proxy
export all_proxy=$http_proxy
export no_proxy="localhost,127.0.0.1,.localhost"

The h in socks5h mean that the proxy request will use the host name instead of the IP address i.e. the name resolution is done on the remote side.

Proxifiers

Some tools do not support SOCKS proxy. One solution is to use a SOCKS proxifier suchs as proxychains or tsocks. They rely on LD_PRELOAD to redirect normal TCP socket calls to the proxy:

proxychains mysql -h 127.0.0.1 -P 3306

Firefox

For Firefox, the configuration is in Preferenced, Advanced, Network, Connexion Parameters.

There is a checkbox to enable remote-side domain name resolution. When checked, the domain name resolution is done by the SSH server (i.e. at the other end of the tunnel) insteaf of by the application.

[DNS  ]-->[DNS ]           [SOCKS]-->[SOCKS|SSH]==>[SSH|DNS]--->[DNS]
[UDP  ]-->[UDP ]           [TCP  ]-->[TCP      ]-->[TCP|UDP]--->[UDP]
[IP   ]---[IP  ]           [IP   ]---[IP       ]---[IP     ]----[IP ]
Client    DNS resolver     Client   SSH client     SSH server   DNS resolver
Local DNS resolution (left) vs. remote DNS resolution (right) when using SSH dynamic port forwarding

Note: using SOCKS over a Unix domain socket with Firefox

Firefox supports connecting to the proxy over Unix domain socket by specifying a file: URI (eg. file:///run/user/1000/ssh-socks) instead of the host name in the proxy configuration (the port is ignored in this case). This feature was requested by the Tor project.

For security reasons, this does not work from proxy.pac files.

When using the FoxyProxy extension with Firefox, you can use the same syntax to declare a SOCKS proxy available through a Unix domain socket.

Chromium

Chromium gets its proxy configuration from the system. It is possible to force the usage of a proxy with: --proxy-server=socks5://127.0.0.1:1080.

Apparently, if you want to ensure that Chromium does not leak DNS queries on the LAN, you should pass: --host-resolver-rules="MAP * 0.0.0.0 , EXCLUDE myproxy" as well.

Using a HTTP proxy

Some tools do not support SOCKS proxy but support HTTP proxy. It is possible to build an HTTP proxy over a SOCKS one using polipo. However, this program is no longer maintained.

Appendix, proxy routing

Some notes on how to use different proxies (or not proxy at all) at the same time.

Proxy routing with FoxyProxy

You might want to use the proxy only for some hosts or you might want to use different proxies for different hosts. FoxyProxy is a browser plugin available both for Firefox (and other Gecko-based applications such as Thunderbird) and for Chromium which gives you more flexibility:

On Firefox, it is possible to declare SOCKS proxies over Unix-socket using FoxyProxy using the file: syntax (eg. file:///run/user/1000/ssh-socks).

Proxy routing with with proxy.pac

Another solution for this which might work for other application, is to use proxy.pac. This is a JavaScript source file which define as FindProxyForURL(url, host) function. This function is called for each connection and returns the connection method for this request:

It is not possible to use proxies over Unix-socket using proxy.pac: you can use FoxyProxy instead.

Example: only use the proxy for some hosts

var hosts = [ "foo.example.com" ];

function FindProxyForURL(url, host)
{
    if (hosts.includes(host))
        return "SOCKS localhost:9080";
    else
        return "DIRECT";
}

Example: only use the proxy for some URI schemes

function FindProxyForURL(url, host)
{
    if (url.startsWith("https://"))
        return "SOCKS localhost:9080";
    else
        return "DIRECT";
}

Browser configuration:

References


  1. This may be because you are not using OpenSSH (but another implementation) on the client or server side. This may be because you are a SSH bastion blocks this OpenSSH extension. ↩︎