{"version": "https://jsonfeed.org/version/1", "title": "/dev/posts/ - Tag index - python", "home_page_url": "https://www.gabriel.urdhr.fr", "feed_url": "/tags/python/feed.json", "items": [{"id": "http://www.gabriel.urdhr.fr/2023/08/29/simple-iterm2-image/", "title": "Simple terminal image display using the iTerm2 image protocol", "url": "https://www.gabriel.urdhr.fr/2023/08/29/simple-iterm2-image/", "date_published": "2023-08-29T00:00:00+02:00", "date_modified": "2023-08-29T00:00:00+02:00", "tags": ["computer", "terminal", "image", "python", "matplotlib"], "content_html": "
A simple way to display image in a terminal using the iTerm2 image protocol.\nThis is supported by iTerm2,\nWezTerm,\nrecent versions of Konsole.
\nIt is very simple to display an image using the iTerm2 protocol\nwithout any dedicated program:
\n(printf \"\\e]1337;File=inline=1:\" ; base64 -w0 image.jpg ; printf \"\\a\\n\")\n
\nSome alternatives:
\nkitty +kitten icat
(using the kitty protocol).References:
\n\nAdding support for iTerm2 to Matplotlib can be done with:
\n# mpliterm.py\nfrom matplotlib.backend_bases import Gcf\nfrom matplotlib.backends.backend_agg import FigureCanvasAgg\nfrom io import BytesIO\nfrom base64 import b64encode\nimport sys\n\nFigureCanvas = FigureCanvasAgg\n\n\ndef show(*args, **kwargs):\n for figmanager in Gcf.get_all_fig_managers():\n buffer = BytesIO()\n figmanager.canvas.figure.savefig(buffer, format=\"png\")\n data = buffer.getvalue()\n data64 = b64encode(data).decode(\"ascii\")\n sys.stdout.write('\\x1b]1337;File=inline=1:' + data64 + \"\\a\\n\")\n
\nUsed with:
\nexport PYTHONPATH=/path-to-mplitem/\nexport MPLBACKEND=module://mpliterm\npython3 test-plot.py\n
\n# test-plot.py\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nx = np.linspace(0, 10, 300)\ny = np.cos(x)\nplt.plot(x, y)\nplt.show()\n
\nReferences:
\nHere is the workflow I am using to generate simple text documents\n(resume, cover letters, etc.) from Markdown, YAML and Jinja2 templates.
\nSummary:
\nGood-old make
coordinates the different steps.
The nice things about this approach is that:
\nThe input of the document is a Markdown file with frontmatter and looks like that:
\nname: John Doe\ntitle: Super hero\naddress:\n - 221B Baker Street\n - London\n - UK\nlang: en\nphone: +XX-X-XX-XX-XX-XX\nemail: john.doe@example.com\nwebsite: http://www.example.com/john.doe/\n---\n
\n## Introduction\n\nLorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor\nincididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,\nquis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo\nconsequat. Duis aute irure dolor in reprehenderit in voluptate velit esse\ncillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat\nnon proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n\n\n## Discussion\n\nLorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod\ntempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,\nquis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo\nconsequat. Duis aute irure dolor in reprehenderit in voluptate velit esse\ncillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat\nnon proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n\n\n\n## Conclusion\n\nLorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod\ntempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,\nquis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo\nconsequat. Duis aute irure dolor in reprehenderit in voluptate velit esse\ncillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat\nnon proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n
\nI am using a Jinja2 template to convert the input Markdown document into HTML:
\n<html xmlns=\"http://www.w3.org/1999/xhtml\" lang=\"{{ lang | escape}}\">\n<head>\n <meta charset=\"utf-8\"/>\n <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\"/>\n <title>{{ title | escape }}</title>\n <link rel=\"stylesheet\" type=\"text/css\" href=\"style.css\"/>\n</head>\n<body>\n\n<header>\n <address>\n <strong>{{ name | escape }}</strong><br/>\n {% for line in address %}\n {{ line | escape }}<br/>\n {% endfor %}\n </address>\n <span class=\"details\">\n <p><span title=\"{{ 'T\u00e9l\u00e9phone' if lang == 'fr' else 'Phone' }}\">\u260e</span>\n <a href=\"tel:{{ phone | escape}}\">{{ phone | escape }}</a></p>\n <p><a href=\"mailto:{{ email | escape }}\">{{ email | escape }}</a></p>\n <p><a href=\"{{ website | escape }}\">{{ website | escape }}</a></p>\n </span>\n</header>\n\n<h1>{{ title | escape }}</h1>\n{{ body }}\n\n</body>\n</html>\n
\nThe conversion is done with a Python script (./render
):
#!/usr/bin/env python3\n\nfrom sys import argv\nimport re\n\nimport yaml\nimport markdown\nfrom markdown.extensions.extra import ExtraExtension\nfrom jinja2 import Environment, FileSystemLoader\nfrom atomicwrites import atomic_write\n\nEXT = [\n ExtraExtension()\n]\n\nenv = Environment(\n loader=FileSystemLoader('.'),\n autoescape=False,\n)\ntemplate = env.get_template('template.j2')\n\nfilename = argv[1]\nout_filename = argv[2]\n\nRE = re.compile(r'^---\\s*$', re.M)\n\n\ndef split_document(data):\n \"\"\"\n Split a document into a YAML frontmatter and a body\n \"\"\"\n lines = str.splitlines(data)\n if not RE.match(lines[0]):\n raise Exception(\"Missing YAML start\")\n for i in range(1, len(lines)):\n if RE.match(lines[i]):\n head_raw = \"\\n\".join(lines[:i+1])\n head = list(yaml.load_all(head_raw))[0]\n body = \"\\n\".join(lines[i+2:])\n return (head, body)\n raise Exception(\"Missing YAML end\")\n\n\nwith open(filename, \"r\") as f:\n content = f.read()\n(head, body) = split_document(content)\nbody_html = markdown.markdown(body, extensions=EXT)\nwith atomic_write(out_filename, overwrite=True) as f:\n f.write(template.render(**head, body=body_html))\n
\nCalled as:
\n./render doc.md doc.html\n
\nI am using WeasyPrint to generate PDF from HTML:
\nweasyprint doc.html doc.pdf\n
\nWeasyPrint has some support for page CSS:
\n@page {\n size: A4;\n margin: 1cm;\n margin-top: 2cm;\n margin-bottom: 2cm;\n}\n\n@media print {\n body {\n margin-top: 0;\n margin-bottom: 0;\n }\n}\n\nh1, h2, h3 {\n page-break-after: avoid;\n page-break-inside: avoid;\n}\n\nli {\n page-break-inside: avoid;\n}\n
\nIt has support for links, PDF bookmarks, attachements, fonts, etc.
\nCurrently I am using a Makefile
to compose the different steps:
.PHONY: all clear\n\nall: doc.html\nclear:\n\trm doc.html\n\ndoc.html: doc.md template.j2 render\n\t./render doc.md doc.html\n\ndoc.pdf: doc.html\n\tweasyprint doc.html doc.pdf\n
\n"}, {"id": "http://www.gabriel.urdhr.fr/2018/03/19/sibling-tco-in-python/", "title": "Sibling tail call optimization in Python", "url": "https://www.gabriel.urdhr.fr/2018/03/19/sibling-tco-in-python/", "date_published": "2018-03-19T00:00:00+01:00", "date_modified": "2018-03-19T00:00:00+01:00", "tags": ["computer", "python", "functional"], "content_html": "In Tail Recursion In Python,\nChris Penner\nimplements (self) tail-call optimization (TCO) in Python using a function decorator.\nHere I am extending the approach for sibling calls.
\nThe example function is a functional-style factorial function defined with\ntail recursion as:
\ndef factorial(n, accumulator=1):\n if n == 0:\n return accumulator\n else:\n return factorial(n-1, accumulator * n)\n
\nThe python interpreter does not implement taill call optimization so calling\nfactorial(1000)
overflows the stack:
\nTraceback (most recent call last):\n File \"plain.py\", line 10, in\n\n print(factorial(2000))\n File \"plain.py\", line 8, in factorial\n return factorial(n-1, accumulator * n)\n File \"plain.py\", line 8, in factorial\n return factorial(n-1, accumulator * n)\n File \"plain.py\", line 8, in factorial\n return factorial(n-1, accumulator * n)\n File \"plain.py\", line 8, in factorial\n return factorial(n-1, accumulator * n)\n File \"plain.py\", line 8, in factorial\n return factorial(n-1, accumulator * n)\n [..]\n File \"plain.py\", line 8, in factorial\n return factorial(n-1, accumulator * n)\nRuntimeError: maximum recursion depth exceeded\n
Chris Penner implements taill call optimization using a function decorator\n(tail_recursive
):
# Normal recursion depth maxes out at 980, this one works indefinitely\n@tail_recursive\ndef factorial(n, accumulator=1):\n if n == 0:\n return accumulator\n recurse(n-1, accumulator=accumulator*n)\n
\nWith the recurse
function triggering the self taill-call\n(factorial
calls itself).
The implementation is:
\nclass Recurse(Exception):\n def __init__(self, *args, **kwargs):\n self.args = args\n self.kwargs = kwargs\n\ndef recurse(*args, **kwargs):\n raise Recurse(*args, **kwargs)\n\ndef tail_recursive(f):\n def decorated(*args, **kwargs):\n while True:\n try:\n return f(*args, **kwargs)\n except Recurse as r:\n args = r.args\n kwargs = r.kwargs\n continue\n return decorated\n
\nThis is implemented by wrapping the original factorial
functions.\nThe recurse
functions throws an exception which is caught be the wrapper\nfunction: the wrapper calls the factorial
function again with the new\narguments.
One limitation of this approach is that it only allows self tail-calls\n(the function calls itself) but not sibling tail-calls (eg. functions a
calls\nfunction b
and function b
calls function a
).
Instead we could use someting like this:
\nfrom tco import tail_recursive, tail_call\n\n\n# Normal recursion depth maxes out at 980, this one works indefinitely\n@tail_recursive\ndef factorial(n, accumulator=1):\n if n == 0:\n return accumulator\n return tail_call(factorial)(n-1, accumulator=accumulator*n)\n\n\nprint(factorial(2000))\n
\nWith the implementation:
\nclass TailCall:\n def __init__(self, f, args, kwargs):\n self.f = f\n self.args = args\n self.kwargs = kwargs\n\n\ndef tail_call(f):\n def wrapper(*args, **kwargs):\n return TailCall(f, args, kwargs)\n return wrapper\n\n\ndef tail_recursive(f):\n def wrapper(*args, **kwargs):\n func = f\n while True:\n res = func(*args, **kwargs)\n if not isinstance(res, TailCall):\n return res\n args = res.args\n kwargs = res.kwargs\n if hasattr(res.f, \"__tc_original__\"):\n func = getattr(res.f, \"__tc_original__\")\n else:\n func = res.f\n wrapper.__tc_original__ = f\n return wrapper\n
\nThis implementation does not use exceptions so we need to return
the\nTailCall
value (otherwise nothing happens).
With this approach, we can have sibling TCO:
\nfrom tco import tail_recursive, tail_call\n\n\n@tail_recursive\ndef factorial(n, accumulator=1):\n if n == 0:\n return accumulator\n return tail_call(factorial2)(n-1, accumulator=accumulator*n)\n\n\n@tail_recursive\ndef factorial2(n, accumulator=1):\n if n == 0:\n return accumulator\n return tail_call(factorial)(n-1, accumulator=accumulator*n)\n\n\nprint(factorial(2000))\n
\nI tend to like the exception free-approach better. It might make the\ntyping system unhappy however.
\nHere is the same thing with exceptions:
\nclass TailCall(BaseException):\n def __init__(self, f, args, kwargs):\n self.f = f\n self.args = args\n self.kwargs = kwargs\n\n\ndef tail_call(f):\n def wrapper(*args, **kwargs):\n raise TailCall(f, args, kwargs)\n return wrapper\n\n\ndef tail_recursive(f):\n def wrapper(*args, **kwargs):\n func = f\n while True:\n try:\n return func(*args, **kwargs)\n except TailCall as e:\n args = e.args\n kwargs = e.kwargs\n if hasattr(e.f, \"__tc_original__\"):\n func = getattr(e.f, \"__tc_original__\")\n else:\n func = e.f\n wrapper.__tc_original__ = f\n return wrapper\n
\nI am deriving TailCall
from BaseException
instead of Exception
because\nthe tail-recursive functions might catch Exception
which would break the\nTCO mechanism.
While looking at the OpenSSH ssh_config
manpage, I found the\nProxyUseFdpass
configuration I did not know about.\nIt is apparently not widely known or used.
Update 2017-08-02: netcat (nc
) has an option\n(nc.openbsd -F www.example.com 80
) to pass the created\nfile descriptor using the \"fdpass\" mechanism.\nIn addition to the straightforward connect()
,\nit can pass a file descriptor having initiated a connection through a SOCKS\nproxy (with -x proxy.example.com
) or with HTTP connect\n(-x proxy.example.com -X connect
).
Update (2021-02-09):\nreplaced fds.fromstring()
with fds.frombytes()
.
ProxyCommand
OpenSSH client has a ProxyCommand
configuration which can be used to\nuse a command as a transport to the server:
\n\nSpecifies the command to use to connect to the server. The command\nstring extends to the end of the line, and is executed using the\nuser's shell \u2018exec\u2019 directive to avoid a lingering shell process.
\n
Instead of opening a socket to the server itself, the OpenSSH client\nspawns the specified command and use its standard input and output to\ncommunicate with the server.
\nThe man page suggests to use (the OpenBSD variant of) netcat to connect\nthrough an HTTP (or SOCKS) proxy:
\nProxyCommand /usr/bin/nc -X connect -x 192.0.2.0:8080 %h %p\n
\nA typical usage is to use a relay/bastion/jump/gateway[^jump] SSH\nserver with ssh -W
[1]:
Host gateway.example.com\nProxyCommand none\n\nHost *.example.com\nProxyCommand ssh gateway.example.com -W %h:%p\n
\nUsing ProxyJump
instead
OpenSSH 7.3 includes\nspecial support for SSH jump servers with the ProxyJump
\nconfiguration,
Host *.example.com\nProxyJump gateway.example.com\n
\nor the -J
flag:
ssh -J gateway.example.com foo.example.com\n
\nCode injection through ProxyCommand
(update 2023-12-21)
Using ProxyCommand
with hostname (%h
) or username (%u
) expansions\nmay expose you to shell command injections:\nit is possible\nto inject arbitrary shell commands through\nthe hostname (%h
) or username (%u
)\n(CVE-2023-51385).
For this to be exploited,\nan attacker would have to trick you into SSH-ing\ninto a host with a weird (containing shell metacharacters) name\nor using a weird username.\nThis can for example happen\nwhen doing a recursive git clone
\nof a malicious git repository.
This has been mitigated in OpenSSH 9.6.p1\nby rejecting host and username containing shell meta-characters\n(which should mostly work on a regular shell\nbut might not work if using an exotic shell).
\nProxyUseFdPass
While looking at the new ProxyJump
configuration, I found a\nProxyUseFdpass
option which:
\n\nSpecifies that ProxyCommand will pass a connected file descriptor\nback to ssh(1) instead of continuing to execute and pass data. The\ndefault is \u201cno\u201d.
\n
When enabled, instead of communicating with the server through the\nProxyCommand
standard input and output, the SSH client expects the\ncommand to give it a file descriptor to use. The idea is to avoid\nhaving an uncessary lingering process and extra write/reads when it is\nnot necessary[2].
The documentation does not explain how it is supposed to work exactly\nand I did not find any working example or any suggestion of a program\nwhich would be able to pass the file descriptor.
\nThe spawned command is expected to:
\nsendmsg
with SCM_RIGHTS
\nusing a one-byte message;exit(0)
.A minimal program which does the job is:
\n#!/usr/bin/env python3\n\nimport sys\nimport socket\nimport array\n\n# Create the file descriptor:\ns = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, 0)\ns.connect((sys.argv[1], int(sys.argv[2])))\n\n# Pass the file descriptor:\nfds = array.array(\"i\", [s.fileno()])\nancdata = [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds)]\nsocket.socket(fileno = 1).sendmsg([b'\\0'], ancdata)\n
\nWhich can be used with:
\nProxyCommand /path/to/passfd %h %p\nProxyUseFdpass yes\n
\nIn its current form, it does not do much.\nIt creates a socket the same way the OpenSSH client\nwould have and pass it to the OpenSSH client.\nHowever, it can extended in order to do things such as:
\nSO_DONTROUTE
, etc.);For testing purpose this receiving program can be used:
\n#!/usr/bin/env python3\n\nimport os\nimport sys\nimport socket\nimport array\n\n(a, b) = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM, 0)\npid = os.fork()\n\ndef recv_fd(sock):\n fds = array.array(\"i\")\n cmsg_len = socket.CMSG_LEN(fds.itemsize)\n msg, ancdata, flags, addr = sock.recvmsg(1, cmsg_len)\n for cmsg_level, cmsg_type, cmsg_data in ancdata:\n if (cmsg_level, cmsg_type) == (socket.SOL_SOCKET, socket.SCM_RIGHTS):\n fds.frombytes(cmsg_data)\n return fds[0]\n sys.exit(1)\n\nif pid == 0:\n # Exec specified command in the child:\n a.close()\n os.dup2(b.fileno(), 0)\n os.dup2(b.fileno(), 1)\n b.close()\n os.execvp(sys.argv[1], sys.argv[1:])\nelse:\n # Receive file descriptor and wait in the parent:\n b.close()\n s = recv_fd(a)\n os.waitpid(pid, 0)\n print(s)\n
\nWhich can be used as:
\nfdrecv fdpass localhost 80\n
\nIt is often suggested to use this configuration instead:
\nProxyCommand ssh gateway.example.com nc %h %p\n
\nThis requires netcat to be available on the server. ssh -W
only\nneeds client-side support which is available in OpenSSH since\n5.4 (released in 2010)\nand the SSH server to accept TCP forwarding. \u21a9\ufe0e
It is not usable for a SSH jump server but can be used in\nsimpler cases. \u21a9\ufe0e
\nThere has been some articles lately about Intel Active Management Technology (AMT)\nand its impact on\nsecurity,\ntrust,\nprivacy\nand free-software.\nAMT supposed to be widely deployed in newest Intel hardware.\nSo I wanted to see if I could find some AMT devices in the wild.
\nUpdate: 2017-05-15 Add references related to CVE-2017-5689 (AMT\nvulerability).
\nAMT is an Intel technology for\nout of band management\n(without cooperation of the OS)\nof a computer over the network even if the computer is turned off.\nIt can be used to do things such as:
\nIt implements the\nDesktop and mobile Architecture for System Hardware (DASH)\nstandard and is similar to the Intelligent Platform Management Interface (IMPI)\nin terms of features.\nIt uses SOAP over HTTP\nwith some WS-* greatness.\nIt comes with bells and whistles such as\nintegration with Active Directory \ud83d\ude31.
\nWhen AMT is enabled, IP packets incoming on the builtin network adapter for some\nTCP and UDP ports are sent directly to the\nME\ninstead of reaching the OS.\nThe ME has its own processor and its own OS and can give access to the hardware\nover the network. Usually, the ME and the main system share the same network\ninterface, MAC address and IPv6 address.
\n\n\n\n\nA physical system\u2019s out-of-band Management Access Point and the In-Band host\nshall share the MAC address and IPv4 address of the network interface.\nManageability traffic shall be routed to the MAP [Management Access Point]\nthrough the well known system ports defined by IANA.
\n
\n\nTCP/UDP messages addressed to certain registered ports are routed to Intel\nAMT when those ports are enabled. Messages received on a wired LAN interface\ngo directly to Intel AMT. Messages received on a wireless interface go to the\nhost wireless driver. The driver detects the destination port and sends the\nmessage to Intel AMT.
\n
My work laptop has a Intel Management Engine Interface (MEI) device\nand the system loads the MEI Linux module:
\n$ lspci\n[...]\n00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)\n[...]\n\n$ lsmod | grep mei\nmei_me 32768 0\nmei 94208 1 mei_me\n\n$ ls -l /dev/mei*\ncrw------- 1 root root 246, 0 juil. 7 08:53 /dev/mei0\n\n$ grep mei /lib/modules/4.6.0-1-amd64/modules.alias\nalias pci:v00008086d00005A9Asv*sd*bc*sc*i* mei_me\nalias pci:v00008086d00001A9Asv*sd*bc*sc*i* mei_me\n[...]\nalias mei:pn544:0bb17a78-2a8e-4c50-94d4-50266723775c:*:* pn544_me\n\n$ cat /sys/bus/pci/drivers/mei_me/0000:00:16.0/uevent\nMAJOR=248\nMINOR=0\nDEVNAME=mei0\n
\nMEI\nis a PCI-based interface to the Management Engine (ME) from within the computer.
\nHowever, there is no option to disable AMT in the BIOS on my laptop.\nApparently,\nAMT is not enabled\non this device even if this not\nabsolutely clear.\nThe hardware seems to be there though.
\nWe can use the discovery mechanism of\nAMT\nin order to detect AMT devices on a network.\nThe AMT (and DASH) discovery uses\ntwo phases:
\nthe first phase uses ASF RMCP (Alert Standard Format - Remote Management and Control Protocol);
\nthe second phase uses the WS-Management Identify method.
\nThe second phase is not so useful so I will focus on the first one.
\nThe first phase is quite simple:
\nthe client sends a (possibly) broadcast RMCP Presence Ping message over\nUDP port 623 (asf-rmcp);
\nthe nodes supporting ASF,\nsuch as DASH/AMT and IPMI (Intelligent Platform Management Interface) nodes,\nsend a RMCP Presence Pong.
\nSize | \nField | \n
---|---|
1B | \nVersion (0x6 for RMCP 1.0) | \n
1B | \nReserved | \n
1B | \nSequence number (0--254, 255 when no no acknowledge is needed) | \n
1B | \nClass of Message | \n
\n | Bit 7, 1 for acknowledge | \n
\n | Bits 6:4, reserved | \n
\n | Bits 3:0, 6 for ASF, 7 for IPMI, etc. | \n
All messages which are not acknowledges have a\nRMCP data\nfield after the header:
\nSize | \nField | \n
---|---|
4B | \nIANA Entreprise Number, servces as a namespace for the message type (4542 for ASF-RMCP) | \n
1B | \nMessage Type (for ASF-RMCP, we have 0x80 for Presence Ping, 0x40 for Presence Pong) | \n
1B | \nMessage Tag | \n
1B | \nReserved | \n
1B | \nData Length | \n
Var | \nData (payload) | \n
We can handle RMCP messages with:
\nASF_RMCP_VERSION1 = 0x6\nIANA_ASF = 4542\nASF_RMCP_FORMAT = \"!BBBBIBBBB\"\n\n# RCMP ASF message (not ack)\nclass Message:\n def __init__(self):\n self.version = ASF_RMCP_VERSION1\n self.reserved = 0x00\n self.seqno = 0x00\n self.message_class = 0x00\n self.entreprise_number = IANA_ASF\n self.message_type = 0x00\n self.message_tag = 0x00\n self.reserved = 0x00\n self.data = bytearray()\n\n def load(self, message):\n if (len(message) < struct.calcsize(ASF_RMCP_FORMAT)):\n raise \"Message too small\"\n (self.version, self.reserved, self.seqno, self.message_class,\n self.entreprise_number, self.message_type, self.message_tag,\n self.reserved, data_length) = \\\n struct.unpack_from(ASF_RMCP_FORMAT, message)\n if len(message) != data_length + struct.calcsize(ASF_RMCP_FORMAT):\n raise \"Bad length\"\n rmcp_size = struct.calcsize(ASF_RMCP_FORMAT)\n self.data = bytearray(memoryview(message)[rmcp_size:])\n\n def to_bytes(self):\n size = struct.calcsize(ASF_RMCP_FORMAT) + len(self.data)\n res = bytearray(size)\n struct.pack_into(ASF_RMCP_FORMAT, res, 0,\n self.version, self.reserved, self.seqno,\n self.message_class, self.entreprise_number,\n self.message_type, self.message_tag, self.reserved,\n len(self.data))\n memoryview(res)[struct.calcsize(ASF_RMCP_FORMAT):] = self.data\n return res\n
\nFor Presence Ping, there is no payload.\nFor Presence Pong,\nthe payload is:
\nSize | \nField | \n
---|---|
4B | \nIANA Entreprise Number (4542 if not OEM specific-things are used) | \n
4B | \nOEM Defined | \n
1B | \nSupported Entities | \n
\n | Bit 7, set if IPMI is supported | \n
\n | Bits 6:4, reserved | \n
\n | Bits 3:0, 1 for ASF version 1.0 | \n
1B | \nSupported interactions | \n
\n | Bit 5: set if DASH (AMT) is supported | \n
5B | \nReserved | \n
We can handle Pong Presence data with:
\nASF_RMCP_PONG_FORMAT = \"!IIBBBBBBBB\"\n\nclass PongData:\n def __init__(self, payload):\n if struct.calcsize(ASF_RMCP_PONG_FORMAT) != len(payload):\n print(\"Bad length for pong payload expected %i but was %i\" %\n (struct.calcsize(ASF_RMCP_PONG_FORMAT), len(payload)))\n (self.entreprise_number, self.oem_defined, self.supported_entities,\n self.supported_interactions, self.reserved1,\n self.reserved2, self.reserved3, self.reserved4, self.reserved5,\n self.reserved6) = struct.unpack_from(ASF_RMCP_PONG_FORMAT, payload)\n\n def ipmi(self):\n return (self.supported_entities & 127) != 0\n\n def asf(self):\n return (self.supported_entities & 15) == 1\n\n def dash(self):\n return (self.supported_interactions & 32) != 0\n\n def features(self):\n res = []\n if self.ipmi():\n res.append(\"ipmi\")\n if self.asf():\n res.append(\"asf\")\n if self.dash():\n res.append(\"dash\")\n return res\n
\nWe send a Presence Ping message to some (possibly broadcast) address:
\nASF_RMCP_PORT = 623\nASF_RMCP_MESSAGE_TYPE_PRESENCE_PING = 0x80\n\nm = Message()\nm.message_class = ASF_RMCP_VERSION1\nm.message_type = ASF_RMCP_MESSAGE_TYPE_PRESENCE_PING\n\nsock = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, socket.IPPROTO_UDP)\nsock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)\nsock.sendto(m.to_bytes(), (address, ASF_RMCP_PORT))\n
\nAnd then we process the messages:
\nASF_RMCP_MESSAGE_TYPE_PRESENCE_PONG = 0x40\nASF_RMCP_MESSAGE_TYPE_PRESENCE_PING_ACK = 0x86\n\nsock.settimeout(1)\n\ntry:\n while True:\n data, addr = sock.recvfrom(1024)\n logging.debug(\"From \" + str(addr[0]) + \": \" + str(data))\n if len(data) == 4 and data[0] == ASF_RMCP_VERSION1 and data[2] == 0 \\\n and data[3] == ASF_RMCP_MESSAGE_TYPE_PRESENCE_PING_ACK:\n logging.debug(\"Ack from \" + str(addr[0]))\n continue\n try:\n m.load(data)\n except:\n continue\n if m.message_type == ASF_RMCP_MESSAGE_TYPE_PRESENCE_PONG:\n # Pong:\n print(str(addr[0]))\n pongData = PongData(m.data)\n features = pongData.features()\n print(\"\\tEntreprise: %s\" %\n entreprise_name(pongData.entreprise_number))\n if len(features) != 0:\n print(\"\\tFeatures: %s\" % \",\".join(features))\nexcept socket.timeout:\n pass\n
\nHere is the full code:
\n#!/usr/bin/env python3\n# Use ASF RMCP to discover RMCP-aware nodes (such as AMT/AMT or IPMI)\n# Keywords: DMTF, ASF RMCP, DASH, AMT, IPMI.\n\nimport socket\nimport ctypes\nimport struct\nimport sys\nimport logging\nimport ipaddress\n\nASF_RMCP_PORT = 623\nASF_RMCP_FORMAT = \"!BBBBIBBBB\"\nASF_RMCP_PONG_FORMAT = \"!IIBBBBBBBB\"\nASF_RMCP_VERSION1 = 0x6\nASF_RMCP_MESSAGE_TYPE_PRESENCE_PONG = 0x40\nASF_RMCP_MESSAGE_TYPE_PRESENCE_PING = 0x80\nASF_RMCP_MESSAGE_TYPE_PRESENCE_PING_ACK = 0x86\nIANA_ASF = 4542\n\naddress = sys.argv[1]\nif ipaddress.ip_address(address).version == 4:\n address = \"::ffff:\" + address\n\nentreprise_names = {\n 343: \"Intel\",\n 3704: \"AMD\",\n 4542: \"Alerting Specifications Forum\",\n}\n\n\ndef entreprise_name(n):\n if n in entreprise_names:\n return entreprise_names[n]\n else:\n return str(n)\n\n\n# RCMP ASF message (not ack)\nclass Message:\n def __init__(self):\n self.version = ASF_RMCP_VERSION1\n self.reserved = 0x00\n self.seqno = 0x00\n self.message_class = 0x00\n self.entreprise_number = IANA_ASF\n self.message_type = 0x00\n self.message_tag = 0x00\n self.reserved = 0x00\n self.data = bytearray()\n\n def load(self, message):\n if (len(message) < struct.calcsize(ASF_RMCP_FORMAT)):\n raise \"Message too small\"\n (self.version, self.reserved, self.seqno, self.message_class,\n self.entreprise_number, self.message_type, self.message_tag,\n self.reserved, data_length) = \\\n struct.unpack_from(ASF_RMCP_FORMAT, message)\n if len(message) != data_length + struct.calcsize(ASF_RMCP_FORMAT):\n raise \"Bad length\"\n rmcp_size = struct.calcsize(ASF_RMCP_FORMAT)\n self.data = bytearray(memoryview(message)[rmcp_size:])\n\n def to_bytes(self):\n size = struct.calcsize(ASF_RMCP_FORMAT) + len(self.data)\n res = bytearray(size)\n struct.pack_into(ASF_RMCP_FORMAT, res, 0,\n self.version, self.reserved, self.seqno,\n self.message_class, self.entreprise_number,\n self.message_type, self.message_tag, self.reserved,\n len(self.data))\n memoryview(res)[struct.calcsize(ASF_RMCP_FORMAT):] = self.data\n return res\n\n\nclass PongData:\n def __init__(self, payload):\n if struct.calcsize(ASF_RMCP_PONG_FORMAT) != len(payload):\n print(\"Bad length for pong payload expected %i but was %i\" %\n (struct.calcsize(ASF_RMCP_PONG_FORMAT), len(payload)))\n (self.entreprise_number, self.oem_defined, self.supported_entities,\n self.supported_interactions, self.reserved1,\n self.reserved2, self.reserved3, self.reserved4, self.reserved5,\n self.reserved6) = struct.unpack_from(ASF_RMCP_PONG_FORMAT, payload)\n\n def ipmi(self):\n return (self.supported_entities & 127) != 0\n\n def asf(self):\n return (self.supported_entities & 15) == 1\n\n def dash(self):\n return (self.supported_interactions & 32) != 0\n\n def features(self):\n res = []\n if self.ipmi():\n res.append(\"ipmi\")\n if self.asf():\n res.append(\"asf\")\n if self.dash():\n res.append(\"dash\")\n return res\n\nm = Message()\nm.message_class = ASF_RMCP_VERSION1\nm.message_type = ASF_RMCP_MESSAGE_TYPE_PRESENCE_PING\n\nsock = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, socket.IPPROTO_UDP)\nsock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)\nsock.sendto(m.to_bytes(), (address, ASF_RMCP_PORT))\nsock.settimeout(1)\n\nlogging.info(\"Listening\")\ntry:\n while True:\n data, addr = sock.recvfrom(1024)\n logging.debug(\"From \" + str(addr[0]) + \": \" + str(data))\n if len(data) == 4 and data[0] == ASF_RMCP_VERSION1 and data[2] == 0 \\\n and data[3] == ASF_RMCP_MESSAGE_TYPE_PRESENCE_PING_ACK:\n logging.debug(\"Ack from \" + str(addr[0]))\n continue\n try:\n m.load(data)\n except:\n continue\n if m.message_type == ASF_RMCP_MESSAGE_TYPE_PRESENCE_PONG:\n # Pong:\n print(str(addr[0]))\n pongData = PongData(m.data)\n features = pongData.features()\n print(\"\\tEntreprise: %s\" %\n entreprise_name(pongData.entreprise_number))\n if len(features) != 0:\n print(\"\\tFeatures: %s\" % \",\".join(features))\nexcept socket.timeout:\n pass\n
\nWe can discover devices on the local network by using its broadcast address:
\n$ ./rmcp-discover 192.0.2.255\n::ffff:192.0.2.56\n\tEntreprise: Intel\n\tFeatures: dash\n::ffff:152.81.7.32\n\tEntreprise: Intel\n\tFeatures: dash\n::ffff:192.0.2.228\n\tEntreprise: Intel\n\tFeatures: dash\n::ffff:192.0.2.230\n\tEntreprise: Intel\n\tFeatures: dash\n::ffff:152.81.3.90\n\tEntreprise: Intel\n\tFeatures: dash\n::ffff:192.0.2.170\n\tEntreprise: Intel\n\tFeatures: dash\n::ffff:152.81.8.105\n\tEntreprise: Intel\n\tFeatures: dash\n::ffff:152.81.5.123\n\tEntreprise: Intel\n\tFeatures: dash\n::ffff:192.0.2.235\n\tEntreprise: Intel\n\tFeatures: dash\n::ffff:192.0.2.29\n\tEntreprise: Intel\n\tFeatures: dash\n::ffff:192.0.2.233\n\tEntreprise: Intel\n\tFeatures: dash\n::ffff:192.0.2.171\n\tEntreprise: Intel\n\tFeatures: dash\n
\nThey advertise Intel and DASH: those are probably AMT devices.
\nWe can use the same script to discover IPMI nodes as well:
\n$ ./rmcp-discover 198.51.100.42\n::ffff:198.51.100.42\n\tEntreprise: Alerting Specifications Forum\n\tFeatures: ipmi,asf\n
\nWe cannot (reliably) use this to detect AMT on the local machine. The reason is\nthat the messages are sent to the ME when they arrive on the hardware Ethernet\nadapter. Messages emitted by the localhost to its own IP address\nare handled internally by the OS: they are received by the Ethernet adapter\nand thus do not reach the ME.\nIn order to communicate to its own ME, the OS needs to communicate using the MEI\ninstead of using IP. The Intel LMS (Local Management Service)\ncan be installed to reach the local ME over\nIP: as far as I know, it listens on the suitable localhost TCP and UDP ports and forwards the request\nto the ME using the MEI.
\nInteresting references following the\nINTEL-SA-00075/CVE-2017-5689\nvulerability:
\nmei-amt-check
: \u00ab\u00a0Check\nwhether AMT is enabled and provisioned under Linux\u00a0\u00bb