At linux.conf.au 2015 in Auckland, Matthew Garrett gave a session about the security risks of the Intelligent Platform Management Interface (IPMI). IPMI is a largely a tool of convenience for administering server farms, and is implemented on supported motherboards as a feature that sits outside of the usual bootloader and operating system. It should come as no surprise, then, that it can be the source of serious security problems if is not prepared for properly.

Garrett began by reminding the audience of his previous experiences dissecting the security hazards of various four-letter words—namely ACPI and UEFI. Much like those examples, IPMI is an "out-of-band" feature: motherboard manufacturers implement it in a separate chipset without source code available, which makes it particularly challenging to insulate the main OS against.

Inside IPMI

Though related to ACPI, IPMI does have a different and reasonably well-defined use case. The underlying principle, Garrett said, is that data centers are awful places where no one wants to be: they are noisy, have nowhere to sit, the temperature is unbearable, and the phone reception is terrible. Thus, any technology that enables people to spend less time in their data center is perceived as good.

That is the point of IPMI; it allows system administrators to monitor the health of their servers remotely and to troubleshoot problems—all without requiring access to the machines' OS. This is convenient for server hosting companies, where the customer expects the provider to maintain the machines, but usually not to interfere with the OS. The IPMI interface provides a serial terminal over its own network connection, so the host OS never even needs to know it is running.

90% of the time, the "management" boils down to rebooting the server, he said, since that solves most computing problems. IPMI simply lets administrators do so without having to walk all the way to the data center and back. The ability for a data center administrator to reboot a server even when the host OS does not want to reboot might be troubling—as could the fact that IPMI can tell the machine to boot from a different device (including network booting). But, Garrett said, the various vendor implementations are where things get truly "interesting" from a security standpoint, since vendors always want to add their own features beyond the specification.

Customizing the IPMI implementation is an attractive prospect for server vendors, he said, since it offers branding opportunities that cannot be removed, no matter what OS the customer installs, as well as the opportunity to incorporate various "value adds" to differentiate against rival vendors. Among the features server vendors add to IPMI are the ability to perform firmware updates, support for virtual hardware devices like a keyboard/monitor combination or CD drive, network service discovery, various web services, and what Garrett called "magic GPUs"—IPMI graphics adapters that can directly grab the contents of the server's actual GPU and make it available over the IPMI interface.

The magic GPUs and virtual hardware devices essentially allow the remote IPMI user to interact with the machine as if they were physically present—but bypassing the host OS entirely, with obvious potential for abuse by data-center administrators. The service-discovery and web-server features, in contrast, present potential attack vectors to people other than the system administrators who happen to be on the same network as the server.

Furthermore, IPMI is implemented on motherboards in a baseboard management controller (BMC), an embedded microcontroller that cannot be removed from the motherboard. BMC CPUs tend to be in the 600-to-800MHz range, Garrett said, with several megabytes of RAM. A variety of processor architectures are used, and almost all BMCs run Linux (with the notable exception of HP's, which run a specialty realtime OS).

The final piece of IPMI design that Garrett explained was the user-authentication scheme. The specification originally did not involve encryption at all, but it was updated in 2004. Now, it presents "a bewildering array of encryption and authentication protocols" from which to choose—but it still is not very good.

For one thing, he said, as part of the authentication handshake process, the BMC sends a hashed version of the correct password back to the user. Thus, all an attacker needs to do is guess one valid username, grab the hashed password, and run a cracking program against it. But even this is not always necessary, he added, since one of the options in the bewildering array is "Cipher Zero"—which, as the name suggests, involves no encryption or authentication at all. Luckily, he added, most vendors released an update to disable Cipher Zero, although not everyone has applied it.

Implementation details

The potential problems with IPMI and the various server-vendor–specific features are one set of threats, Garrett said, but another class entirely comes from implementation problems. Almost all IPMI implementations come from one of two vendors: Avocent (used in Dell, IBM, and Cisco machines) and AMI (used by almost everyone else). There is also a lot of common code shared between the two—for example, both use an embedded web server written by atWare and similar plugins. Bugs found in AMI's implementation of a basic feature tend to be found in Avocent's as well (and vice-versa). To their credit, he said, he has not yet found anything too egregious.

The server vendors' extensions, however, seem to be noticeably worse. They expose a lot of vulnerable interfaces, he said, such as CGI or PHP (the latter with extensions that allow embedded C code). They tend to leave a lot of services running, from SSH and Telnet to non-web GUI applications. Many servers support the Web Services-Management (WSMan) interface, which allows users to query server status with HTTP requests—and sometimes much more, like pushing new firmware. Garrett then referred the audience to firmware_config, a tool he developed for querying and setting firmware configuration options for a variety of IPMI-enabled server brands.

Garrett concluded with a tour of specific, egregious IPMI vulnerabilities he has encountered in the wild, noting that "you're here because you want to see me get really angry." The first example was from Dell's IPMI implementation: on that system, the IPMI console allows arbitrary command execution by appending commands as an "argument" to one of the real IPMI commands. So, for example, typing:

    firmware;/bin/sh

runs the firmware IPMI command, then gives the user a root shell. Another flaw involved the XML parser, which could be crashed and made to dump core by simply feeding it a set of tags in the wrong order, such as:

    </USERNAME><USERNAME>

Other commands, he said, are not written defensively—such as one instance of /sbin/pm that contained:

    "/bin/rm -f %s%s*"

which, he noted, might accidentally or maliciously be exploited to delete important files in certain circumstances.

It is very hard to "fix" IPMI implementations, Garrett said. BMCs are not removable and they cannot be switched off. They piggyback on motherboards' existing network ports and are configured to connect to the network at startup with DHCP. Furthermore, BMCs are fairly easy to discover, since they include SSL certificates that expose the server's serial number. Garrett offered an "educated guess" that scanning the entire IPv4 address space to look for server serial numbers in BMC SSL certificates could be done in well under an hour—gleaning around 100,000 servers. That information could be exploited by an attacker to track down owner information of all of the vulnerable servers.

The good news, he said, is that the industry has realized the problems with IPMI and is in the process of replacing it—although the replacement is not guaranteed to be better. In the meantime, Garrett advised server owners to put their BMCs onto a separate network, filter out all incoming IPMI traffic, and make sure all of the available updates get applied—quickly.

[The author would like to thank LCA 2015 for travel assistance to Auckland.]