IPMI Notes

• (2023/03/31) Some details of how Intel BMCs access IPMB and PMBUS (page 70). Sample ipmi i2c commands. Yet more details with advice for capacitor-hacking!
• (2023/03/31) How to use a redundant power supply module from a server as a standalone power supply.
• (2022/07/30, 2020/12/18) Planned obsolescence. Also happens with AsRock server boards made before 2017. Do not try to update BIOS or BMC firmware. Travel back in time, install a VM with Ubuntu 14.04, install the default JRE (1.7) and icedtea plugin, prevent any updates, then use Firefox. (Ubuntu 16.04 is too recent. Somewhere during JRE 1.8, Firefox Java support was considered "too dangerous" and removed.)

• (2020/10/11) I decided to cobble together a RPi-based BMC for the unspeakable Asus WS-C621E-SAGE motherboard I had bought (see below), starting with these nice instructions and code, but because of the `upgrade' to Rpi3b+, I needed some extra customization. (Power button=GPIO2, reset button=GPIO3, power LED=GPIO4.)
  • As for the little PCB, it is better to clamp the transistor bases to emitters using 10kOhm resistors so that a floating RPi GPIO output pin does not `press' power and reset buttons when the RPi boots and before rc.local kicks in to lower these lines.
  • Python libraries on the Rpi to access GPIO gave strange trouble, so I used the simple raspi-gpio command line instead. E.g., a ``long press on the power button'' amounts to raspi-gpio set 2 dh; sleep 4; raspi-gpio set 2 dl.
  • The trickiest part is that Rpi3b+ has two different UART channels with slightly different capabilities. I had to disable bluetooth to free up the PL011 UART chip and disable getty.
  • Serial console to the motherboard is opened using the command screen -S console /dev/ttyAMA0 38400 after telling the motherboard likewise. The motherboard cannot seem to drive COM1 at faster rates without garbling output.
  • The motherboard has no control on rows and columns of console display; to make sense of its output I have to resize the terminal to exactly 100x33.
  • Linux needs the exact same grub options and boot command line to enable /dev/ttyS0 as grub console and getty target as with any real IPMI solution.
  Anyway, in the end the basic functions seem to work. Took about ten hours of research, soldering and tinkering.
• (2020/10/06) I (and others, in a manner) advise everyone to stay away from the Asus WS-C621E-SAGE motherboard. I bought one because it has four GPU slots and (supposedly) IPMI (via ASMB9-iKVM), despite being billed more as a workstation board. Unfortunately the vendor missed the IPMI part and the board was delivered without the ASMB9 card. No matter, I thought, I will just buy the ASMB9 card and plug it in. But that did not work.
  • The motherboard takes suspiciously long to POST if ASMB9 is present.
  • The IPMI card was preset to IP address 10.10.10.10 and changing it in the BIOS had no effect. I could not connect to 10.10.10.10 from 10.10.10.11 using a crossover Ethernet cable.
  • Naturally I went over to the Asus BIOS and BMC firmware page for this board. The latest BIOS version 6201 of 2020/07/31 installed properly using EzFlash.
  • I could find two ways to install the BMC firmware: booting DOS and running FLASH2.BAT; or running Yafuflash while booted in Linux. Both access the BMC through the KCS interface.
  • Asus provides no prominent link from where Yafuflash can be downloaded. It is unclear if the Intel version of Yafuflash uses the correct KCS commands with Asus motherboards.
  • FLASH2.BAT (or Yaflash.exe that it invokes) hangs for at least half an hour to write 36MB into BMC!
  • Running Yafuflash returns without incident but apparently does nothing.
  After wasting much time I gave up and hacked together the basic IPMI functions (power on/off, reset, serial console) using a Raspberry Pi. Intel manages RMM4LITE support better, and IPMI on ASRock and Supermicro boards just work. Here are some other woes I have faced with this board. Setting serial console to COM1 and setting its baud, stop, parity etc. parameters is occasionally ignored. The board has 8 SATA ports under one chip and 2 more under another. Intel SSDs attached to the 2-port chip are not visible one out of two boots. A cold reboot usually finds them, but sometimes I have needed two. Given the price of the board, all this is a real shame.
• (2020/10/05) Many Intel server motherboards provide basic IPMI which can be spruced up with KVM and remote media by adding the optional RMM4LITE module. But the combination shows some unpredictable behavior, e.g., inconsistent SOL support. To debug, check the output of these commands. Usually the basic IPMI is provided on channel 1 (Ethernet port shared with motherboard/OS) and the advanced support on channel 3 (dedicated Ethernet port the OS does not see).

  # ipmitool sol payload status 1 2
  # ipmitool sol payload status 3 2
  # ipmitool channel info 1
  # ipmitool channel info 3
  # ipmitool sol info 1
  # ipmitool sol info 3

• (2020/05/18) If you have a remote server with one Linux OS1 running and you need to install another OS2, but you do not have remote installation media facility on your IPMI/KVM hardware, you can first create a satisfactory virtual machine locally, collect the filesystem dump from there, restore to a second partition of the remove server, and then get into OS1 grub command line, where you have to maintain serial tty connection to OS2 as it boots, as follows (adjust to your hardware).

  gfxmode text
  insmod part_gpt
  insmod gzio
  insmod ext2
  search.file /etc/issue --set-root
  linux /boot/vmlinuz... root=... ro console=tty0 console=ttyS1,19200n8
  initrd /boot/initrd...
  boot
  

  Once booted to OS2, edit /etc/default/grub to reflect serial tty options, run update-grub, but do not let OS2 overwrite MBR/EFI (unless you wish to). Instead, reboot to OS1 and runs its update-grub so now it knows how to boot OS2 from grub menu.
• (2020/04/23) In case you are marooned beyond the reaches of IPMI and want to set up wake-on-LAN on a bonded interface.
• Different tools show IPMI SEL data in different formats, some more helpful for tracking down errant ECC RAM slots. Sometimes it is useful to dump SEL entries using ipmitool sel writeraw /tmp/host.sel and then display using ipmiutil events -h /tmp/host.sel.
• Many Intel-based motherboards with built-in BMC+IPMI offer to redirect the console to either ttyS0 (serial A) or ttyS1 (serial B), but ttyS1 does not actually work, or works only with weird tweaks. These same motherboards often have a rear panel DB9 connector only for ttyS0, and leave an IDC10 internal header on the motherboard for ttyS1. Once you use up ttyS0 for console redirection, you lose the DB9 connector. To use ttyS1 (e.g. with a SmartUPS), you need to install an internal serial cable from the IDC10 header to an RS232 male socket on a PCI bracket. Note that these cables come in several wiring variants. Here is one, PLATE9M16 from Startech, that will not work with most Intel and Supermicro motherboards. For more details see this tutorial.
• Some older Intel server boards (e.g. S5000VSA) have a confused relationship between the BMC and IPMI. COM2 or ttyS1 may not be found by grub; you may need to refer to it not as --unit=1 but as --port=0x2f8. If SOL is difficult to set up or locks up, follow these instructions. Also see this note about DCD and DTR on null modems. agetty may behave better than getty or mgetty.
• If some init scripts on your Debian-type system are mysteriously failing to start on boot, but can be started manually later with no problems, it could be because of a "misplaced" tty for the kernel. I was mistakenly booting a kernel on a motherboard without IPMI/SOL with kernel flags that used /dev/ttySn as console, and init scripts were randomly failing to start.