Recently I bought several refurbished Lenovo Thinkcentre M73 Tiny for my home lab. Instead of travelling the easy route I wanted to set up Cobbler for remote provisioning the machines. That makes some sense because at least in theory I could re-install the machines with different Linux distros without having to hook them to a display, a keyboard and boot them from USB sticks.
The first obstacle I had to overcome was setting up DHCP/BOOTP for remote booting. I have a flat private network behind a Fritzbox that doesn’t offer many options for DHCP. Also I didn’t want to configure a separate lab subnet because that would require routing from one network to the other. So my idea was to use something that could be called split-horizon DHCP: essentially that means that the standard DHCP server on the router ignores several specific MAC addresses while another machine that acts as the installation server does the opposite.
Obviously the consumer router software on the Fritzbox doesn’t offer such advanced features so I thought about installing OpenWRT. That isn’t supported on the Fritzbox but the OpenWRT site led me to the Freetz open source project that takes the original Fritzbox firmware and augments it with custom modules, among many others a dnsmasq module. Dnsmasq is able to provide DNS and DHCP services and offers many configuration options. The Freetz build configuration menu also offers the possibility to remove selected official Fritzbox modules which enabled me to work around the memory constraints of the router and successfully create a bootable image with dnsmasq and ssh. I disabled the factory default DHCP server and managed to run the home network with dnsmasq quite easily.
To have the lab machines ignored in the regular DHCP on the Fritzbox dnsmasq I use these lines:
dhcp-ignore=tag:tobeignored
dhcp-mac=set:tobeignored,44:8a:5b:dd:c5:26
dhcp-mac=set:tobeignored,44:8a:5b:dd:67:dc
dhcp-mac=set:tobeignored,00:23:24:81:6c:42
Cobbler installation is almost easy apart from the usual quirks. Makes you wonder why they still can’t have the installer set the required SELinux boolean after all those years, but well .. then download an Ubuntu 18.04 iso and import the distro into Cobbler. I used dnsmasq with Cobbler, too, so I didn’t have to care for ISC DHCPD configuration. To make Cobbler ignore the non-managed hosts you set up dnsmasq like this (in “/etc/cobbler/dnsmasq.template”, then “cobbler sync”):
dhcp-host=net:x86_64,44:8a:5b:dd:c5:26,192.168.2.201
dhcp-host=net:x86_64,44:8a:5b:dd:6c:42,192.168.2.203
dhcp-host=net:x86_64,44:8a:5b:dd:67:dc,192.168.2.202
dhcp-ignore=tag:!known
Also be sure to set “pxe_just_once” in “/etc/cobbler/settings” to reset the configuration for network install so you don’t end up in an endless network installation loop.
Then the fun begins: trying to figure out the preseed options with no decent documentation. Can you believe that there’s not a single official web page that lists all options the Debian installer offers?! The internet is full of preseed examples and desperate people trying to figure out what works and what doesn’t through the evolution of decades of Debian installer development. In an obscure StackExchange post you can learn that there’s a brave soul who wrote a Haskell program to extract the available options from the installer templates.
Then there’s the “expert_recipe_file” of the partitioner. Tim Bishop has written a blog post that does a great job of explaining how to provide the required values for min/max size and weights.
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/alignment string "optimal"
d-i partman-auto-lvm/guided_size string max
d-i partman-efi/non_efi_system boolean true
d-i partman-basicfilesystems/choose_label string gpt
d-i partman-basicfilesystems/default_label string gpt
d-i partman-partitioning/choose_label string gpt
d-i partman-partitioning/default_label string gpt
d-i partman/choose_label string gpt
d-i partman/default_label string gpt
d-i partman-auto/expert_recipe string \
boot-root :: \
538 538 1075 free \
$iflabel{ gpt } \
$primary{ } \
$reusemethod{ } \
method{ efi } \
format{ } \
. \
128 100 256 ext3 \
$defaultignore{ } \
$primary{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext2 } \
mountpoint{ /boot } \
. \
8192 8241 8192 linux-swap \
$lvmok{ } \
lv_name{ swap } \
method{ swap } format{ } \
. \
65536 65538 -1 ext4 \
$lvmok{ } \
lv_name{ root } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ / } \
. \
409600 409602 500600 ext4 \
$lvmok{ } \
lv_name{ storage } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /storage } \
.
To have the installer reset the network installation as mentioned above you need to add a specific snippet at the end of the preseed file:
$SNIPPET('kickstart_done')
Many times the installation failed because the installer couldn’t find a library that was required by the ntp package. In the end the only way to solve this problem was to remove the Apt sources that Cobbler injects into the preseed files and have the installer download the packages from the official Ubuntu repositories. Depending on your setup that might require to also change the preseed template of the Cobbler “profile” because it gets used in addition to the preseed file of the actual “system”.
After several hours I finally got one of the lab computers to boot and install via PXE but for unknown reasons it failed to boot. I managed to track the problem down to a missing LVM module in the initial ramdisk but gave up on this problem because I thought it might be “better” to boot the machine from UEFI anyway.
After much trying and erroring with the partitioner I learned that it only creates an EFI partition when the machine is booted via EFI, in our case that is via the network (also mandatory: partitioning with GPT). So I tried my luck setting up Cobbler to enable remote booting an EFI firmware. You can set up dnsmasq to set the boot file depending on the architecture that the network booted computer provides which in turn is determined by its BIOS settings.
dhcp-boot=net:normalarch,pxelinux.0
dhcp-match=set:efi-x86_64,option:client-arch,7
dhcp-boot=tag:efi-x86_64,grub/grubx64.efi
To make a long story short the Grub loaders that Cobbler ships are only Grub version 1 and I didn’t get them to work. What finally worked was to take a Grub 2 EFI image (“grubx64.efi”) and provide the appropriate “grub.cfg” that loads the Linux kernel and the initial ram disk from the Ubuntu distro and starts the installer. Debugging this was not easy because when you get Grub loaded the UEFI firmware tells you that it found the “NBP” (network boot program) and then delivers no more clues about what works and what doesn’t. It did help to run the TFTP service on the Cobbler server in verbose mode that at least logged when the NBP requested a file that couldn’t be found. I didn’t enable secure boot but during debugging I also managed to first boot the “shim.efi” (from the shim package) that chain loads Grub 2, so that might be a way to go.
This way I finally got a EFI booted Linux installation with LVM partitions remotely installed. Too bad the BIOS got reset to boot from hard disk by default after the installation because this made all my efforts futile (see first paragraph). Setting the boot order with the “efibootmgr” on Linux also didn’t work. On the Lenovo site I found some BIOS updates for the Thinkcentre machines that had a specific “Respect the UEFI Boot Order” bugfix. Nice. Until I understood that this was for the “tower and small form factor” boxes while I own “tiny” machines. They didn’t list the specific bugfix for the “tiny” updates but you never know what hides behind a “maintenance release” so I tried the BIOS update. Unfortunately for the M73 Tiny there’s neither a way to use the Linux firmware interface for updates nor a UEFI capsule that you could install on the EFI partition and run with the bootloader. So the only way I got that to work was write the BIOS update image to a USB stick and update all three machines manually. Lucky for me the BIOS patches did seem to work.
Now I can remote install the machines from my Cobbler box without having to attach monitor and keyboard or taking the boxes out of the cabinet. With the Wake on LAN function I can even remote start them from my laptop.
All in all I think it’s quite messy to get all of this stuff working and it took me days and a lot of endurance to do it. Granted it is a bit of a special setup but remote booting and installing things like LVM and EFI shouldn’t be so hard.