+ + +

Testbox Imaging (Backup / Restore)

Introduction

This document is explores deploying a very simple drive imaging solution to help +avoid needing to manually reinstall testboxes when a disk goes bust or the OS +install seems to be corrupted.

Definitions / Glossary

See AutomaticTestingRevamp.txt.

Objectives

+
+
Off site, no admin interaction (no need for ILOM or similar).
+
OS independent.
+
Space and bandwidth efficient.
+
As automatic as possible.
+
Logging.
+
+

Overview of the Solution

Here is a brief summary:

+
+
Always boot testboxes via PXE using PXELINUX.
+
Default configuration is local boot (hard disk / SSD)
+
Restore/backup action triggered by machine specific PXE config.
+
Boots special debian maintenance install off NFS.
+
A maintenance service (systemd style) does the work.
+
The service reads action from TFTP location and performs it.
+
When done the service removes the TFTP machine specific config +and reboots the system.
+
+

Maintenance actions are:

backup
backup-again
restore
refresh-info
rescue

Possible modifier that indicates a subset of disk on testboxes with other OSes +installed. Support for partition level backup/restore is not explored here.

How to use

To perform one of the above maintenance actions on a testbox, run the +testbox-pxe-conf.sh script:

+/mnt/testbox-tftp/pxeclient.cfg/testbox-pxe-conf.sh 10.165.98.220 rescue
+

Then trigger a reboot. The box will then boot the NFS rooted debian image and +execute the maintenance action. On success, it will remove the testbox hex-IP +config file and reboot again.

Storage Server

The storage server will have three areas used here. Using NFS for all three +avoids extra work getting CIFS sharing right too (NFS is already a pain).

+
+
/export/testbox-tftp - TFTP config area. Read-write.
+
/export/testbox-backup - Images and logs. Read-write.
+
/export/testbox-nfsroot - Custom debian. Read-only, no root squash.
+
+

TFTP (/export/testbox-tftp)

The testbox-tftp share needs to be writable, root squashing is okay.

We need files from both PXELINUX and SYSLINUX to make this work now. On a +debian system, the pxelinux and syslinux packages needs to be +installed. We actually do this further down when setting up the nfsroot, so +it's possible to get them from there by postponing this step a little. On +debian 8.6.0 the PXELINUX files are found in /usr/lib/PXELINUX and the +SYSLINUX ones in /usr/lib/syslinux.

The initial PXE image as well as associated modules comes in three variants, +BIOS, 32-bit EFI and 64-bit EFI. We'll only need the BIOS one for now. +Perform the following copy operations:

+cp /usr/lib/PXELINUX/pxelinux.0 /mnt/testbox-tftp/
+cp /usr/lib/syslinux/modules/*/ldlinux.* /mnt/testbox-tftp/
+cp -R /usr/lib/syslinux/modules/bios  /mnt/testbox-tftp/
+cp -R /usr/lib/syslinux/modules/efi32 /mnt/testbox-tftp/
+cp -R /usr/lib/syslinux/modules/efi64 /mnt/testbox-tftp/
+

For simplicity, all the testboxes boot using good old fashioned BIOS, no EFI. +However, it doesn't really hurt to be prepared.

The PXELINUX related files goes in the root of the testbox-tftp share. (As +mentioned further down, these can be installed on a debian system by running +apt-get install pxelinux syslinux.) We need the *pxelinux.0 files +typically found in /usr/lib/PXELINUX/ on debian systems (recent ones +anyway). It is possible we may need one ore more fo the modules [1] that +ships with PXELINUX/SYSLINUX, so do copy /usr/lib/syslinux/modules to +testbox-tftp/modules as well.

The directory layout related to the configuration files is dictated by the +PXELINUX configuration file searching algorithm [2]. Create a subdirectory +pxelinux.cfg/ under testbox-tftp and create the world readable file +default with the following content:

+PATH bios
+DEFAULT local-boot
+LABEL local-boot
+LOCALBOOT
+

This will make the default behavior to boot the local disk system.

Copy the testbox-pxe-conf.sh script file found in the same directory as +this document to /mnt/testbox-tftp/pxelinux.cfg/. Edit the copy to correct +the IP addresses near the top, as well as any linux, TFTP and PXE details near +the bottom of the file. This script will generate the PXE configuration file +when performing maintenance on a testbox.

Images and logs (/export/testbox-backup)

The testbox-backup share needs to be writable, root squashing is okay.

In the root there must be a file testbox-backup so we can easily tell +whether we've actually mounted the share or are just staring at an empty mount +point directory.

The testbox-maintenance.sh script maintains a global log in the root +directory that's called maintenance.log. Errors will be logged there as +well as a ping and the action.

We use a directory layout based on dotted decimal IP addresses here, so for a +server with the IP 10.40.41.42 all its file will be under 10.40.41.42/:

<hostname>: The name of the testbox (empty file). Help finding a testbox by name.
testbox-info.txt: Information about the testbox. Starting off with the name, decimal IP, +PXELINUX style hexadecimal IP, and more.
maintenance.log: Maintenance log file recording what the maintenance service does.
disk-devices.lst: Optional list of disk devices to consider backuping up or restoring. This is +intended for testboxes with additional disks that are used for other purposes +and should touched.
sda.raw.gz: The gzipped raw copy of the sda device of the testbox.
sd[bcdefgh].raw.gz: The gzipped raw copy sdb, sdc, sde, sdf, sdg, sdh, etc if any of them exists +and are disks/SSDs.
Note! If it turns out we can be certain to get a valid host name, we might just: switch to use the hostname as the directory name instead of the IP.

Debian NFS root (/export/testbox-nfsroot)

The testbox-nfsroot share should be read-only and must not have root +squashing enabled. Also, make sure setting the set-uid-bit is allowed by the +server, or su` and ``sudo won't work

There are several ways of creating a debian nfsroot, but since we've got a +tool like VirtualBox around we've just installed it in a VM, prepared it, +and copied it onto the NFS server share.

As of writing debian 8.6.0 is current, so a minimal 64-bit install of it was +done in a VM. After installation the following modifications was done:

+
+
apt-get install pxelinux syslinux initramfs-tools zip gddrescue sudo joe +and optionally apt-get install smbclient cifs-utils.
+
+
/etc/default/grub was modified to set GRUB_CMDLINE_LINUX_DEFAULT to +"" instead of "quiet". This allows us to see messages during boot +and perhaps spot why something doesn't work on a testbox. Regenerate the +grub configuration file by running update-grub afterwards.
+
+
/etc/sudoers was modified to allow the vbox user use sudo without +requring any password.
+
+
Create the directory /etc/systemd/system/getty@tty1.service.d and create +the file noclear.conf in it with the following content:
+
+[Service]
+TTYVTDisallocate=no
+
+
This stops getty from clearing VT1 and let us see the tail of the boot up +messages, which includes messages from the testbox-maintenance service.
+
+
Mount the testbox-nfsroot under /mnt/ with write privileges. (The write +privileges are temporary - don't forget to remove them later on.):
+
+mount -t nfs myserver.com:/export/testbox-nfsroot
+
+
Note! Adding -o nfsvers=3 may help with some NTFv4 servers.
+
+
Copy the debian root and dev file system onto nfsroot. If you have ssh +access to the NFS server, the quickest way to do it is to use tar:
+
+tar -cz --one-file-system -f /mnt/testbox-maintenance-nfsroot.tar.gz . dev/
+
+
An alternative is cp -ax . /mnt/. && cp -ax dev/. /mnt/dev/. but this +is quite a bit slower, obviously.
+
+
Edit /etc/ssh/sshd_config setting PermitRootLogin to yes so we can ssh +in as root later on.
+
+
chroot into the nfsroot: chroot /mnt/
+
+
+
mount -o proc proc /proc
+
+
mount -o sysfs sysfs /sys
+
+
mkdir /mnt/testbox-tftp /mnt/testbox-backup
+
+
Recreate /etc/fstab with:
+
+proc                             /proc               proc  defaults   0 0
+/dev/nfs                         /                   nfs   defaults   1 1
+10.42.1.1:/export/testbox-tftp   /mnt/testbox-tftp   nfs   tcp,nfsvers=3,noauto  2 2
+10.42.1.1:/export/testbox-backup /mnt/testbox-backup nfs   tcp,nfsvers=3,noauto  3 3
+
+
We use NFS version 3 as that works better for our NFS server and client, +remove if not necessary. The noauto option is to work around mount +trouble during early bootup on some of our boxes.
+
+
Do mount /mnt/testbox-tftp && mount /mnt/testbox-backup to mount the +two shares. This may be a good time to execute the instructions in the +sections above relating to these two shares.
+
+
Edit /etc/initramfs-tools/initramfs.conf and change the MODULES +value from most to netboot.
+
+
Append aufs to /etc/initramfs-tools/modules. The advanced +multi-layered unification filesystem (aufs) enables us to use a +read-only NFS root. [3] [4] [5]
+
+
Create /etc/initramfs-tools/scripts/init-bottom/00_aufs_init as +an executable file with the following content:
+
+#!/bin/sh
+# Don't run during update-initramfs:
+case "$1" in
+    prereqs)
+        exit 0;
+        ;;
+esac
+
+modprobe aufs
+mkdir -p /ro /rw /aufs
+mount -t tmpfs tmpfs /rw -o noatime,mode=0755
+mount --move $rootmnt /ro
+mount -t aufs aufs /aufs -o noatime,dirs=/rw:/ro=ro
+mkdir -p /aufs/rw /aufs/ro
+mount --move /ro /aufs/ro
+mount --move /rw /aufs/rw
+mount --move /aufs /root
+exit 0
+
+
+
Update the init ramdisk: update-initramfs -u -k all
+
+
Note! It may be necessary to do mount -t tmpfs tmpfs /var/tmp to help
+
this operation succeed.
+
+
+
+
Copy /boot to /mnt/testbox-tftp/maintenance-boot/.
+
+
Copy the testbox-maintenance.sh file found in the same directory as this +document to /root/scripts/ (need to create the dir) and make it +executable.
+
+
Create the systemd service file for the maintenance service as +/etc/systemd/system/testbox-maintenance.service with the content:
+
+[Unit]
+Description=Testbox Maintenance
+After=network.target
+Before=getty@tty1.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=True
+ExecStart=/root/scripts/testbox-maintenance.sh
+ExecStartPre=/bin/echo -e \033%G
+ExecReload=/bin/kill -HUP $MAINPID
+WorkingDirectory=/tmp
+Environment=TERM=xterm
+StandardOutput=journal+console
+
+[Install]
+WantedBy=multi-user.target
+
+
+
Enable our service: systemctl enable /etc/systemd/system/testbox-maintenance.service
+
+
xxxx ... more ???
+
+
Before leaving the chroot, do mount /proc /sys /mnt/testbox-*.
+
+
+
+
+
Testing the setup from a VM is kind of useful (if the nfs server can be +convinced to accept root nfs mounts from non-privileged clinet ports):
+
+
+
Create a VM using the 64-bit debian profile. Let's call it "pxe-vm".
+
+
Mount the TFTP share somewhere, like M: or /mnt/testbox-tftp.
+
+
Reconfigure the NAT DHCP and TFTP bits:
+
+VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/AboveDriver       NAT
+VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Action            mergeconfig
+VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Config/TFTPPrefix M:/
+VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Config/BootFile   pxelinux.0
+
+
+
Create the file testbox-tftp/pxelinux.cfg/0A00020F containing:
+
+PATH bios
+DEFAULT maintenance
+LABEL maintenance
+  MENU LABEL Maintenance (NFS)
+  KERNEL maintenance-boot/vmlinuz-3.16.0-4-amd64
+  APPEND initrd=maintenance-boot/initrd.img-3.16.0-4-amd64 ro ip=dhcp aufs=tmpfs \
+         boot=nfs root=/dev/nfs nfsroot=10.42.1.1:/export/testbox-nfsroot
+LABEL local-boot
+LOCALBOOT
+
+
+
+
+
+
+

Troubleshooting

PXE-E11 or something like No ARP reply: You probably got the TFTP and DHCP on different machines. Try move the TFTP +to the same machine as the DHCP, then the PXE stack won't have to do any +additional ARP resolving. Google results suggest that a congested network +could use the ARP reply to get lost. Our suspicion is that it might also be +related to the PXE stack shipping with the NIC.

+ + + + + +

[1]	See http://www.syslinux.org/wiki/index.php?title=Category:Modules

+ + + + + +

[2]	See http://www.syslinux.org/wiki/index.php?title=PXELINUX#Configuration

+ + + + + +

[3]	See https://en.wikipedia.org/wiki/Aufs

+ + + + + +

[4]	See http://shitwefoundout.com/wiki/Diskless_ubuntu

+ + + + + +

[5]	See http://debianaddict.com/2012/06/19/diskless-debian-linux-booting-via-dhcppxenfstftp/

+ +++ + + + + + +

Status:	$Id: TestBoxImaging.html $
Copyright:	Copyright (C) 2010-2023 Oracle Corporation.