This document gather some notes on how to setup booting a Raspberry Pi (3B+ and later) over network.

1. U-Boot on RPi waits 10s before trying the DHCP (?speedup by boot order?)
2. DHCP IP requests is sent over to DHCP server (target address is 255.255.255.255 - IPv4 broadcast, IPv6 is available starting RPi 4B+)
3. DHCP server reponds with IPv4 and few more options (43,66,67 - where only the 66/next-server is required)
4. RPi using the option 66 requests bootcode.bin
5. The bootcode.bin is executed and extra files are requested from TFTP server
6. When all files are

The following was recorded by tftpd-hpa daemon on Ubuntu 20.04. Some files are listed and requested twice and not all files are present in boot.tar.xz or BOOT/FAT partition on your micro SD card or image.

Note: the <serial> is placeholder for actuall serial number of your RPi.

The whole process spans across period of aprox. 3 seconds running on 1Gbps network (though RPi 3B+ has USB LAN chip capable of ~ 300Mbps).

For details see #Boot folder and/or Raspberry Pi

Important files:

cmdline.txt - points to the NFS and sets the kernel options including splash and quiet (ref.conf/cmdline.txt.sample) which could look like:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=100.64.0.1:/nfs/client1,vers=4.2,proto=tcp rw ip=dhcp rootwait elevator=deadline quiet splash

config.txt - the usual RPi config (as found on FAT /boot partition)

files present in boot.tar.xz:

bcm2710-rpi-cm3.dtb
bcm2710-rpi-zero-2.dtb
bcm2710-rpi-zero-2-w.dtb
bcm2710-rpi-2-b.dtb
bcm2710-rpi-3-b.dtb
bcm2710-rpi-3-b-plus.dtb
bcm2711-rpi-cm4.dtb
bcm2711-rpi-cm4s.dtb
bcm2711-rpi-4-b.dtb
bcm2711-rpi-400.dtb
cmdline.txt
config.txt
COPYING.linux
fixup_cd.dat
fixup.dat
fixup_db.dat
fixup_x.dat
fixup4cd.dat
fixup4.dat
fixup4db.dat
fixup4x.dat
issue.txt
kernel8.img
LICENCE.broadcom
overlays
start_cd.elf
start_db.elf
start.elf
start_x.elf
start4cd.elf
start4db.elf
start4.elf
start4x.elf

Some manapages (eventually search man dnsmasq):

• man dnsmasq to provide DHCP, TFTP and DNS
• man rpcinfo to list registered SUN RPC services
• man exportfs re-export -r as root and show -v
• man tar
• man losetup
• man rsync

Ubuntu server 20.04 LTS

as root (sudo -i):

apt install nfs-kernel-server dnsmasq tcpdump

cat >>/etc/exports <<DATA
/nfs *(rw,sync,no_subtree_check,no_root_squash)
/tftp *(rw,sync,no_subtree_check,no_root_squash)
DATA

To list available SUN RPC services on local computer rpcinfo.

Run the services:

• dnsmasq provides the DHCP(68), TFTP(69) and DNS(53) services (ports)
• rpcbind provides service 2 port mapping for SUN RPC
• nfs-kernel-server is the file service (use of NFS v4.2 over TCP is recommended)

#
systemctl enable dnsmasq
systemctl restart dnsmasq
#
systemctl enable rpcbind
systemctl restart rpcbind
#
systemctl enable nfs-kernel-server
systemctl restart nfs-kernel-server
#
systemctl enable nfs-server
systemctl restart nfs-server

• prepare_pxetools.sh
• pxetools.py

To get serial number cat /proc/cpuinfo which is then used in TFTP boot or can be observed in dnsmasq logs when enabled as file request path .../<serial-number>/start4.elf

${TFTP_ROOT}/bootcode.bin is the bootcode downloaded by RPi U-Boot from rom, which in turn loads the rest from ${TFTP_ROOT}/<serial-number>/...

• Main download space @ downloads.raspberrypi.org
• raspios_lite_arm64/boot.tar.xz - extract to ${TFTP_ROOT}/orig
• raspios_lite_arm64/root.tar.xz - extract to ${NFS_ROOT}/orig

The boot partition of image 2022-04-04-raspios-bullseye-arm64-lite.img.xz to expand unxz *.img.xz and kpartx -a *.img to prepare loopbacks and which then can be mounted via mount /dev/mapper/loopXpY where Y=1 = /boot and Y=2 = /.

Or use the pre-prepared tar.xz note use GNU tar (BSD does not have all options, also MacOS X)

raspios_lite_arm64 @ RFP/boot.tar.cz

mkdir -p ${TFTP_ROOT}/orig
tar -xvJ --xattrs --acls -f boot.tar.xz --directory=${TFTP_ROOT}/orig

raspios_lite_arm64 @ RFP/root.tar.cz

mkdir -p ${NFS_ROOT}/orig
tar -xvJ --xattrs --acls -f root.tar.xz --directory=${NFS_ROOT}/orig

Note: x - extract, v - verbose, J - use unxz, directory - extract to this directory/folder

Then copy ${TFTP_ROOT}/orig/bootcode.bin to ${TFTP_ROOT}/bootcode.bin to let the U-Boot to find it.

The rest of ${TFTP_ROOT}/orig/. should be copied to per RPi sub-folder like ${TFTP_ROOT}/<serial-number>/.

Make a copy and update conf/cmdline.txt.sample to ${TFTP_ROOT}/<serial-number>/ to update:

nfsroot=100.64.0.1:/nfs/client1 to reflect the roof filesystem path on NFS server.

you might add quiet and splash and eventually ``

Install splash screen (plymouth) sudo apt -y install rpd-plym-splash (ref.plymouth.ignore_serial_console see rpi/issue)

• RPi Chromium dashboard
• Pi Kiosk
• Web dashboard
• Another dashboard
• RPi 4 wall dashboard
• OctoDash
• Productivity dashboard
• understand cmdline.txt
• Raspberry Valley - inspiration via RPi Leds

The root filesystem should be copied by 11-replicate_nfs.sh for each client rsync -av --info=progress2 --xattrs --acls "${SRC_FS}/." "${DST_FS}/". Some experiments were done on using overlayfs so far without success.

• Pi Server
• deprecated Pi Net
• LTSP - linux terminal server project
• Domoticz OverlayFS
• Extended Attrs in NFS

RPF = Raspberry Pi foundation

• Leds on raspi
• RPF docs on net boot

source ~/.config/raspi/netboot/config.inc
iptables -I FORWARD -i ${IF_NAME} \! -o ${IF_NAME} -j ACCEPT
iptables -I FORWARD \! -i ${IF_NAME} -o ${IF_NAME} -j ACCEPT
iptables -t nat -I POSTROUTING -s ${SUBNET} \! -o ${IF_NAME} -j MASQUERADE

Swap does not make much sense when RPi is netbooted

# as root "sudo -i"
sync
swapoff -a
apt-get purge -y dphys-swapfile
rm /var/swap
sync

sudo mount --bind /mnt/nvme/nfs /nfs

To stop all the rpcbind service to port mapper (locking, fileserver, ...)

#
systemctl stop rpcbind
systemctl disable rpcbind
systemctl mask rpcbind
#
systemctl stop rpcbind.socket
systemctl disable rpcbind.socket
#
systemctl status rpcbind

In general it is recomended to use DNS (as It's Always DNS)

address=/pi-server/192.168.x.y

DHCP Proxy explainer @ FOG