PXEBoot and TFTP Troubleshooting¶

This guide covers low-level debugging techniques for TFTP and PXE boot issues in DRP. It walks the boot sequence in order, so work through it from the top when you do not yet know where the boot is failing.

Understanding the PXE Boot Process¶

Before troubleshooting, it helps to understand what happens during a typical successful PXE boot:

1. Machine sends DHCP discover broadcast
2. DRP DHCP server responds with IP address and boot filename
3. Machine requests bootloader via TFTP (port 69/udp)
4. Bootloader (iPXE/pxelinux) loads and requests configuration
5. Bootloader fetches kernel and initrd via TFTP or HTTP
6. Machine boots into the target boot environment (e.g., Sledgehammer for discovery)

DRP Server Diagnostic Commands¶

Before diving into step-by-step debugging, verify DRP is healthy:

# Verify DRP is running and listening on expected ports
drpcli info status

# Check if the API is responding
drpcli info check

Enable debug logging for more detail during troubleshooting:

# All service logging
drpcli prefs set logLevel debug

# Individual service logging
drpcli prefs set debugDhcp debug
drpcli prefs set debugStatic debug

# Remember to set back to the 'warn' default after debugging
drpcli prefs set logLevel warn
drpcli prefs set debugDhcp warn
drpcli prefs set debugStatic warn

Stream logs in real-time while attempting a boot:

drpcli logs watch

Watch file service requests as they arrive, which is the quickest way to tell whether a booting machine is reaching the endpoint at all:

drpcli events watch "tftp,static.serve.*"

See How do I confirm a machine's TFTP request actually reached DRP? for how to read these events.

Step 1: DHCP Discover¶

What happens: Machine sends DHCP discover broadcast.

Test DHCP Server is Responding¶

# Verify DHCP port is listening
ss -uln | grep 67

# Test DHCP from another Linux machine on the same network (requires root)
sudo dhclient -v -1 <interface>

Capture DHCP Traffic¶

tcpdump -i <interface> -n port 67 -vv

You should see DISCOVER, OFFER, REQUEST, and ACK packets.

Common Issues¶

• Firewall blocking port 67/udp

  Bash

  iptables -L -n | grep 67
  firewall-cmd --list-ports | grep 67
  

• Another DHCP server on the network - Check for unexpected OFFER packets in tcpdump output

Step 2: DHCP Response with Boot Filename¶

What happens: DRP DHCP server responds with IP address and boot filename.

Verify Subnet Configuration¶

# Check NextServer and the boot filename options
drpcli subnets show <subnet-name> | jq '{NextServer, Options}'

NextServer is empty when DRP nominates itself, which is correct for most single-homed deployments. A non-empty value that does not resolve to a reachable address on the machine's network stops the boot before any TFTP request is made.

Verify Boot Environment is Available¶

# List available boot environments
drpcli bootenvs list | jq '.[].Name'

# Check a specific boot environment exists and is valid
drpcli bootenvs show sledgehammer

Step 3: Bootloader Request via TFTP¶

What happens: Machine requests bootloader via TFTP (port 69/udp).

Verify TFTP is Enabled and Listening¶

drpcli info get | jq '{tftp_enabled, tftp_port}'
ss -uln | grep 69

tftp_enabled: false means the endpoint was started with --disable-tftp or RS_DISABLE_TFTP_SERVER, and nothing will answer on port 69.

Test TFTP End to End¶

drpcli info status performs a real TFTP fetch of lpxelinux.0 against the endpoint, so Alive: true confirms the full round trip rather than just an open port:

drpcli info status

Using tftp-hpa¶

# Install tftp client if needed
# RHEL/CentOS: yum install tftp
# Ubuntu/Debian: apt-get install tftp-hpa

# Test fetching the bootloader
tftp <drp-server-ip>
tftp> get lpxelinux.0
tftp> quit

# Verify file was retrieved
ls -la lpxelinux.0

Built-in Client¶

drpcli also has a built-in TFTP client, so no tftp-hpa package is needed. Run this from a host on the same VLAN as the machines that are failing:

drpcli static download lpxelinux.0 to /tmp/lpxelinux.0 --tftp
ls -la /tmp/lpxelinux.0

Confirm the Request Reaches DRP¶

drpcli events watch "tftp.serve.*"

Power on the machine. Any tftp.serve event naming the machine's IP proves the request arrived, including an event with "Status": "FAILED", which rules out firewalls, VLAN isolation, and routing as the cause. See How do I confirm a machine's TFTP request actually reached DRP?.

Capture TFTP Traffic¶

tcpdump -i any -n -vv 'host <client-ip> and udp'

Common Issues¶

• Firewall blocking port 69/udp

  Bash

  iptables -L -n | grep 69
  
  # Open TFTP port if needed
  iptables -A INPUT -p udp --dport 69 -j ACCEPT
  
  # For firewalld
  firewall-cmd --permanent --add-port=69/udp
  firewall-cmd --reload
  

• Firewall allowing only port 69 - The reply and all data blocks come from an ephemeral port. Stateless rules that permit only port 69 let the request in and drop every reply. Allow established UDP return traffic, or use connection tracking.

• Network isolation - Client and DRP server on different VLANs without proper routing

Step 4: Bootloader Fetches Configuration¶

What happens: Bootloader (iPXE/pxelinux) loads and requests its configuration script.

Test Configuration Fetch via TFTP¶

drpcli static download default.ipxe to /tmp/default.ipxe --tftp
cat /tmp/default.ipxe

A valid response contains iPXE commands like #!ipxe, kernel, and initrd.

Configuration files are requested in a waterfall, most specific name first, so requests for files that do not exist are expected rather than a fault. See Which files come over TFTP and which come over HTTP? for the request order per bootloader, and lpxelinux.0 error: no such file or directory for why the resulting "no such file or directory" messages are normal.

Check for Boot Environment Errors¶

# Verify templates are rendered correctly
drpcli bootenvs show sledgehammer | jq '.Templates'

Common Issues¶

• Missing or invalid templates - Check bootenv template syntax
• Machine not assigned a bootenv - Verify with drpcli machines show <uuid> | jq '.BootEnv'

Step 5: Kernel and Initrd Fetch¶

What happens: Bootloader fetches kernel and initrd via TFTP or HTTP.

Which protocol is used depends on the bootloader, so establish that before assuming TFTP is involved at all. Legacy BIOS pxelinux and UEFI Secure Boot shim/grub pull the kernel and initrd over TFTP; iPXE and UEFI HTTP Boot pull them over HTTP. See Which files come over TFTP and which come over HTTP? for the full matrix.

Test Boot File Fetch via TFTP¶

tftp <drp-server-ip>
tftp> get machines/[machine-uuid]/boot/[sledgehammer-uuid]/vmlinuz0
tftp> get machines/[machine-uuid]/boot/[sledgehammer-uuid]/stage1.img
tftp> quit

# Verify files were retrieved
ls -la vmlinuz0 stage1.img

TFTP is a lock-step protocol with no windowing, so kernels and initrds take minutes to transfer. A transfer that stalls and retries the same block is usually an MTU mismatch (see The transfer starts, then stalls and retries the same block).

Check ISO/File Server Content¶

# Check specific bootenv files are uploaded
drpcli isos list

Common Issues¶

• Missing ISO - Sledgehammer or OS ISO not uploaded

• Transfer stalls partway through - Block size exceeds the path MTU. See The transfer starts, then stalls and retries the same block

Step 6: Boot into Target Environment¶

What happens: Machine boots into the target boot environment (e.g., Sledgehammer for discovery).

Verify Machine Discovered Successfully¶

# List all machines
drpcli machines list | jq '.[] | {Name, Uuid, Stage, BootEnv}'

Check Machine Reached DRP¶

# View machine events/jobs
drpcli machines show <uuid> | jq '.CurrentJob'

References¶

• TFTP Debugging FAQ: symptom-driven answers, event watching, and packet capture recipes
• Discovery Troubleshooting: other discovery troubleshooting topics
• How dr-provision Boots Machines: how DRP decides what to serve a booting machine
• DHCP Architecture and External DHCP Architecture: DHCP architecture, internal and external
• TFTP Error: the benign TFTP Aborted messages
• lpxelinux.0 error: no such file or directory: the benign pxelinux.cfg waterfall messages