How to Recover from a Corrupted Installation on FreeBSD Operating System

A corrupted FreeBSD installation can be a stressful situation for system administrators and users alike. Whether caused by hardware failure, improper shutdown, filesystem corruption, or botched updates, a system that refuses to boot properly requires methodical troubleshooting and recovery procedures. Fortunately, FreeBSD provides robust tools and mechanisms that can help you recover your system without resorting to a complete reinstallation.

This guide will walk you through a comprehensive recovery process, covering various corruption scenarios and their remedies. By following these steps, you’ll be equipped to diagnose and resolve most FreeBSD system corruption issues.

Understanding Types of System Corruption

Before diving into recovery procedures, it’s important to understand the different ways a FreeBSD system can become corrupted:

1. Boot loader corruption: Problems with the boot code in the MBR (Master Boot Record) or GPT (GUID Partition Table) preventing the system from initializing the boot process.

2. Filesystem corruption: Damage to UFS, ZFS, or other filesystem structures, often caused by improper shutdowns or power failures.

3. Kernel corruption: Problems with the kernel or kernel modules that prevent proper system operation.

4. Configuration corruption: Incorrect settings in critical configuration files like /etc/rc.conf, /etc/fstab, or boot loader configurations.

5. Package or software corruption: Issues with installed software or packages that render the system unusable.

Each type of corruption requires different recovery approaches, which we’ll explore in this guide.

Prerequisites for Recovery

Before beginning the recovery process, gather these essentials:

• FreeBSD installation media (USB drive or DVD) matching your installed version (or close to it)
• External storage device for backups (if possible)
• Hardware documentation for your system
• Network connectivity (helpful but not always required)
• Time and patience

Booting into Single-User Mode

For many recovery scenarios, single-user mode is your first option:

1. Restart your system
2. At the FreeBSD boot menu, select “Boot FreeBSD in Single-User Mode”
3. If prompted, enter the path to the shell (usually just press Enter to accept the default /bin/sh)

Single-user mode provides a minimal environment with the root filesystem mounted read-only, allowing for filesystem checks and basic repairs.

# Mount the root filesystem in read-write mode
mount -u /

# Mount other critical filesystems
mount -a -t ufs
swapon -a

If these commands succeed, your filesystem structure may be intact, and the issue could be related to configuration or software.

Using FreeBSD Recovery Media

If single-user mode is inaccessible, boot from your FreeBSD installation media:

1. Insert your FreeBSD installation media and boot from it
2. From the FreeBSD installer menu, select “Live CD” or “Shell” (depending on the version)
3. You’ll now have access to a complete FreeBSD environment for diagnosis and repair

Diagnosing Filesystem Corruption

Filesystem corruption is one of the most common issues. Here’s how to check and repair it:

For UFS Filesystems

1. Identify your partitions:

gpart show

2. Run filesystem check on the root partition (typically ada0p2 for the first disk’s second partition, but verify with gpart show):

fsck -y /dev/ada0p2

3. Check all UFS filesystems referenced in /etc/fstab:

fsck -y -t ufs

The -y flag automatically answers “yes” to all repair questions, which is usually safe but be cautious on critical systems.

For ZFS Filesystems

1. Import and check your ZFS pool (commonly “zroot”):

zpool import -f zroot
zpool status zroot

2. If errors are reported, try a repair:

zpool scrub zroot

3. For more severe corruption:

zpool clear zroot

4. If the pool cannot be imported normally, try recovery mode:

zpool import -fF zroot

Repairing Boot Loader Issues

If your system won’t boot due to boot loader corruption:

1. Boot from installation media and access the shell
2. For GPT-based systems:

gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0

3. For MBR-based systems:

gpart bootcode -b /boot/mbr ada0

4. If using ZFS with UEFI:

mount -t msdosfs /dev/ada0p1 /mnt
cp /boot/loader.efi /mnt/efi/boot/BOOTx64.efi
umount /mnt

Fixing Configuration Corruption

Improper configuration changes can prevent a system from booting properly:

For rc.conf Issues

1. Mount your root filesystem (if not already mounted):

mount /dev/ada0p2 /mnt

2. Edit the configuration file:

ee /mnt/etc/rc.conf

3. Look for syntax errors or problematic service configurations

For Loader Configuration

1. Examine and repair loader configuration:

ee /mnt/boot/loader.conf

2. Remove any problematic module loading or configuration entries

For fstab Issues

1. Check filesystem table configuration:

ee /mnt/etc/fstab

2. Verify UUIDs, device names, and mount options

Kernel and World Recovery

If your kernel or base system components are corrupted, you may need to reinstall them:

1. Boot from installation media
2. Mount your existing installation:

mount /dev/ada0p2 /mnt
mount -t devfs devfs /mnt/dev

3. Chroot into your installation:

chroot /mnt

4. Reinstall the kernel:

cd /usr/src
make kernel KERNCONF=GENERIC

5. If the entire base system needs rebuilding:

cd /usr/src
make buildworld
make installworld

Note that this requires having the FreeBSD source code available. If it’s not present, you can install it from installation media or fetch it using Git:

pkg install git
git clone https://git.freebsd.org/src.git /usr/src
cd /usr/src
git checkout release/13.2  # Replace with your FreeBSD version

Recovering from Failed Updates

Failed system updates can leave your system in an inconsistent state:

1. Boot into single-user mode or from installation media
2. Mount all filesystems and chroot if necessary
3. If a freebsd-update failed, you can try to roll back:

freebsd-update rollback

4. For ports or package issues:

pkg check -dsa
pkg upgrade -f

5. If the ABI changed (such as during a major version upgrade), reinstall all packages:

pkg-static upgrade -f

Encrypted System Recovery

If you’re using GELI encryption, additional steps are needed:

1. Boot from installation media
2. Load the GELI module:

kldload geom_eli

3. Attach your encrypted partitions:

geli attach /dev/ada0p2

4. Mount the decrypted filesystem:

mount /dev/ada0p2.eli /mnt

5. Proceed with the appropriate recovery methods

Data Recovery When All Else Fails

If you can’t recover the system but need to save your data:

1. Boot from live media
2. Mount your filesystems read-only to prevent further damage:

mount -o ro /dev/ada0p2 /mnt

3. Copy important data to an external drive:

mkdir /tmp/backup
mount /dev/da0 /tmp/backup  # Assuming da0 is your external drive
cp -Rpv /mnt/home/* /tmp/backup/

4. Consider using specialized data recovery tools:

pkg install testdisk recoverdm

Preventive Measures for the Future

After recovery, implement these preventive measures:

1. Regular backups using dump, tar, or rsync
2. For ZFS systems, regular snapshots:

zfs snapshot zroot/usr/home@daily

3. Configure periodic filesystem checks in /etc/periodic.conf
4. Use UPS (Uninterruptible Power Supply) for protection against power failures
5. Maintain a rescue USB drive with your system’s configuration files
6. Document your system’s partition layout and critical configurations

Testing Your Recovered System

Before considering your system fully recovered:

1. Reboot into normal multi-user mode
2. Check system logs for errors:

dmesg | grep -i error
cat /var/log/messages

3. Verify all filesystems are correctly mounted:

mount
df -h

4. Test critical system services:

service -e

5. Run basic system diagnostics:

uname -a
uptime
top -b

Conclusion

Recovering from a corrupted FreeBSD installation requires a systematic approach to diagnosis and repair. By understanding the different types of corruption and the appropriate recovery methods, you can often restore your system without resorting to a complete reinstallation.

Remember that prevention is always better than recovery. Implement regular backups, keep your system updated, and maintain documentation of your configuration to minimize downtime in case of future issues.

FreeBSD’s robust design and recovery tools make it resilient even in challenging corruption scenarios. With patience and the methodical application of the techniques outlined in this guide, you’ll be well-equipped to handle almost any system recovery situation.

Finally, consider contributing your experiences to the FreeBSD community. Your recovery story could help others facing similar issues and might even lead to improvements in the FreeBSD recovery tools and documentation.