Linux for Newbies, pt. 23:
Disaster Recovery Tips
by Gene Wilburn(The Computer Paper, June 2001. Copyright © Wilburn Communications Ltd. All rights reserved)
No matter which operating system you're using, at some point you'll be faced with an emergency recovery operation. Your MBR gets overwritten, your hard disk fails, an important file gets accidentally deleted, your video card gives one last flicker and dies. No matter the cause, you need some recovery procedures. Here are a few tips and techniques for some common mishaps.
The first item you need in your disaster recover kit is a floppy boot diskette. You probably created one when you first installed Linux, but do you know where it is? Take it for a test drive to make sure it still lets you boot into your system. Need a new one? Let's look at two options. (For any newcomers, this series is based on Red Hat 6.X Linux and some of the following procedures are Red Hat specific.)
Emergency Boot Diskettes
What makes the Red Hat boot disk you created for your system particularly valuable is that it contains any special drivers, such as SCSI drivers, that may be required to boot your system. If you need a new boot disk or would like to keep a second one on hand just in case, you can create a new boot disk by logging in as root and executing the command mkbootdisk. You need to know your kernel version and your floppy device (usually /dev/fd0). Use the uname command to display your kernel version as in this example session:# uname -a Linux frodo 2.2.14-5.0 #1 Tue Mar 7 20:53:41 EST 2000 i586 unknown # mkbootdisk --device /dev/fd0 2.2.14-5.0 Insert a disk in /dev/fd0. Any information on the disk will be lost. Press
to continue or ^C to abort:
Insert the disk, wait a bit for the boot disk to be built, and now you have a working boot disk. Test it, then store it in a safe place.
A second alternative (it doesn't hurt to use both) is to create a Tom's Root Boot diskette. The tomsrtbt distribution may be one of the coolest items on the Internet: a complete Linux system on a single diskette. To create a tomsrtbt diskette, set your browser to www.toms.net/rb/ and download the latest version. To prepare an emergency boot disk, place the zipped tar file into a directory such as /usr/local/src and follow the install procedures:# gunzip tomsrtbt-1.7.218.tar.gz # tar -xvf tomsrtbt-1.7.218.tar # rm tomsrtbt-1.7.218.tar # cd tomsrtbt-1.7.218/ # ./install.s
Just follow the onscreen instructions. The package uses high compression to squeeze an amazing number of Linux utilities onto a single 1.7Mb floppy. You may go through a blank floppy or two before you find one that can stand up to the compression. The program does a careful check to make sure that the sectors are packed correctly and that they can be read.
After you've created the diskette, read the tomsrtbt.FAQ file in the install directory to get a full appreciation of the product and learn some ways to customize it. The default disk contains the following programs, drivers and kernel:
2.0.37 3c589_cs BusLogic DEC_ELCP EEXPRESS EEXPRESS_PRO EL2 EL3 EXT2 FAT FAT32 FD IDE IDECD IDEFLOPPY IDEPCMCIA IDETAPE ISO9660 JOLIET LOOP MATH_EMULATION MINIX MSDOS NE2000 NFS PROC RAM SD SERIAL SLIP SMC SR ST TR ULTRA VFAT VORTEX WD80x3 ah152x_cs aha152x aha1542 aic7xxx ash awk badblocks bdflush bzip2 cardbus cardmgr cat ce ce.help chattr chgrp chmod chown chroot clear cmp cp cpio cut date dd ddate debugfs df dirname dmesg ds du dumpe2fs e2fsck eata echo egrep elvis emacs extend false fdflush fdformat fdisk fdomain filesize find findsuper fmt fsck.ext2 fsck.msdos fstab grep gzip halt head hexedit hostname i82365 ifconfig ifport ile init inittab insmod kill killall5 ksyms length less libc.so.5.4.13 lilo lilo.conf ln loadkeys login losetup ls lsattr mawk md5sum memtest mingetty miterm mkdir mkdosfs mke2fs mkfifo mkfs.minix mklost+found mknod mkswap mnsed more mount mt mv nc ncr53c8xx nmclan_cs ntfs pax pcmcia pcmcia_core pcnet_cs ping plip ppa printf ps pwd qlogic_cs qlogicfas reboot rescuept reset rm rmdir rmmod route rsh rshd script scsi_info seagate sed serial_cs setserial sh slattach sleep slip snarf sort split stty swapoff swapon sync tail tar tcic tee telnet test touch tune2fs umount undeb update vi vi.help wc
Impressive, and even better, tomsrtbt is not distribution specific. In fact you can even use it to perform certain rescue operations on Windows 9X.
To The Rescue
Let's look at a common mishap: in this scenario you have a dual-boot Linux/Windows computer and you happily use LILO in your master boot record partition (MBR) to boot to one OS or the other. You decide to upgrade or re-install your Windows setup. Zap! There goes the MBR (Windows ALWAYS zaps the MBR when it installs even though it's unnecessary and in decidedly poor form).
Now you're in a pickle. Your Linux partition is still fine, but it's inaccessible. How do you get back into Linux in order to re-run lilo so that it can rewrite the multi-boot loader into the MBR? Why with your boot disk, of course.
With the Red Hat boot disk, it's a piece of cake. Just insert the floppy, boot, log in as root and type lilo at the command line. Everything's back to normal.
You can also re-run lilo using tomsrtbt, but the procedure is slightly different. Because it's not a boot disk for your setup, you must mount the Linux partition(s) and chroot to your root filesystem before running LILO. To perform this procedure with tomsrtbt, pop in the emergency boot disk, reboot, log in as root, then remove the floppy disk. Here is a sample LILO recovery session in which the Linux root system is /dev/hda4:# mount -t ext2 /dev/hda4 /mnt # chroot /mnt # /sbin/lilo Added linux * Added dos # exit # cd / # umount mnt
Prior to release 6.2 Red Hat offered a rescue.img that could be used to create a separate "rescue disk" used in conjunction with the boot disk. When the LILO: prompt appeared, you would type in linux rescue and follow the instructions. From 6.2 onwards Red Hat has moved the rescue function over to its installation CD. You boot with the CD then type "rescue" at the first prompt.
This method doesn't work well with old BIOSes that cannot boot from CD-ROM. You first have to boot with the installation diskette, then type "rescue" then insert the CD-ROM. It's often easier to perform these rescue operations with tomsrtbt instead.
To The Rescue, Continued
Here's another situation: you've accidentally made a typo in /etc/fstab when you reconfigured your system. You meant to write /dev/hda4 as your boot drive but accidentally entered /dev/hda44. Your system is now unable to boot.
Pop in a tomsrtbt floppy, reboot, log in as root, remove the floppy, and proceed as follows:# mount -t ext2 /dev/hda4 /mnt # vi /mnt/etc/fstab # umount /mnt
Another common mishap that affects computers is being shutdown incorrectly, say when the lights go out during a power outage and your hard disk suffers some corruption as a consequence.
You can use tomsrtbt to try to fix a corrupted partition (e.g. /dev/hda2). Boot with the tomsrtbt floppy and work on the affected partition using e2fsk:# e2fsk -f /dev/hda2
For a badly damaged partition, this command may need to be repeated several times.
If a partition can't be fully repaired, you can sometimes grab off an important file or two anyway, writing them to floppy. When tomsrtbt boots it loads itself completely into memory, freeing up the floppy drive for other uses. Here's an example session showing how you can use a tomsrtbt instance to format a floppy in Linux EXT2 format, mount the newly formatted floppy, mount a HD partition, then copy some important files to the floppy:# fdformat /dev/fd0H1440 # mke2fs /dev/fd0 # mkdir /mnt/image # mkdir /mnt/floppy # mount -t ext2 /dev/hda4 /mnt/image # mount -t ext2 /dev/fd0 /mnt/floppy # cd /mnt/floppy # cp /mnt/image/etc/*.conf . # cp /mnt/image/etc/conf.modules . # cp /mnt/image/var/named/* . # cp /mnt/image/home/gwilburn/docs/myrecipes.txt . # cd /mnt # umount floppy # umount image
The preceding session used the fdformat, mke2fs, mount, cd and cp commands, all of which are included on tomsrtbt. In this session we rescued our configuration files from /etc, our DNS files from /var/named and an important user-created file.
Odd as it may seem, tomsrtbt can be used a rescue/salvage disk for Windows. Here's a sample session to rescue some files from a Windows system that no longer boots (assuming you've got some spare DOS-formatted diskettes on hand). You can also edit the autoexec.bat file while you're there:# mkdir /mnt/win # mkdir /mnt/floppy # mount -t vfat /dev/hda1 /mnt/win # mount -t vfat /dev/fd0 /mnt/floppy # vi /mnt/win/autoexec.bat # cd /mnt/floppy # cp /mnt/win/autoexec.bat . # cp /mnt/win/config.sys . # cp "/mnt/win/program files/netscape/bookmarks.htm" . # cd /mnt # umount floppy # umount win
By now you should be getting the message about tomsrtbt. Tuck one into your briefcase, knapsack or shirt pocket and carry it around as faithfully as your cell phone. You never know when you might need it. And imagine your friends' astonishment when you help them recover a Windows file using "pocket Linux".
Backup is your Friend
While it's good to be prepared for rescue operations, there are times when the damage is beyond repair. If you accidentally delete a file in Linux, for instance, kiss it goodbye. There are some convoluted methods of trying to recover deleted files, but they're rarely successful.
Worse, hard disks tend to fail all at once, with no warning. No matter your operating system of choice, there's no substitute for good backup.
Good backup is your best friend. It's not very glamorous or exciting and it takes a little effort, but you only need to lose your work once before you become one of the converted.
At the very least, you can back up certain files as you create them by copying them to floppy. This works for small files such as configuration files and other text files. Most Linux distributions include a set of tools called mtools. The mtools can be used to work with Windows floppies, letting you easily move small amounts of data between Linux and Windows. Mtools can create DOS/Windows floppy disks and to manage their contents.
If you want to make your text files Windows friendly, so that you can continue to work on them in Windows if you suffer any downtime, you can even invoke a line-ending flag that will convert the Linux LF endings to the CR/LF endings of the DOS world, and vice-versa.
Here's a sample session in which I format a DOS floppy, copy one set of text files with Linux line endings to floppy, then copy one text file with translated line endings to floppy. The session starts with inserting a floppy disk (blank or otherwise) that you can reformat.$ mformat a: $ cd /etc $ mcopy *conf* a: $ cd /home/gwilburn/docs $ mcopy -t myrecipes.txt a:
To reverse the process, copying a file from DOS floppy to Linux and translating the line endings, insert the floppy disk and follow a pattern such as this:$ cd /home/gwilburn/docs $ mcopy -t a:myoldrecipes.txt
Notice the dollar-sign prompts ($), indicating that I was able to do these diskette formatting and copying exercises while logged in on a non-root account. To allow any users to access floppies via the mtools kit, you must first assign them to group floppy and give the group read-write-execute permission on /dev/fd0.
More robust, regular backup is usually done via a tape drive or by installing a CD-RW device and burning backups to CD-R or CD-RW. If you can afford it, CD-RW makes an inexpensive and sturdy backup solution. Tape has proven its reliability over time and usually holds more data than CD-RW.
With a tape device you ordinarily use a program like tar, cpio, or dump to back up data. With CD-RW you can use the copy command (cp). In both cases it's best to automate backup so you don't forget to do it. In the event of total disk failure, the main filesystems you want to recover to a new HD are /etc, /home, and /usr/local (if you've been installing software from source). You will also want /var/named if you're running a local DNS server. The rest you can rebuild from scratch via your distribution CD and Internet updates.
Disasters are not common and Linux is one of the most stable operating systems you could place on a home system, but all systems eventually fail and knowing some emergency repair procedures can lessen the impact. Being proactive about disaster possibilities is the first step towards being a professional system administrator.
Gene Wilburn (firstname.lastname@example.org) is a Toronto-based IT specialist, musician and writer who operates a small farm of Linux servers.