While goofing around on our proxy server last night, the system became quite rude.

[USER@SERVERNAME]- sudo rc-update add named default
sudo: /sbin/rc-update: Permission denied
[USER@SERVERNAME]-

excuse me?

After digging around for a while I decided to perform the usual quick fix operation, a reboot.

[USER@SERVERNAME]- sudo reboot
sudo: /sbin/reboot: Permission denied
[USER@SERVERNAME]-

um, not cool.

Lets see what the permissions are like in the /sbin directory:

[USER@SERVERNAME]- ls /sbin/
ls: /sbin/rc: Permission denied
ls: /sbin/halt: Permission denied
ls: /sbin/shutdown: Permission denied
ls: /sbin/rc-update: Permission denied
total 6137
-rw-r–r–  1 root root      0 Oct 27 16:29 .keep
-rwxr-xr-x  1 root root  44442 Oct 27 16:29 MAKEDEV
-rwxr-xr-x  1 root root  14960 Dec 24 14:12 agetty
-rwxr-xr-x  1 root root  44268 Oct 18 13:33 arp
… etc

Super weird. Eventually I did a dmesg and found the problem:

hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=62589081, sector=62589081
ide: failed opcode was: unknown
end_request: I/O error, dev hda, sector 62589081
ReiserFS: hda3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [236 361555 0x0 SD]
…etc

Managed to copy all the good files off the bad hard drive and get the server back up in about an hour. Thanks ryan! Only about 8 files had read errors when I copied to a new drive, including /sbin/init, /sbin/shutdown, /sbin/halt, /sbin/rc-update. Even if I had managed to reboot the machine, it never would have come back up because init was hosed.

Mike :: Mar.07.2005 :: General :: 2 Comments »