SLES File System(s) turns Read-Only

A VM with SLES 11.4 (64bit) was installed a while ago,  and is used as application server.

Everything has been working properly for a while (~1 month), but as of a few days back, some filesystems (/  and /opt)  have turned to read-only (!?).

Pretty much every task fails, be it starting vi or even man pages (can’t create files on disk).

Background and solutions

  • try to backup the affected FS to another server
  • try a remount to rw of the affected FS (be careful with rootFS) :   mount -o remount,rw <FS>
  • reboot and either go to single user mode and fsck the disk or sometimes the automatic fsck on reboot would catch it
  • automatic fsck on reboot : touch /forcefsck and then reboot,  with SLES the script does not exist,  here it is : /etc/init.d/boot.rootfsck

When on VMware ESXi systems with back-ended to SAN storage

modify the disk timeouts for the block devices in /sys

# cat /sys/block/sd*/device/timeout


Increased the value of this number from 60 to 180, this modifies the time before a “timeout” occurs, essentially allowing you to suffer more latency.