A VM with SLES 11.4 (64bit) was installed a while ago, and is used as application server.
Everything has been working properly for a while (~1 month), but as of a few days back, some filesystems (/ and /opt) have turned to read-only (!?).
Pretty much every task fails, be it starting vi or even man pages (can’t create files on disk).
Background and solutions
- try to backup the affected FS to another server
- try a remount to rw of the affected FS (be careful with rootFS) :
mount -o remount,rw <FS>
- reboot and either go to
single user mode
and fsck
the disk or sometimes the automatic fsck
on reboot would catch it
- automatic fsck on reboot : touch /forcefsck and then reboot, with SLES the checkfs.sh script does not exist, here it is : /etc/init.d/boot.rootfsck
When on VMware ESXi systems with back-ended to SAN storage
modify the disk timeouts for the block devices in /sys
# cat /sys/block/sd*/device/timeout
60
Increased the value of this number from 60
to 180,
this modifies the time before a “timeout” occurs, essentially allowing you to suffer more latency.
Source: http://unix.stackexchange.com/questions/16156/suse-filesystem-turns-read-only
http://unix.stackexchange.com/questions/145882/read-only-root-filesystem
http://unix.stackexchange.com/questions/123963/how-to-force-fsck-at-every-boot-all-relevant-filesystems