mkdir: cannot create directory ‘test’: Read-only file system

We had to reboot a server we manage which had frozen/crashed during the night. The server rebooted but none of the services would come online. We tried filtering the logs to a new file to drill down the errors at the time of the crash when the system threw the following error:

We couldn’t do anything with the filesystem mounted so we rebooted the server to recovery and tried to fix the filesystem with the following command:

/dev/md2 because this was our OS raid mount point.

The scan returned the following:

We proceeded to reboot the server, this time the partition was mounted with write support.

This isn’t the end, technically. The way the server crashed and then booting back with the filesystem mounted as reboot could signal a failing drive. We will cover how to scan and check the health of your drives to see if they are failing and need to be replaced.