Clients re-mount filesystems as Read-only after disk failure

[niklasmato]

Hi,

We have a freenas setup with 6 disks.
On this 6 disks we have a freenas zfs filesystem with on-top a iscsi shared storage for our vm environment which is xenserver.

After that 1 of the new disks failed (after less than 1 week) we received a mail from the freenas system that the system was degraded.
At the same time our virtual machines running which use the freenas iscsi storage their filesystems became read-only.

We replaced the defunct disk and rebuild the zfs filesytem but we also needed to reboot all our vm's and run fsck on their mounted filesystems.

Is this normal behaviour?
I must say that this behavior scares me, we are using this setup as a development environment and we cannot have these kinds of errors.

Thanks !

 

[ixdwhite]

When you say "became read-only", do you mean the Linux OS in the VM triggered the remount-ro recovery behavior, or did the actual target become unwriteable (i.e., return a WRITE PROTECT sense or similar SCSI error to writes)?

The distinction is significant for debugging purposes.

Can you check /var/log/messages* on the system to get the exact errors that triggered the behavior you are reporting?

 

[niklasmato]

The Linux OS in the VM triggered the remount-ro recovery behaviour due to too many errors.

Below are the errors on the freenas :


Jun 19 19:00:00 freenas newsyslog[53041]: logfile turned over due to size>100K
Jun 19 19:03:47 freenas kernel: da3 at mps0 bus 0 scbus0 target 13 lun 0
Jun 19 19:03:47 freenas kernel: da3: <ATA WDC WD20EFRX-68E 0A80> s/n WD-WCC
4M1038939 detached
Jun 20 00:55:39 freenas smartd[3339]: Device: /dev/da3 [SAT], previous self-test
completed with error (read test element)
Jun 20 00:55:39 freenas smartd[3339]: Device: /dev/da3 [SAT], Self-Test Log erro
r count increased from 0 to 1
Jun 20 12:55:39 freenas smartd[3339]: Device: /dev/da3 [SAT], previous self-test
completed with error (read test element)
Jun 20 12:55:39 freenas smartd[3339]: Device: /dev/da3 [SAT], Self-Test Log erro
r count increased from 1 to 2
Jun 23 13:38:22 freenas kernel: da7 at mps0 bus 0 scbus0 target 17 lun 0
Jun 23 13:38:22 freenas kernel: da7: <ATA WDC WD20EFRX-68E 0A80> Fixed Direct Ac
cess SCSI-6 device
Jun 23 13:38:22 freenas kernel: da7: Serial Number WD-WCC4M2993398
Jun 23 13:38:22 freenas kernel: da7: 300.000MB/s transfers
Jun 23 13:38:22 freenas kernel: da7: Command Queueing enabled
Jun 23 13:38:22 freenas kernel: da7: 1907729MB (3907029168 512 byte sectors: 255
H 63S/T 243201C)
Jun 23 13:38:22 freenas kernel: da7: quirks=0x8<4K>
Jun 23 13:48:03 freenas notifier: 1+0 records in
Jun 23 13:48:03 freenas notifier: 1+0 records out
Jun 23 13:48:03 freenas notifier: 1048576 bytes transferred in 0.601571 secs (17
43063 bytes/sec)
Jun 23 13:48:04 freenas notifier: dd: /dev/da7: short write on character device
Jun 23 13:48:04 freenas notifier: dd: /dev/da7: end of device
Jun 23 13:48:04 freenas notifier: 5+0 records in
Jun 23 13:48:04 freenas notifier: 4+1 records out
Jun 23 13:48:04 freenas notifier: 4284416 bytes transferred in 0.060911 secs (70
338747 bytes/sec)
Jun 23 13:48:10 freenas kernel: GEOM_ELI: Device da7p1.eli created.
Jun 23 13:48:10 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128

Jun 23 13:48:10 freenas kernel: GEOM_ELI: Crypto: software

 

[dlavigne]

Hmm, how did you paste those errors? They are invisible unless you highlight them with a mouse.

 

[niklasmato]

Sorry just see it. Just copied the output of the terminal = white on black bg.
Corrected.

Jun 19 19:00:00 freenas newsyslog[53041]: logfile turned over due to size>100K
Jun 19 19:03:47 freenas kernel: da3 at mps0 bus 0 scbus0 target 13 lun 0
Jun 19 19:03:47 freenas kernel: da3: <ATA WDC WD20EFRX-68E 0A80> s/n WD-WCC
4M1038939 detached
Jun 20 00:55:39 freenas smartd[3339]: Device: /dev/da3 [SAT], previous self-test
completed with error (read test element)
Jun 20 00:55:39 freenas smartd[3339]: Device: /dev/da3 [SAT], Self-Test Log erro
r count increased from 0 to 1
Jun 20 12:55:39 freenas smartd[3339]: Device: /dev/da3 [SAT], previous self-test
completed with error (read test element)
Jun 20 12:55:39 freenas smartd[3339]: Device: /dev/da3 [SAT], Self-Test Log erro
r count increased from 1 to 2
Jun 23 13:38:22 freenas kernel: da7 at mps0 bus 0 scbus0 target 17 lun 0
Jun 23 13:38:22 freenas kernel: da7: <ATA WDC WD20EFRX-68E 0A80> Fixed Direct Ac
cess SCSI-6 device
Jun 23 13:38:22 freenas kernel: da7: Serial Number WD-WCC4M2993398
Jun 23 13:38:22 freenas kernel: da7: 300.000MB/s transfers
Jun 23 13:38:22 freenas kernel: da7: Command Queueing enabled
Jun 23 13:38:22 freenas kernel: da7: 1907729MB (3907029168 512 byte sectors: 255
H 63S/T 243201C)
Jun 23 13:38:22 freenas kernel: da7: quirks=0x8<4K>
Jun 23 13:48:03 freenas notifier: 1+0 records in
Jun 23 13:48:03 freenas notifier: 1+0 records out
Jun 23 13:48:03 freenas notifier: 1048576 bytes transferred in 0.601571 secs (17
43063 bytes/sec)
Jun 23 13:48:04 freenas notifier: dd: /dev/da7: short write on character device
Jun 23 13:48:04 freenas notifier: dd: /dev/da7: end of device
Jun 23 13:48:04 freenas notifier: 5+0 records in
Jun 23 13:48:04 freenas notifier: 4+1 records out
Jun 23 13:48:04 freenas notifier: 4284416 bytes transferred in 0.060911 secs (70
338747 bytes/sec)
Jun 23 13:48:10 freenas kernel: GEOM_ELI: Device da7p1.eli created.
Jun 23 13:48:10 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128

Jun 23 13:48:10 freenas kernel: GEOM_ELI: Crypto: software

​