Freenas Rebooted Following Drive Failure

daveymg · Jul 17, 2016

FreeNAS-9.3-STABLE-201501162230
AMD Athlon(tm) II X2 260 Processor
7908MB non-ecc
Asus M5A78L-M LX mainboard

Freenas alert emails were generated early morning saying that a drive had recorded a SMART error followed by an alert to say the volume had been degraded. When I went to check the Freenas webpage the next day the freenas box was completely offline, no ping response. When I checked the physical box it had rebooted but was trying to boot off of a hard drive instead of the USB (I'm pretty sure I had the boot config setup for USB but this system had been up for over a year now so maybe not). After resetting the boot to USB, Freenas started Ok.

My question is what has made Freenas reboot after the drive failure? I've pasted logs below

The errors started at 23:52 with the SMART errors logged at 00:42
Many errors logged until at 01:19 the drive is logged as being detached but drive errors continue to be logged until at 01:19:34 a swap_pager: I/O error appears followed by vm_fault.

These two last errors appear to be repeated for many megabytes of message logfile until the system reboots itself.

The drive failure and degradation appear to have completed gracefully, but what caused the swap_pager errors?

The pool is 3 x 2Tb vdev's, each vdev is a 2Tb mirrored pair.

Code:

[root@freenas] ~# zpool status
  pool: bigvol
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 22h5m with 0 errors on Wed Jun 29 23:06:03 2016
config:

        NAME                                            STATE     READ WRITE CKSUM
        bigvol                                          DEGRADED     0     0     0
          mirror-0                                      ONLINE       0     0     0
            ada0p2                                      ONLINE       0     0     0
            ada1p2                                      ONLINE       0     0     0
          mirror-1                                      DEGRADED     0     0     0
            ada2p2                                      ONLINE       0     0     0
            5190664789547636286                         UNAVAIL      0     0     0  was /dev/ada3p2
          mirror-2                                      ONLINE       0     0     0
            gptid/4292fe69-c9fe-11e0-a5db-f46d04cdc0ac  ONLINE       0     0     0
            gptid/43b1637b-c9fe-11e0-a5db-f46d04cdc0ac  ONLINE       0     0     0

errors: No known data errors

Code:

Jul 12 23:52:40 freenas ahcich3: Timeout on slot 1 port 0
Jul 12 23:52:40 freenas ahcich3: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd c0 serr 00000800 cmd 0000e117
Jul 12 23:52:40 freenas (ada3:ahcich3:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 0040 00 00 00 00 00 00
Jul 12 23:52:40 freenas (ada3:ahcich3:0:0:0): CAM status: Command timeout
Jul 12 23:52:40 freenas (ada3:ahcich3:0:0:0): Retrying command
Jul 13 00:00:00 freenas syslog-ng[76777]: Configuration reload request received, reloading configuration;
Jul 13 00:01:41 freenas ahcich3: Timeout on slot 21 port 0
Jul 13 00:01:41 freenas ahcich3: is 00000000 cs 00200000 ss 00000000 rs 00200000 tfd c0 serr 00000000 cmd 0000f517
Jul 13 00:01:41 freenas (ada3:ahcich3:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 0040 00 00 00 00 00 00
Jul 13 00:01:41 freenas (ada3:ahcich3:0:0:0): CAM status: Command timeout
Jul 13 00:01:41 freenas (ada3:ahcich3:0:0:0): Retrying command
Jul 13 00:18:17 freenas ahcich3: Timeout on slot 3 port 0
Jul 13 00:18:17 freenas ahcich3: is 00000000 cs 00000008 ss 00000000 rs 00000008 tfd c0 serr 00000000 cmd 0000e317
Jul 13 00:18:17 freenas (ada3:ahcich3:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 0040 00 00 00 00 00 00
Jul 13 00:18:17 freenas (ada3:ahcich3:0:0:0): CAM status: Command timeout
Jul 13 00:18:17 freenas (ada3:ahcich3:0:0:0): Retrying command
Jul 13 00:19:16 freenas ahcich3: Timeout on slot 30 port 0
Jul 13 00:19:16 freenas ahcich3: is 00000000 cs 00000000 ss c00001ff rs c00001ff tfd 40 serr 00000000 cmd 0000e817
Jul 13 00:19:16 freenas (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 0b ca d1 58 40 d6 00 00 00 00 00
Jul 13 00:19:16 freenas (ada3:ahcich3:0:0:0): CAM status: Command timeout
Jul 13 00:19:16 freenas (ada3:ahcich3:0:0:0): Retrying command
Jul 13 00:29:54 freenas ahcich3: Timeout on slot 21 port 0
Jul 13 00:29:54 freenas ahcich3: is 00000000 cs 00200000 ss 00000000 rs 00200000 tfd c0 serr 00000000 cmd 0000f517
Jul 13 00:29:54 freenas (ada3:ahcich3:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 0040 00 00 00 00 00 00
Jul 13 00:29:54 freenas (ada3:ahcich3:0:0:0): CAM status: Command timeout
Jul 13 00:29:54 freenas (ada3:ahcich3:0:0:0): Retrying command
Jul 13 00:30:28 freenas ahcich3: Timeout on slot 0 port 0
Jul 13 00:30:28 freenas ahcich3: is 00000000 cs 00000000 ss 000007ff rs 000007ff tfd 40 serr 00000000 cmd 0000ea17
Jul 13 00:30:28 freenas (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 0b 7c dc a6 40 e6 00 00 00 00 00
Jul 13 00:30:28 freenas (ada3:ahcich3:0:0:0): CAM status: Command timeout
Jul 13 00:31:49 freenas ahcich3: Timeout on slot 3 port 0
Jul 13 00:31:49 freenas ahcich3: is 00000000 cs 00000000 ss 000000f8 rs 000000f8 tfd 40 serr 00000000 cmd 0000e717
Jul 13 00:31:49 freenas (ada3:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 c7ca ed 40 50 00 00 00 00 00
Jul 13 00:31:49 freenas (ada3:ahcich3:0:0:0): CAM status: Command timeout
Jul 13 00:31:49 freenas (ada3:ahcich3:0:0:0): Retrying command
Jul 13 00:38:47 freenas ahcich3: Timeout on slot 3 port 0
Jul 13 00:38:47 freenas ahcich3: is 00000000 cs 00000000 ss 00001ff8 rs 00001ff8 tfd 40 serr 00000000 cmd 0000ec17
Jul 13 00:38:47 freenas (ada3:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 cc0a 2b 40 51 00 00 01 00 00
Jul 13 00:38:47 freenas (ada3:ahcich3:0:0:0): CAM status: Command timeout
Jul 13 00:38:47 freenas (ada3:ahcich3:0:0:0): Retrying command
Jul 13 00:42:25 freenas smartd[89600]: Device: /dev/ada3, FAILED SMART self-check. BACK UP DATA NOW!
Jul 13 00:42:50 freenas smartd[89600]: Device: /dev/ada3, 1 Currently unreadable (pending) sectors
Jul 13 00:42:52 freenas smartd[89600]: Device: /dev/ada3, 1 Offline uncorrectable sectors
Jul 13 00:42:53 freenas smartd[89600]: Device: /dev/ada3, Failed SMART usage Attribute: 5 Reallocated_Sector_Ct.
Jul 13 00:48:45 freenas (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 8d fe 91 40 00 00 00 01 00 00
Jul 13 00:48:45 freenas (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Jul 13 00:48:45 freenas (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error:40 (UNC )
Jul 13 00:48:45 freenas (ada3:ahcich3:0:0:0): RES: 41 40 dd fe 91 00 00 00 00 00 01
Jul 13 00:48:45 freenas (ada3:ahcich3:0:0:0): Retrying command
Jul 13 00:48:54 freenas (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 8d fe 91 40 00 00 00 01 00 00
Jul 13 00:48:54 freenas (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Jul 13 00:48:54 freenas (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error:40 (UNC )
Jul 13 00:48:54 freenas (ada3:ahcich3:0:0:0): RES: 41 40 e0 fe 91 00 00 00 00 00 01
Jul 13 00:48:54 freenas (ada3:ahcich3:0:0:0): Retrying command
Jul 13 00:49:01 freenas (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 8d fe 91 40 00 00 00 01 00 00
Jul 13 00:49:01 freenas (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Jul 13 00:49:01 freenas (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error:40 (UNC )
Jul 13 00:49:01 freenas (ada3:ahcich3:0:0:0): RES: 41 40 e0 fe 91 00 00 00 00 00 01
Jul 13 00:49:01 freenas (ada3:ahcich3:0:0:0): Retrying command
Jul 13 00:49:09 freenas (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 8d fe 91 40 00 00 00 01 00 00
Jul 13 00:49:09 freenas (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Jul 13 00:49:09 freenas (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error:40 (UNC )
Jul 13 00:49:09 freenas (ada3:ahcich3:0:0:0): RES: 41 40 e0 fe 91 00 00 00 00 00 01
Jul 13 00:49:09 freenas (ada3:ahcich3:0:0:0): Retrying command
Jul 13 00:49:18 freenas (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 8d fe 91 40 00 00 00 01 00 00
Jul 13 00:49:18 freenas (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Jul 13 00:49:18 freenas (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error:40 (UNC )
Jul 13 00:49:18 freenas (ada3:ahcich3:0:0:0): RES: 41 40 e0 fe 91 00 00 00 00 00 01
Jul 13 00:49:18 freenas (ada3:ahcich3:0:0:0): Error 5, Retries exhausted
Jul 13 00:49:28 freenas (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 80 8d ff 91 40 00 00 00 00 00 00
Jul 13 00:49:28 freenas (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
...
Jul 13 01:19:34 freenas (aprobe0:ahcich3:0:0:0): Error 5, Retries exhausted
Jul 13 01:19:34 freenas ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
Jul 13 01:19:34 freenas ada3: <ST32000644NS SN12> s/n 9WM66121 detached
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 00 00 00 00 00 00
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): ATA status: 71 (DRDY DF SERV ERR), error: 04 (ABRT )
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): RES: 71 04 91 00 32 40 00 00 00 04 00
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): Error 5, Periph was invalidated
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): WRITE_DMA48. ACB: 35 00 c6 84 07 40 ac 00 00 00 00 01
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): CAM status: Unconditionally Re-queue Request
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): Error 5, Periph was invalidated
Jul 13 01:19:34 freenas ahcich3: Error while READ LOG EXT
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 c683 07 40 ac 00 00 01 00 00
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): ATA status: 00 ()
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): ATA status: 00 ()
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): RES: 00 00 00 00 00 00 00 00 00 00 00
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): Error 5, Periph was invalidated
Jul 13 01:19:34 freenas ahcich3: Error while READ LOG EXT
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 c686 07 40 ac 00 00 01 00 00
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): ATA status: 00 ()
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): RES: 00 00 00 00 00 00 00 00 00 00 00
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): Error 5, Periph was invalidated
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 c687 07 40 ac 00 00 01 00 00
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): ATA status: 00 ()
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): RES: 00 00 00 00 00 00 00 00 00 00 00
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): Error 5, Periph was invalidated
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 c688 07 40 ac 00 00 01 00 00
Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): CAM status: ATA Status Error

Jul 13 01:19:34 freenas (ada3:ahcich3:0:0:0): Error 5, Periph was invalidated
Jul 13 01:19:34 freenas (0:4:0/0): WRITE(16). CDB: 8a 08 00 00 00 00 00 5f 18 f8 00 00 01 58 00 00
Jul 13 01:19:34 freenas (0:4:0/0): Tag: 0x7205423a, type 1
Jul 13 01:19:34 freenas (0:4:0/0): ctl_process_done: 148 seconds
Jul 13 01:19:34 freenas swap_pager: I/O error - pagein failed; blkno 1584607,size 8192, error 6
Jul 13 01:19:34 freenas vm_fault: pager read error, pid 1338 (devd)
Jul 13 01:19:35 freenas swap_pager: I/O error - pagein failed; blkno 1573901,size 4096, error 6
Jul 13 01:19:35 freenas vm_fault: pager read error, pid 1338 (devd)
Jul 13 01:19:35 freenas kernel: Failed to write core file for process devd (error 14)
Jul 13 01:19:35 freenas kernel: Failed to write core file for process devd (error 14)
Jul 13 01:19:35 freenas kernel: pid 1338 (devd), uid 0: exited on signal 11
Jul 13 01:19:35 freenas swap_pager: I/O error - pagein failed; blkno 1582485,size 4096, error 6
Jul 13 01:19:35 freenas vm_fault: pager read error, pid 1 (init)
Jul 13 01:19:36 freenas swap_pager: I/O error - pagein failed; blkno 1574041,size 8192, error 6
Jul 13 01:19:36 freenas vm_fault: pager read error, pid 1 (init)
Jul 13 01:19:36 freenas swap_pager: I/O error - pagein failed; blkno 1574041,size 8192, error 6
...
Jul 13 01:21:56 freenas vm_fault: pager read error, pid 1 (init)
Jul 13 01:21:56 freenas swap_pager: I/O error - pagein failed; blkno 1574041,size 8192, error 6

Stux · Jul 17, 2016

You had swap in use on a drive that failed. The swap went away, the system crashed.

I refer to this problem in this thread (9.10 but still applies), my solution would've helped, i.e. to periodically swap off/on the swaps ;)

daveymg · Jul 17, 2016

Stux said:
You had swap in use on a drive that failed. The swap went away, the system crashed.

I refer to this problem in this thread (9.10 but still applies), my solution would've helped, i.e. to periodically swap off/on the swaps ;)

So is Freenas using the raw disks for swap space even though the vdev's are mirrored? (Sorry if I'm mixing terminology here, just trying to understand why mirroring didn't mitigate in this case)

Stux · Jul 17, 2016

The swap is striped with no redundancy.

If there is swap on use on a disk that goes away the virtual memory pager will crash.

It's a design decision which could be considered a design flaw, but was obviously necessary to allow booting off a USB and having swap.

The real solution would be to have your swap on a separate SSD (perhaps a boot SSD)

daveymg · Jul 17, 2016

Stux said:
The swap is striped with no redundancy.

If there is swap on use on a disk that goes away the virtual memory pager will crash.

So the failure of any disk can cause a crash if there is swap in use. There's only about 200Mb or so of swap ever in use on this system, I guess if I add more ram that will eliminate the need for swap?

Stux · Jul 17, 2016

daveymg said:
So the failure of any disk can cause a crash if there is swap in use. There's only about 200Mb or so of swap ever in use on this system, I guess if I add more ram that will eliminate the need for swap?

I wouldn't say that. See the thread I linked.

200MB swap usage with 8GB, but still swap in use at 32GB.

jgreco · Jul 17, 2016

daveymg said:
So the failure of any disk can cause a crash if there is swap in use. There's only about 200Mb or so of swap ever in use on this system, I guess if I add more ram that will eliminate the need for swap?

The problem is that even modest memory pressure can cause "inactive" pages to be swapped, but those pages may be the ones responsible for key bits of the system that cope with unusual events. There's an argument to be made that critical path items should be locked in core with mlock(2) but in practice that's probably a difficult solution to implement.

SweetAndLow · Jul 17, 2016

You should also get smart data for /dev/ada3p2 and probably replace the drive. Why does it have a strange drive label that includes a partition?

jgreco · Jul 17, 2016

SweetAndLow said:
You should also get smart data for /dev/ada3p2 and probably replace the drive. Why does it have a strange drive label that includes a partition?

That's an interesting question. Did you build the pool by hand? That's potentially very dangerous.

daveymg · Jul 18, 2016

First up, thanks for the replies and support, appreciated.

I built this server back at Freenas 8.3 and don't recall how I created the pool. Apart from restarts for upgrades it's run without problem up until this event. If building the pool by hand means using the CLI, then no, I would have built it via the GUI.

I have already replaced the drive, it's resilvering now.

As far as the root cause of the crash it concerned, I must say that it's taken the gloss off of my perception of having a storage system with redundancy built in. I built the pool using "best practice" approach with mirrored vdevs but nowhere in my reading of the docs did I notice mention of the multi-point of failure created by the way swap is implemented. I understand that the two are separate issues ie. data redundancy presented by ZFS and the reliability of the underlying operating system.

There are a number of bug reports and other comments related to swap implementation. Some say that loss of swap should just result in apps being closed and shouldn't cause a system crash(clearly not true in my case). Best practice is said to be to not disable swap as, although it should never be needed (and one post stated that if it was needed then the system has other problems), if it is needed you'd better have it.

Rather than have a system critical piece of storage striped across a non-redundant disk volume, it would make more sense (to me) to just put it on one disk, at least then there's a 1 in 6 chance of a system crash, in my case, rather than a 6 in 6 chance. My system has 10G of swap space but typically only around 200Mb in use. I expect there are good reasons for it being implemented as it is.

I could add an ssd disk and use it for swap but there are no spare sata slots left on the mainboard. I'll maybe look around for a sata controller to use.

SweetAndLow · Jul 18, 2016

Usually systems use 0 swap space and if it does something is wrong with your system. Usually under memory constraints the arc will release memory hopefully faster than your system consumers it. In 9.10 people have been mentioning swap being used and I suspect there is a bug here.

Stux · Jul 19, 2016

I first came across the swap related crash because I like to test the redundancy... This means pulling out disks... the beauty of a hotswap bay. This led to swap related crashes pretty quick.

Did you know that the resilvers are very quick when you put the disk back... when the swap doesn't crash. It seems it only has to resilver the transactions where the disk was not present... or some such.

Works with the boot mirror too ;)

daveymg · Jul 20, 2016

So, I decided to try the script offered by Stux in the linked thread above so as to try to keep swap space use at 0 but I've ended up in an odd situation. Looking at swapctl -l this morning (before running the script), all drives apart from ada3 (the newly replaced drive) had swap in use. Looking at fstab showed that ada3p1.eli was no longer listed, the other drives were listed.

I ran Stux's script but at the swapon stage received No such file or directory errors for all drives except ada3 (as it wasn't listed in fstab).

Looking in /dev there is only a .eli entry for ada3p1.eli none of the other drives are listed. So it looks as though the swapoff command has removed the entries in /dev. Are these swap areas created at system boot or are they permanent?

I'm not sure where to go from here, reboot or try to recreate the swap manually?

Code:

[root@freenas ~]# swapctl -l                                                    
Device:       512-blocks      Used:                                             
/dev/ada3p1.eli    4194304          0                                           
[root@freenas ~]# cat /etc/fstab                                                
freenas-boot/grub       /boot/grub      zfs     rw,noatime      1       0       
/dev/ada0p1.eli none                    swap            sw              0      0
/dev/ada1p1.eli none                    swap            sw              0      0
/dev/ada2p1.eli none                    swap            sw              0      0
/dev/ada4p1.eli none                    swap            sw              0      0
/dev/ada5p1.eli none                    swap            sw              0      0
[root@freenas ~]# swapon /dev/ada0p1.eli                                        
swapon: /dev/ada0p1.eli: No such file or directory                              
[root@freenas ~]#   

[root@freenas /dev]# ls | grep ada                                              
ada0                                                                            
ada0p1                                                                          
ada0p2                                                                          
ada1                                                                            
ada1p1                                                                          
ada1p2                                                                          
ada2                                                                            
ada2p1                                                                          
ada2p2                                                                          
ada3                                                                            
ada3p1                                                                          
ada3p1.eli                                                                      
ada3p2                                                                          
ada4                                                                            
ada4p1                                                                          
ada4p2                                                                          
ada5                                                                            
ada5p1                                                                          
ada5p2                                                                          
[root@freenas /dev]#

daveymg · Jul 20, 2016

Edit to the above, I just found the following in the message log

Code:

Jul 21 10:04:37 freenas GEOM_ELI: Device ada0p1.eli destroyed.
Jul 21 10:04:37 freenas GEOM_ELI: Detached ada0p1.eli on last close.
Jul 21 10:04:37 freenas GEOM_ELI: Device ada1p1.eli destroyed.
Jul 21 10:04:37 freenas GEOM_ELI: Detached ada1p1.eli on last close.
Jul 21 10:04:37 freenas GEOM_ELI: Device ada2p1.eli destroyed.
Jul 21 10:04:37 freenas GEOM_ELI: Detached ada2p1.eli on last close.
Jul 21 10:04:37 freenas GEOM_ELI: Device ada4p1.eli destroyed.
Jul 21 10:04:37 freenas GEOM_ELI: Detached ada4p1.eli on last close.
Jul 21 10:04:37 freenas GEOM_ELI: Device ada5p1.eli destroyed.
Jul 21 10:04:37 freenas GEOM_ELI: Detached ada5p1.eli on last close.

So the swapoff has removed the devices. How should I reinstate them?

Stux · Jul 20, 2016

Try

swapon -a

Fwiw, I believe the fstab is rebuilt at reboot.

daveymg · Jul 20, 2016

I did try that already, same result. I can see that the partitions still exist on the drives via gpart. There must be another step required to actually recreate the swap device entries in /dev before they can be used.

daveymg · Jul 20, 2016

I just rebooted the box and all of the swap locations are now listed correctly.

Stux, do you run your script on your Freenas box? It looks like an additional step is needed after swapoff to recreate the swap entries in /dev before swapon can be used.

Stux · Jul 21, 2016

I just re-ran the script on my box (9.10). Worked fine.

It's been running a full data scan/check with another system and swap was usage was up to 400MB

Its not actually designed (or tested) when you replace a drive.

Think I might modify it to use swapinfo instead of fstab as I think that will be the current live result.

Important Announcement for The TrueNAS Community.

Freenas Rebooted Following Drive Failure

daveymg

Dabbler

Stux

MVP

daveymg

Dabbler

Stux

MVP

daveymg

Dabbler

Stux

MVP

jgreco

Resident Grinch

SweetAndLow

Sweet'NASty

jgreco

Resident Grinch

daveymg

Dabbler

SweetAndLow

Sweet'NASty

Stux

MVP

daveymg

Dabbler

daveymg

Dabbler

Stux

MVP

daveymg

Dabbler

daveymg

Dabbler

Stux

MVP

Similar threads