How screwed does it look like I am

[Th3RadMan]

I was having errors with a drive so I decided to replace it with a shucked 8tb WD white (unlabed red) drive, but I'm getting lots of problems with the resilver process.

Code:

pool: MediaVolume																												
 state: DEGRADED																													
status: One or more devices is currently being resilvered.  The pool will														  
	   continue to function, possibly in a degraded state.																		
action: Wait for the resilver to complete.																						
  scan: resilver in progress since Wed Jan 10 07:17:21 2018																		
	   2.30T scanned at 73.5M/s, 464G issued at 14.5M/s, 2.32T total															  
	   77.3G resilvered, 19.57% done, 1 days 13:21:56 to go																		
config:																															
																																  
	   NAME											STATE	 READ WRITE CKSUM												
	   MediaVolume									 DEGRADED	 0	 0	 0												
		 raidz2-0									  DEGRADED	 0	 0	 0												
		   gptid/9f478c76-a4ea-11e7-83d5-00e05b880d05  FAULTED	  8	68	 0  too many errors								
		   gptid/a004519a-a4ea-11e7-83d5-00e05b880d05  ONLINE	   0	 0	 0  (resilvering)								  
		   gptid/ca19d5ca-f4f9-11e7-88de-00e05b880d05  ONLINE	   0	 0	 0  (resilvering)								  
		   gptid/a18f35ed-a4ea-11e7-83d5-00e05b880d05  ONLINE	   0	 0	 0												
		   gptid/a24ca2d3-a4ea-11e7-83d5-00e05b880d05  ONLINE	   0	 0	 0												
		   gptid/a3104a14-a4ea-11e7-83d5-00e05b880d05  DEGRADED	 0	 0   331  too many errors								
																																  
errors: No known data errors											

I had cksum errors with the faulted drive, which I was going to replace, then I came back today and saw another degraded drive and a second resilvering drive, which really threw me off, since I'm using a Z2 array. All the data is still there, but for some reason when I try to copy it from freenas, to my windows system on an external drive, but only reads at a few MB/s, then drops down to 0. Any ideas how screwed I am?

 

[Ericloewe]

Resilvering on 11.1. Living dangerously, huh?

It's basically impossible to understand what's going on without SMART data for those disks. Could be two independent failures, could be a software issue, could be anything really.

The good news it that gptid/a3104a14-a4ea is still semi-usable and is not completely gone. Though those 331 checksum errors are disturbing - could be a really bad cable or something.

 

[Th3RadMan]

huh, guess I'll just let it go until it says it finishes. I've been using Cable Matters miniSAS to 4xsata cables, and many of the errors I've had seem to have the cables as a common factor. Should I get them replaced with something different?

I want to just get all the 3TB drives I have in there replaced with 8TB ones, but I can't backup all the data so I can just delete the volume and start fresh. Any idea why my server will write just fine, but can't read to copy for crap?

 

[Th3RadMan]

I hooked up an external hdd and made it a volume, but I don't know how to just add my mediavolume volume to the external drive.

Code:

NAME																USED  AVAIL  REFER  MOUNTPOINT								 
Backup															  316K  7.02T	88K  /mnt/Backup								 
MediaVolume														1.54T  8.94T   176K  /mnt/MediaVolume							
MediaVolume/.system												89.5M  8.94T   192K  legacy									 
MediaVolume/.system/configs-a7c4a4a3d45a4720a1e6b8ad799731fd	   13.1M  8.94T  13.1M  legacy									 
MediaVolume/.system/configs-cdca2cbb2f1b460b99bc2866bf58ed26	   1007K  8.94T  1007K  legacy									 
MediaVolume/.system/cores										  2.01M  8.94T  2.01M  legacy									 
MediaVolume/.system/rrd-a7c4a4a3d45a4720a1e6b8ad799731fd		   35.6M  8.94T  35.6M  legacy									 
MediaVolume/.system/rrd-cdca2cbb2f1b460b99bc2866bf58ed26			176K  8.94T   176K  legacy									 
MediaVolume/.system/samba4										  591K  8.94T   591K  legacy									 
MediaVolume/.system/syslog-a7c4a4a3d45a4720a1e6b8ad799731fd		36.7M  8.94T  36.7M  legacy									 
MediaVolume/.system/syslog-cdca2cbb2f1b460b99bc2866bf58ed26		 176K  8.94T   176K  legacy									 
MediaVolume/Media												  1.54T  8.94T  1.54T  /mnt/MediaVolume/Media					 
PlexJail														   3.43G   442G	88K  /mnt/PlexJail							   
PlexJail/Jail													  3.41G   442G   136K  /mnt/PlexJail/Jail						 
PlexJail/Jail/.warden-template-pluginjail-11.0-x64				  546M   442G   539M  /mnt/PlexJail/Jail/.warden-template-pluginja
il-11.0-x64																														 
PlexJail/Jail/.warden-template-pluginjail-11.0-x64-20180108151614   539M   442G   539M  /mnt/PlexJail/Jail/.warden-template-pluginja
il-11.0-x64-20180108151614																										 
PlexJail/Jail/plexmediaserver_2									2.35G   442G  2.87G  /mnt/PlexJail/Jail/plexmediaserver_2		
freenas-boot													   2.97G  25.9G	64K  none										
freenas-boot/ROOT												  2.95G  25.9G	29K  none										
freenas-boot/ROOT/11.0-U3										   354K  25.9G   726M  /										   
freenas-boot/ROOT/11.0-U4										   190K  25.9G   728M  /										   
freenas-boot/ROOT/11.1-RELEASE									 2.95G  25.9G   830M  /										   
freenas-boot/ROOT/Initial-Install								  1.87M  25.9G   736M  legacy									 
freenas-boot/ROOT/default										   139K  25.9G   736M  legacy									 
freenas-boot/grub												  6.85M  25.9G  6.85M  legacy

Hopefully if I do this it will back up correctly, right? If so What do I have to use in shell to back it up?

 

[rs225]

What is the power supply, and are the drives using splitters?

 

[Th3RadMan]

I'm using a Seasonic M12II 520 psu and 2 minisas to sata breakout cables connect to a LSI 9211-8i in IT mode

 

[rs225]

What is the firmware on the LSI?

 

[Th3RadMan]

I believe it's version 20

 

[danb35]

I believe it's version 20

Let's confirm that. What's the output of sas2flash -listall?

 

[Th3RadMan]

yep, FW version 20... I think I might have lost everything though, while I was backing up to an external drive, 3 total drives faulted, I think because of the resilvering that I couldn't stop, now the volume is unavailable. I'm going to replace the sas cables with some supermicro ones today, but I don't even know if that will save anything at this point