While I wait for the new hardware, I collected smart data from the drives, and found a few surprising things..
The following smart parameters are insanely high
- "Raw_Read_Error_Rate"    on da0, da2, da4 & da6
 
- "Seek_Error_Rate"    on da0, da2, da4 & da6
 
- "Hardware_ECC_Recovered"    on da0, da2
 
- "Command_Timeout"    on da2...
 
All da0, da2, da4 & da6 are Seagate Barracuda's.... Yep pieces of sh**.  All other drives are HGST NAS drives.
Can I expect a massive catastrophic pool failure soon?  Should I just go ahead and replace these now even of scrubs and smart are no issuing errors per-se?  How serious are the parameters above?
Code:
[root@freenas] ~# smartctl -A /dev/da0
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   118   099   006    Pre-fail  Always       -       179052893
  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       220
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   082   060   030    Pre-fail  Always       -       186977691
  9 Power_On_Hours          0x0032   060   060   000    Old_age   Always       -       35225
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       218
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   099   099   000    Old_age   Always       -       1
190 Airflow_Temperature_Cel 0x0022   065   058   045    Old_age   Always       -       35 (Min/Max 23/36)
194 Temperature_Celsius     0x0022   035   042   000    Old_age   Always       -       35 (0 18 0 0 0)
195 Hardware_ECC_Recovered  0x001a   036   027   000    Old_age   Always       -       179052893
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       88983132473802
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       2760294427
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       1246150209
 
Code:
[root@freenas] ~# smartctl -A /dev/da1
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   135   135   054    Pre-fail  Offline      -       98
  3 Spin_Up_Time            0x0007   134   134   024    Pre-fail  Always       -       412 (Average 406)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       104
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   146   146   020    Pre-fail  Offline      -       29
  9 Power_On_Hours          0x0012   097   097   000    Old_age   Always       -       21685
10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       103
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       308
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       308
194 Temperature_Celsius     0x0002   181   181   000    Old_age   Always       -       33 (Min/Max 18/39)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0
 
Code:
[root@freenas] ~# smartctl -A /dev/da2
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   114   099   006    Pre-fail  Always       -       76493248
  3 Spin_Up_Time            0x0003   078   070   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       99
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   084   060   030    Pre-fail  Always       -       246489059
  9 Power_On_Hours          0x0032   076   076   000    Old_age   Always       -       21774
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       98
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       4295032833
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   069   062   045    Old_age   Always       -       31 (Min/Max 20/32)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       93
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       99
194 Temperature_Celsius     0x0022   031   040   000    Old_age   Always       -       31 (0 17 0 0 0)
195 Hardware_ECC_Recovered  0x001a   018   004   000    Old_age   Always       -       76493248
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       230957571396874
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       1132304118
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       1724526318
 
Code:
[root@freenas] ~# smartctl -A /dev/da3
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000b  100  100  016  Pre-fail  Always  -  0
  2 Throughput_Performance  0x0005  134  134  054  Pre-fail  Offline  -  100
  3 Spin_Up_Time  0x0007  134  134  024  Pre-fail  Always  -  410 (Average 407)
  4 Start_Stop_Count  0x0012  100  100  000  Old_age  Always  -  257
  5 Reallocated_Sector_Ct  0x0033  100  100  005  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000b  100  100  067  Pre-fail  Always  -  0
  8 Seek_Time_Performance  0x0005  148  148  020  Pre-fail  Offline  -  28
  9 Power_On_Hours  0x0012  095  095  000  Old_age  Always  -  38035
10 Spin_Retry_Count  0x0013  100  100  060  Pre-fail  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  256
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  692
193 Load_Cycle_Count  0x0012  100  100  000  Old_age  Always  -  692
194 Temperature_Celsius  0x0002  193  193  000  Old_age  Always  -  31 (Min/Max 17/41)
196 Reallocated_Event_Count 0x0032  100  100  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0022  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0008  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x000a  200  200  000  Old_age  Always  -  0
 
Code:
[root@freenas] ~# smartctl -A /dev/da4
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000f  117  099  006  Pre-fail  Always  -  145994152
  3 Spin_Up_Time  0x0003  096  095  000  Pre-fail  Always  -  0
  4 Start_Stop_Count  0x0032  100  100  020  Old_age  Always  -  79
  5 Reallocated_Sector_Ct  0x0033  100  100  010  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000f  081  060  030  Pre-fail  Always  -  8846044156
  9 Power_On_Hours  0x0032  076  076  000  Old_age  Always  -  21291
10 Spin_Retry_Count  0x0013  100  100  097  Pre-fail  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  020  Old_age  Always  -  79
183 Runtime_Bad_Block  0x0032  100  100  000  Old_age  Always  -  0
184 End-to-End_Error  0x0032  100  100  099  Old_age  Always  -  0
187 Reported_Uncorrect  0x0032  100  100  000  Old_age  Always  -  0
188 Command_Timeout  0x0032  100  100  000  Old_age  Always  -  0 0 0
189 High_Fly_Writes  0x003a  052  052  000  Old_age  Always  -  48
190 Airflow_Temperature_Cel 0x0022  066  059  045  Old_age  Always  -  34 (Min/Max 20/35)
191 G-Sense_Error_Rate  0x0032  100  100  000  Old_age  Always  -  0
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  47
193 Load_Cycle_Count  0x0032  001  001  000  Old_age  Always  -  396836
194 Temperature_Celsius  0x0022  034  041  000  Old_age  Always  -  34 (0 17 0 0 0)
197 Current_Pending_Sector  0x0012  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0010  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x003e  200  200  000  Old_age  Always  -  0
240 Head_Flying_Hours  0x0000  100  253  000  Old_age  Offline  -  16830h+02m+24.358s
241 Total_LBAs_Written  0x0000  100  253  000  Old_age  Offline  -  67664160577
242 Total_LBAs_Read  0x0000  100  253  000  Old_age  Offline  -  126347092720
 
Code:
[root@freenas] ~# smartctl -A /dev/da5
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000b  100  100  016  Pre-fail  Always  -  0
  2 Throughput_Performance  0x0005  133  133  054  Pre-fail  Offline  -  104
  3 Spin_Up_Time  0x0007  139  139  024  Pre-fail  Always  -  395 (Average 392)
  4 Start_Stop_Count  0x0012  100  100  000  Old_age  Always  -  122
  5 Reallocated_Sector_Ct  0x0033  100  100  005  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000b  100  100  067  Pre-fail  Always  -  0
  8 Seek_Time_Performance  0x0005  150  150  020  Pre-fail  Offline  -  27
  9 Power_On_Hours  0x0012  097  097  000  Old_age  Always  -  21903
10 Spin_Retry_Count  0x0013  100  100  060  Pre-fail  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  118
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  326
193 Load_Cycle_Count  0x0012  100  100  000  Old_age  Always  -  326
194 Temperature_Celsius  0x0002  187  187  000  Old_age  Always  -  32 (Min/Max 18/38)
196 Reallocated_Event_Count 0x0032  100  100  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0022  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0008  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x000a  200  200  000  Old_age  Always  -  0
 
Code:
[root@freenas] ~# smartctl -A /dev/da6
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000f  118  099  006  Pre-fail  Always  -  185216640
  3 Spin_Up_Time  0x0003  094  094  000  Pre-fail  Always  -  0
  4 Start_Stop_Count  0x0032  100  100  020  Old_age  Always  -  24
  5 Reallocated_Sector_Ct  0x0033  100  100  010  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000f  052  051  030  Pre-fail  Always  -  4621557391380
  9 Power_On_Hours  0x0032  086  086  000  Old_age  Always  -  12976
10 Spin_Retry_Count  0x0013  100  100  097  Pre-fail  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  020  Old_age  Always  -  22
183 Runtime_Bad_Block  0x0032  099  099  000  Old_age  Always  -  1
184 End-to-End_Error  0x0032  100  100  099  Old_age  Always  -  0
187 Reported_Uncorrect  0x0032  100  100  000  Old_age  Always  -  0
188 Command_Timeout  0x0032  100  100  000  Old_age  Always  -  0 0 0
189 High_Fly_Writes  0x003a  100  100  000  Old_age  Always  -  0
190 Airflow_Temperature_Cel 0x0022  067  061  045  Old_age  Always  -  33 (Min/Max 19/33)
191 G-Sense_Error_Rate  0x0032  100  100  000  Old_age  Always  -  0
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  14
193 Load_Cycle_Count  0x0032  001  001  000  Old_age  Always  -  244719
194 Temperature_Celsius  0x0022  033  040  000  Old_age  Always  -  33 (0 17 0 0 0)
197 Current_Pending_Sector  0x0012  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0010  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x003e  200  200  000  Old_age  Always  -  0
240 Head_Flying_Hours  0x0000  100  253  000  Old_age  Offline  -  10745h+31m+24.877s
241 Total_LBAs_Written  0x0000  100  253  000  Old_age  Offline  -  41427173244
242 Total_LBAs_Read  0x0000  100  253  000  Old_age  Offline  -  66169052358
 
Code:
[root@freenas] ~# smartctl -A /dev/da7
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000b  100  100  016  Pre-fail  Always  -  0
  2 Throughput_Performance  0x0005  135  135  054  Pre-fail  Offline  -  84
  3 Spin_Up_Time  0x0007  100  100  024  Pre-fail  Always  -  0
  4 Start_Stop_Count  0x0012  100  100  000  Old_age  Always  -  4
  5 Reallocated_Sector_Ct  0x0033  100  100  005  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000b  100  100  067  Pre-fail  Always  -  0
  8 Seek_Time_Performance  0x0005  124  124  020  Pre-fail  Offline  -  33
  9 Power_On_Hours  0x0012  100  100  000  Old_age  Always  -  6026
10 Spin_Retry_Count  0x0013  100  100  060  Pre-fail  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  4
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  244
193 Load_Cycle_Count  0x0012  100  100  000  Old_age  Always  -  244
194 Temperature_Celsius  0x0002  162  162  000  Old_age  Always  -  37 (Min/Max 23/39)
196 Reallocated_Event_Count 0x0032  100  100  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0022  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0008  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x000a  200  200  000  Old_age  Always  -  0