SMART error

[cyberjock]

Each additional layer of redundancy does result in a slight performance penalty. spares do not currently come on line automatically, so having "hot spares" is useful if you can't do hotswap/hotplug on a running system. Other than that, they are basically useless and a waste. If you're just doing word files and stuff(no VMs or anything with extremely high I/O) then RAIDZ3 will be unnoticeable in terms of the performance difference.

As for your disk replacement, you've failed to perform the last step of the replacement procedure. See the manual. ;)

 

[tstorzuk]

Yup, I had missed the last step of the procedure. It's all good now, and I scrubbed it just to make myself feel good about it. This morning my email from the server was all good news;

Code:

Checking status of zfs pools:
NAME              SIZE  ALLOC  FREE    CAP  DEDUP  HEALTH  ALTROOT
TRINITY_RAID-01    19T  18.4T  632G    96%  1.00x  ONLINE  /mnt
TRINITY_RAID-02    19T  17.9T  1.09T    94%  1.00x  ONLINE  /mnt
TRINITY_RAID-03    19T  3.97T  15.0T    20%  1.00x  ONLINE  /mnt
 
all pools are healthy

I typically use the NAS for watching video, ripping video to and encoding. I find it slow quite often when browsing for files, since there are so many. It takes quite a while to grab the list (from a Windows or Linux PC).

As for switching from RAIDZ2 to RAIDZ3 and using the hot spare. Does that have to be done by wiping the existing RAIDZ2 pool to created the new RAIDZ3? Or can the hot spare drive be incorporated and the pool changed to RAIDZ3 without moving all the files? I'm pretty certain the answer to this is that all the files have to be dumped.....but I have my fingers crossed.

If the files have to be dumped, I'll wait until I upgrade my next server. I'll use all 4tb WD Reds. I have the memory on order, and am looking for a reasonable price for the M1015 cards. Then it will be a matter of buying a few drives here and there, testing them out as I get them, then putting them into the old case with the new hardware.

At that point I'll re-create the pools on this server and use up the hot spares.

 

[Ericloewe]

Yup, I had missed the last step of the procedure. It's all good now, and I scrubbed it just to make myself feel good about it. This morning my email from the server was all good news;

Code:

Checking status of zfs pools:
NAME              SIZE  ALLOC  FREE    CAP  DEDUP  HEALTH  ALTROOT
TRINITY_RAID-01    19T  18.4T  632G    96%  1.00x  ONLINE  /mnt
TRINITY_RAID-02    19T  17.9T  1.09T    94%  1.00x  ONLINE  /mnt
TRINITY_RAID-03    19T  3.97T  15.0T    20%  1.00x  ONLINE  /mnt
 
all pools are healthy

I typically use the NAS for watching video, ripping video to and encoding. I find it slow quite often when browsing for files, since there are so many. It takes quite a while to grab the list (from a Windows or Linux PC).

As for switching from RAIDZ2 to RAIDZ3 and using the hot spare. Does that have to be done by wiping the existing RAIDZ2 pool to created the new RAIDZ3? Or can the hot spare drive be incorporated and the pool changed to RAIDZ3 without moving all the files? I'm pretty certain the answer to this is that all the files have to be dumped.....but I have my fingers crossed.

If the files have to be dumped, I'll wait until I upgrade my next server. I'll use all 4tb WD Reds. I have the memory on order, and am looking for a reasonable price for the M1015 cards. Then it will be a matter of buying a few drives here and there, testing them out as I get them, then putting them into the old case with the new hardware.

At that point I'll re-create the pools on this server and use up the hot spares.

You have to destroy it, no way around it.

 

[tstorzuk]

You have to destroy it, no way around it.

Yup, that's what I thought. I'll just upgrade my oldest NAS with the new set-up. Once it's built, I'll make the pools with RAIDZ3 instead of RAIDZ2, and keep none of the hot spares. It'll take about 6-12 months to buy all of the HDD's. I did just get a good price on the M1015 cards off eBay though, so those should take about 2-3 weeks to arrive.

Thanks for all the help. From everyone. It is always greatly appreciated.

 

[joeschmuck]

Because of what you use your NAS for, I would recommend just sticking with RAIDZ2 and remove the spare drive all together. I'm on travel right now but maybe someone could confirm that you could remove the spare and maintain the pool as RAIDZ2 without any issues, I'm pretty sure it can be done, I just did it on a VM without issue but it was a quick test. All you would need to do is use the GUI, go into Volume Status and for the spare drive select remove. But I'd like someone like Cyberjock to say it's safe, or someone else who has a great reputation.

 

[cyberjock]

It is safe. Spare disks are basically a waste of breath unless you can't do hotswap/hotplug and need high uptime.

 

[joeschmuck]

Cool, that is what I thought but other than a VM, I've never used a spare drive.

 

[tstorzuk]

Instead of using many M1015s in one server, you can also use an SAS expander (like the Intel RES2SV240) - it's cheaper once you start talking about three M1015s in many cases.

Ericloewe,

The biggest concern I have with using a SAS expander like the one you suggest is the overall amount of memory that my motherboards can handle. The maximum for the ones I have is 32Gb (and they are maxed out). The recommended amount of RAM is approximately 1GB per 1TB of storage. I thought that I could never get away with using a SAS expander due to that limitation. Am I wrong in that assumption?

I would not worry about the LLC (head parking) on the drives you have, the count isn't bad for the drive you have and how long it's been in use. Now if you do purchase a WD drive, checking the timer setting I feel is important because an 8 second timer would toss the LLC through the roof, but this is vendor and drive model specific.

joeschmuck & Ericloewe,

I have been replacing bad drives with WD Reds. Where do I set the timer settings and how do I change the timer settings? What would a good timer setting be for WD Red drives? I now have 5 WD Reds in this server, and it's going to be 6 after I replace another one tonight. It would be awesome to have the server running optimally, so your help would be greatly appreciated with this.

 

[Ericloewe]

The biggest concern I have with using a SAS expander like the one you suggest is the overall amount of memory that my motherboards can handle. The maximum for the ones I have is 32Gb (and they are maxed out). The recommended amount of RAM is approximately 1GB per 1TB of storage. I thought that I could never get away with using a SAS expander due to that limitation. Am I wrong in that assumption?

The rule is flexible, not a strict law. Also, having an expander is in, with regard to RAM requirements, no way worse or better than using more HBAs.

I have been replacing bad drives with WD Reds. Where do I set the timer settings and how do I change the timer settings? What would a good timer setting be for WD Red drives? I now have 5 WD Reds in this server, and it's going to be 6 after I replace another one tonight. It would be awesome to have the server running optimally, so your help would be greatly appreciated with this.

Reds are set to 300s or similar from the factory. No action is needed if the Load Cycle Count grows slowly or not at all.

 

[joeschmuck]

The RAM to TB of HD rule really is for the lower RAM capacities like 8GB, and it matters what your use of the FreeNAS system is. In a home system you can typically max out at 16GB RAM and be fine however again depending on what you are doing with your NAS, maybe 32GB or 64GB is what you need. Deduplication requires a ton of RAM and on a home system I would never recommend it, NEVER. For a small business then 64GB RAM or even more may be needed and I'd carefully plan those uses out.

With respect to the load cycle count, I agree with the statement above that the new REDs are coming with the timer set to 300 seconds but if you have an old drive it could be a different story. Check the SMART data to see what the head loading count is and if it appears high then post the SMART output so we can look at it to see what is causing the count to go up, maybe the drive is sleeping frequently, who knows.

 

[tstorzuk]

Yes, I just checked the SMART data and it looks like one of the drives needs to have the Wdidle3.exe program run on it (da18 looks to be suspect);

Code:

[root@trinity] ~# smartctl -a /dev/da8 | grep '4 St\|9 Po\|192 Po\|193 Lo'
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       21
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       1847
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       19
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       33
[root@trinity] ~# smartctl -a /dev/da15 | grep '4 St\|9 Po\|192 Po\|193 Lo'
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       2
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       11
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       3
[root@trinity] ~# smartctl -a /dev/da16 | grep '4 St\|9 Po\|192 Po\|193 Lo'
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       18
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       1714
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       16
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       34
[root@trinity] ~# smartctl -a /dev/da17 | grep '4 St\|9 Po\|192 Po\|193 Lo'
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       22
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       1966
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       20
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       52
[root@trinity] ~# smartctl -a /dev/da18 | grep '4 St\|9 Po\|192 Po\|193 Lo'
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       25
  9 Power_On_Hours          0x0032   097   097   000    Old_age   Always       -       2559
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       23
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2481
[root@trinity] ~# smartctl -a /dev/da23 | grep '4 St\|9 Po\|192 Po\|193 Lo'
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       3
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       46
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       1
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       7

 

[joeschmuck]

da18 might not be set to 300 seconds but it's not bad either. At your current rate of almost one per hour, you will never come close to the maximum rating which is currently 600,000 load cycles. 3 Years = 25,632 hours and if it survived double that you are still under 100,000 cycles, one 6th the rated value. Personally I wouldn't do anything to the drive as there is always some risk when re-flashing the firmware so why take it.

 

[Ericloewe]

da18 might not be set to 300 seconds but it's not bad either. At your current rate of almost one per hour, you will never come close to the maximum rating which is currently 600,000 load cycles. 3 Years = 25,632 hours and if it survived double that you are still under 100,000 cycles, one 6th the rated value. Personally I wouldn't do anything to the drive as there is always some risk when re-flashing the firmware so why take it.

I'll echo this statement. LCC is only an issue in drives with very low settings, like the Greens' (and one defective batch of Reds from December 2013 or so) 8s default.