Hardware question - Am I cooking my HDDs?

[jeffsimply]

Hi guys,

I must admit I'm what you'd consider a noob. I built my FreeNAS system without reading any guides thinking it would be designed to be plug and play.

I realise the error of my ways and some of my reading has actually been scaring me!!

Anyway, I have on my list to buy some new items (supermicro mobo, ideally that will fit my G3220 since the CPU seems fit for purpose and, more crucially, ECC RAM).

My HDDs are 2x HGST 3TB units and 2x WD random 2TB units that I had in my PC before hand. I have the intention of swapping the WDs for HGST 3TBs if/when they die. Both pairs are in RAID1 mirrors.

I read some comments about keeping HDD temperatures below 40C, put my hand infront of my computer which is a nonbranded ITX case with 4 removable drive bays at the front. There is no fan directing air over the HDDs.

I put a thermometer probe in between 2 drives and was slightly alarmed to find the air temperature was around 43C.

Was the 40C figure plucked from the air by someone? I have checked the HGST spec sheets and the operating temperature is quoted at 5C to 60C which gives me some headroom.

I like the compactness of my case, but admit it's hard to source an ITX board that supports ECC RAM (mine is a B85 chipset and whilst the CPU supports ECC RAM I'm lead to believe the Motherboard will not- although I've not tried it).

 

[hugovsky]

You should do "smartctl -a /dev/ada0(change accordingly)" and read temps there. Do that to all your disks.

 

[DataKeeper]

Agreed with hugovsky above in running the smartctl command on each of your drives from the command line. This will tell you what your drive temps are. If the air temp around the drives is 43* then the drives themselves are likely higher.

Is there a specific reason you want the HGST drives over say WD RED drives? The reds run quieter, use less power and generate less heat then the HGST drives. Food for thought there for future drive purchases.

 

[jeffsimply]

Went with HGST as I read a report where that particular drive had a failure rate of 3% compared to about 10% of any other brand, that and they're 7200rpm it seemed like a good idea. I've always liked Hitachi HDDs, and for the £10 extra on Amazon figured it's a no brainer.

I will run the smartctl command later (thank you for the tip) and have a look at temps- if they're indeed higher than the air temp then I'll look at getting a bigger case with some airflow.

I must say, I'd love a rack mount chassis, but where do you mount them? I presume in a server environment you have frames to install multiple racks, but if you only have 1 because it's just for home use and you want to be cool and have a rack server then where do you put it ?

 

[Pheran]

I must say, I'd love a rack mount chassis, but where do you mount them? I presume in a server environment you have frames to install multiple racks, but if you only have 1 because it's just for home use and you want to be cool and have a rack server then where do you put it ?

It doesn't make much sense to buy a rackmount chassis if you don't have a rack to mount it in. :)

 

[danb35]

If you want to put lots of drives in one chassis, often rack mount is your only option. I'm not aware of many (if any) towers that have even 12 drive bays, much less the 40+ you can get in a 4U chassis.

 

[Pheran]

If you want to put lots of drives in one chassis, often rack mount is your only option. I'm not aware of many (if any) towers that have even 12 drive bays, much less the 40+ you can get in a 4U chassis.

This is true, but the OP of this thread only has 4 drives.

 

[danb35]

Good point. In that case, I'm not sure why OP would want a rack chassis other than for aesthetics. Although they do seem to have pretty good cooling, at least based on my two...

 

[Nick2253]

Hi guys,

I must admit I'm what you'd consider a noob. I built my FreeNAS system without reading any guides thinking it would be designed to be plug and play.

I realise the error of my ways and some of my reading has actually been scaring me!!

Welcome! I'm glad you've realized the error of your ways. FreeNAS is an incredibly powerful and easy to use system, as long as you've got the right hardware underneath it.

Anyway, I have on my list to buy some new items (supermicro mobo, ideally that will fit my G3220 since the CPU seems fit for purpose and, more crucially, ECC RAM).

My HDDs are 2x HGST 3TB units and 2x WD random 2TB units that I had in my PC before hand. I have the intention of swapping the WDs for HGST 3TBs if/when they die. Both pairs are in RAID1 mirrors.

You're on the right track. However, on the off-chance that you really mean RAID1 when you say RAID1, I'm obligated to point out that you should not use controller-based RAID with FreeNAS. You should be exposing the raw disks to FreeNAS, and using mirroring. And if you're looking to combine those drives, I would strongly recommend that you look towards a system that supports RAIDZ2, because RAIDZ1 (or RAID5) is practically dead given our large drive size (http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/).

I read some comments about keeping HDD temperatures below 40C, put my hand infront of my computer which is a nonbranded ITX case with 4 removable drive bays at the front. There is no fan directing air over the HDDs.

I put a thermometer probe in between 2 drives and was slightly alarmed to find the air temperature was around 43C.

Was the 40C figure plucked from the air by someone? I have checked the HGST spec sheets and the operating temperature is quoted at 5C to 60C which gives me some headroom.

This number was pulled from Google's comprehensive study of their data-center drives: https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf. You can find the appropriate graph on pg. 6. Based on the Google data, failure rates were minimized around 40C. The failure rate of drives was minimized across the temperature range of ~35C-40C. Importantly, drives in that temperature range had the lowest AFP during the first year, and a fairly low AFP during the 2, 3, and 4th years.

The number from the spec sheet indicates safe operating temperature, but that doesn't speak to its longevity.

I like the compactness of my case, but admit it's hard to source an ITX board that supports ECC RAM (mine is a B85 chipset and whilst the CPU supports ECC RAM I'm lead to believe the Motherboard will not- although I've not tried it).

It is not hard to source an ITX board that supports ECC RAM. Your complaint is that it is hard to source a cheap ITX board that supports ECC RAM. Supermicro, ASRock, and Asus all have a number of boards. Supermicro is the tried and true board, so it's recommended. An ASRock board is what I use (I got it on a good sale), and I'm happy with it, FWIW. However, ASRock/Asus are generally not "officially" recommended, because they are generally not the quality of the Supermicro boards, and are usually not significantly less expensive.

Went with HGST as I read a report where that particular drive had a failure rate of 3% compared to about 10% of any other brand, that and they're 7200rpm it seemed like a good idea. I've always liked Hitachi HDDs, and for the £10 extra on Amazon figured it's a no brainer.

I will run the smartctl command later (thank you for the tip) and have a look at temps- if they're indeed higher than the air temp then I'll look at getting a bigger case with some airflow.

The "report" you read was most likely the Backblaze summary of HDD failures, which is a terrible misuse of statistics. Their "conclusion" is based on a faulty premise, and is biased based on their drive selection criteria. Please take any of their conclusions with a huge grain of salt.

7200RPM is generally a bad idea in a home NAS. The performance benefits you get over 5400RPM is negligible for a typical home user, but you get all the downsides of increased power consumption and higher temperatures. If you are looking to run VMs off your NAS, or access it with a number of clients simultaneously, 7200RPM becomes more useful. But at that point, since you're only using four drives in two pairs, you'll gain more benefits buy adding drives rather than using 7200RPM drives.

I must say, I'd love a rack mount chassis, but where do you mount them? I presume in a server environment you have frames to install multiple racks, but if you only have 1 because it's just for home use and you want to be cool and have a rack server then where do you put it ?

You mount a rack mount chassis in a rack. If you don't have a rack, then it doesn't make sense to get a rack mount chassis. Rack mount chassis are designed to fit a specific form factor, with the biggest compromise to fit that factor being noise. Furthermore, racks are generally installed so that servers have reliable access to cold air in front, so cooling is generally not as good in a home environment.

If you insist on getting the rack mount chassis, then you can just set it on a table or shelf. But be cognizant of the cooling issue: don't compromise your server because you don't provide it cool enough air, or because you aren't happy with the noise.

EDIT: Minor clarifications.

 

[danb35]

However, on the off-chance that you really mean RAID1 when you say RAID1, I'm obligated to point out that you should not use RAID with FreeNAS. You should be using RAIDZ1.

I think it much more likely that the OP has two, two-disk mirrors (what the rest of the world calls RAID1), striped together. RAID is used with FreeNAS all the time, as I'm sure you know perfectly well--what I expect you meant to say is that "you should not use hardware RAID with FreeNAS." If the OP's configuration is RAID1 on a hardware RAID controller (or, heaven forbid, RAID1 on a motherboard fake-RAID controller), then there's a definite problem. If he's using ZFS mirrors, no problem. Your recommendation of RAIDZ1 is inappropriate, primarily because it's impossible to create a RAIDZ1 vdev with only two disks. It would be an option for his entire volume (i.e., all four disks), but not recommended for the reasons you point out.

My experience with rack-mount chassis (which is based on a sample size of two) is that they do a much better job of cooling airflow than other enclosures, even in less-than-optimal environments. That typically makes them noisy. But my two servers are in an enclosed closet with no airflow, and I don't see drive temps exceed 40 deg C at any point. Just for curiosity's sake, I've put a probe thermometer in there to see what the ambient temps look like in that closet--I'll report back in a bit.

 

[Nick2253]

I think it much more likely that the OP has two, two-disk mirrors (what the rest of the world calls RAID1), striped together. RAID is used with FreeNAS all the time, as I'm sure you know perfectly well--what I expect you meant to say is that "you should not use hardware RAID with FreeNAS." If the OP's configuration is RAID1 on a hardware RAID controller (or, heaven forbid, RAID1 on a motherboard fake-RAID controller), then there's a definite problem. If he's using ZFS mirrors, no problem. Your recommendation of RAIDZ1 is inappropriate, primarily because it's impossible to create a RAIDZ1 vdev with only two disks. It would be an option for his entire volume (i.e., all four disks), but not recommended for the reasons you point out.

TIL. I always thought the distinction between mirroring and RAIDZ1 was a distinction without a difference. I've always used 4+ disks, so it never came up. I've corrected this, and clarified my post wrt RAID.

 

[danb35]

RAIDZ1 is not mirroring. RAIDZ1 is the approximate equivalent of RAID5, not of RAID1. There are technical differences, but from an end user's perspective, they can generally be considered the same. Similarly, ZFS mirrors are not RAID1, but they are very similar in function from an end user's perspective.

 

[jeffsimply]

Right... Thank you for all the helpful comments!

Thanks for the info on the HGST drives. If it's just that I didn't need to spend as much then I won't worry about that- if they're causing excessive heat that's another thing though...

Yes to clarify my volume setup, I did NOT use hardware RAID. I added the drives in pairs using the FreeNAS GUI and added them as striped volumes. I think this would technically be known as RAID10 ? I.e. a pair of striped mirrors? Or am I just being even more of a noob and making things up now? :)
(I am under the impression, if I want to migrate to RAIDZ2 then I will need to backup all the data from my zpool, then destroy the pool and create a new one- moving the data back on after creating the new volume in RAIDZ2 ?)

Does anyone know of a good resource that would explain what I'm looking at on SmartCTL ? And more specifically, what to look for to prepare for failures. Looking at the temps (I booted FreeNAS up about 10 minutes ago) they're between 30-35 if I'm looking at the right thing. However the air infront of the case is not nearly as warm as it was yesterday morning before posting here.

On the WD drives (ada2 and 3) they quote (Min/Max 20/60) after the temp value. I am truly hoping this isn't a recorded value of the min/max while running as I can see how that would be bad!

In all seriousness I think I will reshell this machine into a tower with some decent fans. I got suckered into this case on the promise of removable drives, but how often am I really going to be removing them....

 

[Nick2253]

In all seriousness I think I will reshell this machine into a tower with some decent fans. I got suckered into this case on the promise of removable drives, but how often am I really going to be removing them....

I'm in the same boat. I bought a DS380 because of the "value" of removable drives. The only time I've ever removed the drives is because I'm tinkering with the case, trying to prevent my drives from overheating. The case has terrible airflow over the drives, and they pretty much go over 40C every day in the summer, unless I leave the side of the case open, and point a box fan at it.

I'm strongly tempted to just get a different case if I can find one I like, for a good price. Probably going to get the Fractal Design Node 304.

The thing about removable drives is that they are only useful when you have a failure. And even then, they are only required when you have a failure, and can't afford any downtime. For most home users, you can shut down your FreeNAS server for a few hours while you tinker with the machine, removing and adding hard drives. Obviously, for business users, that may not be acceptable; hence, removable drives.

 

[Robert Trevellyan]

if I want to migrate to RAIDZ2 then I will need to backup all the data from my zpool, then destroy the pool and create a new one- moving the data back on after creating the new volume in RAIDZ2

Correct.

Does anyone know of a good resource that would explain what I'm looking at on SmartCTL ?

Wikipedia is usually a good place to start.

On the WD drives (ada2 and 3) they quote (Min/Max 20/60) after the temp value. I am truly hoping this isn't a recorded value of the min/max while running as I can see how that would be bad!

If you're reading this in the output of the smartctl command, then they are indeed the extremes that your drives have experienced in the past.

 

[DataKeeper]

Are you shutting down the server or letting it run 24/7? FreeNAS should be plugged into a UPS, turned on and left running. This is part of the reason for the RED drives instead of the HGSTs since they run slower, cooler and use less power. Very few reasons to run HGST drives in a NAS as Nick pointed out above.

Man pages! :D If you haven't read one before prepare to run screaming away after a few seconds! LOL they aren't that bad ;)
https://www.freebsd.org/cgi/man.cgi?query=smartctl&manpath=FreeBSD+9.0-RELEASE+and+Ports&format=html

How are you gaining shell access? If via the web interface stop.. there are much better ways! Download the 'putty' SSH client and use that to connect. Its free and likely the most used client out there.


A few ways to run the smartctl command to help you out. Ignore the initial > mark I include!
You can use -a for all SMART info or -x for ALL drive info.

The following will scroll by quickly.
> smartctl -a /dev/ada1

This will allow you to view it easier.. Use the up/down arrows to move line by line and hit "q" to quit.
> smartctl -a /dev/ada1 | less

This will display all SMART info lines with the word temperature.
> smartctl -a /dev/ada1 | grep -i temperature

This will display ALL drive info lines with the word temperature. Much more info then just -a provides and gives the history of your drives temps. This is a good one to use to start learning.
> smartctl -x /dev/ada1 | grep -i temperature

This will display ALL drive info more selectively by matching "current temperature"
> smartctl -x /dev/ada1 | grep -i "current temperature"

An easy way to save the output to a file is like so.
> smartctl -x /dev/ada1 > drive_ada1.txt

To view the file use the 'less' command and control with up/down arrow keys and "q" to quit.
> less drive_ada1.txt

For example here is the temperature info for /dev/da1 on my system.
Using -a at first pulls the temperature data and at the end of the output shows the current drive temperature is 32.
Using -x reads a bit differently but still shows the temp as 32. Its lowest temp ever was 19 and highest was 33. Since the last boot the lowest temp was 24. Drive has never been below or over the min/max limits. I bolded a few select lines to look at.

[root@FileServ] ~# smartctl -a /dev/da1 | grep -i temperature
194 Temperature_Celsius 0x0022 120 119 000 Old_age Always - 32

[root@FileServ] ~# smartctl -x /dev/da1 | grep -i temperature
194 Temperature_Celsius -O---K 120 119 000 - 32
Current Temperature: 32 Celsius
Power Cycle Min/Max Temperature: 24/33 Celsius
Lifetime Min/Max Temperature: 19/33 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (62)
Index Estimated Time Temperature Celsius

There is a lot more then just temperature data and some of it, if monitored, can show where a drive may be beginning to fail jumping you time to make corrections or get a replacement before a failure.

The best thing is to take the time and go through the output line by line. If you don't understand a line a quick Google should explain it quick enough and if not just ask. Make sure you look at each drive (da1, da2, etc) and see if anything stands out.

Good luck and hope this helps...

 

[jeffsimply]

Great stuff, thanks for the help.

I have been leaving the server online, but after reading the temps I decided to go into damage control and shut it down for the time being. I don't need it online all the time and it's only me and my GF that live at our place, it stores backups for my PC and all of our shared media and what with it being summer we are not watching a great deal of films etc so can live without it being online all the time.

I've spent all my spare money this month on PC upgrades so I'll order a decent tower case/motherboard/HDD coolers next month.

Thanks for the tip regarding Putty- I will give it a go!

 

[jeffsimply]

Thanks everyone.

I did a bit of PC tinkering over the weekend, installed 6x Noctua fans in my gaming PC and now I can actually feel a breeze coming from the thing!! :D

Also reshelled my FreeNAS into an old tower case I had laying around. I will most likely do a bit more tinkering soon (for that ECC memory and a premium PSU)- but I've spent more than my permitted limit on PC bits this month (probably year too) already.

HDDs were sat at 27/29/30/31 after it had been online for an hour. I'm trying to think what a good test would be for temps, checking them whilst doing a ZFS scrub perhaps? The HGST drives don't appear to log min/max temps, and the WD ones are still showing 57/60... Is there a way to reset this as I haven't found a "min/max since boot" on my SMART readings?

A very helpful post

Thanks for taking the time to write this! I found it helpful in clearing up any doubts and I'm sure it will help others if they find this thread. Also lol at the Man Pages. I saw that exact page from a Google before I posted this topic and yes, was a bit like "Oh gawd, I'll post instead!!".

 

[DataKeeper]

No, you can't reset the SMART data so the max numbers are what they are unless they go higher in the future. If they are idling now at around 27/29/30/31 then its unlikely they would get too high under a heavy load and since this is a home system sustained heavy loads are not likely I'm guess.

Check out @Bidule0hm and his scripts in his signature. He has some temperature scripts to help monitor temps as well as some other very useful scripts. Well worth installing. Will email you drive stats nightly as well.

Google UNIX, Linux, BSD cheat sheet for some more info. One of the best books I found years ago was Unix for Dummies and the Linux one as well. I think there might also be a BSD one but not sure. Each UNIX 'OS' varies slightly from the others but its easy enough to work around the differences.

 

[jeffsimply]

Thanks again @DataKeeper . That was actually going to be my next question, where to start with creating a script to include some drive related stuff in my daily emails.