Pheran's 32TB FreeNAS build with photos

Vivan

Dabbler
Joined
Aug 11, 2015
Messages
24
Odd. Either the Reds are significantly louder than my 4TB HGST drives, or we've done something else differently (did you open the top vents?), or we just have a different perception of the loudness. I actually have a dB meter, but the server is so quiet I doubt I could even get it to register.

Everything is closed for me :(. I was running SMART tests when I noticed the noise - perhaps they were putting the drives under extra loud load?
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
The WD Reds are the quietest drives I've ever used. However, seeking is the noisier operation a drive can do and high load things like SMART tests and scrubs lead to a lot of seeks so you've probably heard the worst case scenario ;)
 

Vivan

Dabbler
Joined
Aug 11, 2015
Messages
24
The WD Reds are the quietest drives I've ever used. However, seeking is the noisier operation a drive can do and high load things like SMART tests and scrubs lead to a lot of seeks so you've probably heard the worst case scenario ;)
Weird, I can hear it from like a metre away even in a noisy room - not doing any tests right now :(.
 

solarisguy

Guru
Joined
Apr 4, 2014
Messages
1,125
Does the case has its own fan and/or power supply? Is noise coming from them?

Are the disks properly mounted? Did you use any rubber or plastic washers or grommets to decrease vibrations?

Is the case solid and stable by itself?
 

Vivan

Dabbler
Joined
Aug 11, 2015
Messages
24
Does the case has its own fan and/or power supply? Is noise coming from them?

Are the disks properly mounted? Did you use any rubber or plastic washers or grommets to decrease vibrations?

Is the case solid and stable by itself?

The case is solid and stable. The disks are properly mounted as far as I'm aware - I mounted them using the standard rubber mickey mouse things that came with the case.

I can actually hear it from several metres away in a quiet room... thought it would be much quieter than that :(.
 
Last edited:

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
I just successfully upgraded to FreeNAS 9.3.1 and the SuperMicro/LSI 20.00.04 firmware, no problems. My process was to first prepare a USB stick with the firmware and verify that I could access it from within the UEFI shell. Then I installed the FreeNAS update and when the system rebooted I inserted the USB stick and hit F11 to access the UEFI shell again. I then installed the LSI firmware update. Once it was completed I power cycled the system just for good measure. By doing this, the FreeNAS 9.3.1 code was never running without the v20 firmware, and vice-versa.
 

bmcclure937

Contributor
Joined
Jul 13, 2012
Messages
110
Thanks for this great guide. I am building a new FreeNAS and found value in reviewing your process to learn about memtest86+, SMART, and other tests I should run prior to configuring services, transferring data, and starting to use the server in production mode.

Do you have a quick list of things you recommend (like a short summary)?
 

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
Thanks for this great guide. I am building a new FreeNAS and found value in reviewing your process to learn about memtest86+, SMART, and other tests I should run prior to configuring services, transferring data, and starting to use the server in production mode.

Do you have a quick list of things you recommend (like a short summary)?

That's kind of a broad question. The only thing I can recommend it to spend time reading the links in the first post and this forum in general because you will learn a lot. Going into a FreeNAS build (or any storage solution) without understanding what you are configuring is a good way to lose data.
 

bmcclure937

Contributor
Joined
Jul 13, 2012
Messages
110
That's kind of a broad question. The only thing I can recommend it to spend time reading the links in the first post and this forum in general because you will learn a lot. Going into a FreeNAS build (or any storage solution) without understanding what you are configuring is a good way to lose data.

I have been running a FreeNAS 0.7.x (and prior) for over 6 years... just never got around to upgrading and my existing NAS server was fitting my needs and chugging along without any hiccups. I use normal SMART and scrubs on my RAIDZ1 (4x1TB) but I have not needed to configure or tinker with any of that for a long time.

On top of that, I have not completed a new build for quite a long time too. I wanted to be sure my memory and drives did not have issues by running some checks and burn-in like you have done in your build. I will follow some of those same concepts.

1. Which memtest86+ check did you run?
2. Did you run any other burn-in for your drives? This is the area I am most curious because I do not know exactly what tests would be good to run for checking the drives and also performing some basic burn-in.
 

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
I have been running a FreeNAS 0.7.x (and prior) for over 6 years... just never got around to upgrading and my existing NAS server was fitting my needs and chugging along without any hiccups. I use normal SMART and scrubs on my RAIDZ1 (4x1TB) but I have not needed to configure or tinker with any of that for a long time.

On top of that, I have not completed a new build for quite a long time too. I wanted to be sure my memory and drives did not have issues by running some checks and burn-in like you have done in your build. I will follow some of those same concepts.

1. Which memtest86+ check did you run?
2. Did you run any other burn-in for your drives? This is the area I am most curious because I do not know exactly what tests would be good to run for checking the drives and also performing some basic burn-in.

Memtest86+ is pure default, in fact when you boot it it immediately starts running tests, no questions asked. Just be sure to let it go long enough to get a good number of passes - if you look at my screenshot, you can see it made it to pass 11 before I shut it off.

For burn-in, I used SMART tests and badblocks as detailed in the 2nd/3rd posts. Check out the Hard Drive Burn-In testing thread for more info.
 

Marc Leonard

Dabbler
Joined
Nov 24, 2015
Messages
10
Do you have a windows 10 box there? Could you try the Chelsio S310E-CR NIC in that box? I would love to know if its possible to get it up and running in W10.
I'll contribute to this :)
I took Pherans 10Gb experiment here and did it for myself. I did this config for both Win7 and Win10. Check it:

Windows 7
I put the Chelsio S310E card and put it in my Win7 machine. Sure enough, I had that weird sleep issue as well. Not as big of a deal for me, I need to keep my computer NOT sleeping all the time :)
Also, FWIW, I took two FTLX8511D3 transceivers and connect fiber between my FreeNAS box and Win7 machine. ONE NOTE: I had to manually/physically crossover the fiber cable to make it work. Maybe that Chelsio card is too old to do it automatically?
And regarding Pheran's speed issue. I saw something similar. The transfer started off at about 500MB/sec, then quick dove down to about 200-220 MB/sec - sustained...
... But, it seems this may be attributed to SMB2... Read on.

Windows 10
I decided to give Windows 10 a try with the same setup. In addition to the Chelsio card, I have some more older hardware. So I created a dual boot just in case.
The Chelsio drivers installed without issue! I was able to connect to my FreeNAS box using the same fiber setup. After changing the max connection type in FreeNAS, I used the Get-SmbConnection
command in powershell to see how Windows was connecting to FreeNAS. It automatically switched to SMB3! Sweet!
I tried some transfers, and I was getting SUSTAINED 500 MB/sec! Awesome!
I tried to sleep and the computer... Unfortunately, I got a BSOD with the error being DPC_Watchdog_Violation
After reading a bunch, it sounds like this error can come from a slew of things. I'm going to do a bit more testing, but I think we might be SOL with this sleeping bug. But whatever, it's super super old hardware.

As always, user beware. I know Cyberjock has mentioned there are some problems with SMB3. So only further testing will prove if I can move to Windows 10 permanently with this setup.

UPDATE:
I've got some other strange BSOD errors from time to time. I don't yet know if this is related to the card or not. One interesting one is as follows - when I have the computer off for an extended period of time, and turn it back on, I'll get a DRIVER_POWER_FAILURE_STATE error. It will automatically restart, and it boots up just fine.

Next UPDATE:
I've changed my Win10 10Gb cards to a Mellanox ConnectX-2 card. I can connect to my FreeNAS Chelsio card perfectly. The sustained transfer speeds seem to be a bit slower though. Now, I am getting a different error upon sleep, but it actually might be my raid card... Not the Connectx-2. Does anyone have any experience with ConnectX-2 cards in Windows 10?
 
Last edited:
Joined
Nov 11, 2014
Messages
1,174
SFP+ twinax is fairly likely to work, but not absolutely guaranteed. The only thing that I would expect to work as consistently as possible is to actually get the proper SFP+'s and to use fiber. 10GigE has a very good interoperability track record within the realm of hardware we've been suggesting to users.

I just wanted to agree with you:smile:

I don't know why would one spend $79 on short cooper twinax if you can get 2 optical transiver for $20 each and fiber for another $20 which totals around $60 and will give the proper setup. Dell switch will have its supported dell optics, your 10Gb nic Chelsio will have it's chelsio optics and fiber between, instead of putting a trip-lite twinax between that will make both devices unhappy !:smile: And is not even much cheaper this twinax.
 
Joined
Nov 11, 2014
Messages
1,174
I saw a 5524 on eBay that went for $200 the other day. Wish I could have picked it up, but my wife would have killed me for dumping any more money into this project of mine (at least for now).

I feel your pain :smile:) Try what I call a shock therapy: Keep buying expensive switches and then when you pick something cheaper she won't feel that bad cause she seen worst and kind of being ok with spending less that what could of been.:smile:))
 

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
I just wanted to agree with you:)

I don't know why would one spend $79 on short cooper twinax if you can get 2 optical transiver for $20 each and fiber for another $20 which totals around $60 and will give the proper setup. Dell switch will have its supported dell optics, your 10Gb nic Chelsio will have it's chelsio optics and fiber between, instead of putting a trip-lite twinax between that will make both devices unhappy !:) And is not even much cheaper this twinax.

I'm assuming that $79 figure was pulled from my original post - the $79 was for 2 Chelsio 10Gbps cards and a twinax cable. The twinax cable only cost me $20. Twinax isn't appropriate for all situations, but for a short run between my 2 boxes it has worked perfectly fine and was the lowest-cost option. The only change I've made to my original config was to replace one of the Chelsio cards, because they don't work well in Windows PCs.
 
Joined
Nov 11, 2014
Messages
1,174
I'm assuming that $79 figure was pulled from my original post - the $79 was for 2 Chelsio 10Gbps cards and a twinax cable. The twinax cable only cost me $20. Twinax isn't appropriate for all situations, but for a short run between my 2 boxes it has worked perfectly fine and was the lowest-cost option. The only change I've made to my original config was to replace one of the Chelsio cards, because they don't work well in Windows PCs.

I see wast you are saying, to go fiber you spend $40 for 2 optics and another $12 for optic cable but , it will give you flexibility to use it for many other thing in a future: add a switch, use longer distance, and the advantages of fiber over copper.

This is interesting to know about the chelsio cards. I never got chance to buy one , by the time I understood that the were most recommended for freebsd they were gone of ebay. Perhaps I am glad I didn't buy them if what you are saying is the case. I do have a lot of intel 10Gb cards, but not newer x520 model , but older model , the one that come with optics on it. Never had an issue with freeenas or windows, or ESXI and never had to install or tweak a driver. Cyberjock told me about them , and I've been buying them since.Hope this will help you
 

Caladin9

Dabbler
Joined
Mar 17, 2016
Messages
16
@katit has it right; I installed my fan by screwing it onto the LSI chip's heatsink. I run it at 7V, too, because at 12V it sounds like a swarm of angry bees.

View attachment 8294

Anyone removed the black heatsink and tried another option? Something maybe copper and with a quieter fan?
 
Top