Pheran's 32TB FreeNAS build with photos

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
This writeup is incredible. Bravo.

I wish I had found it earlier. I just went through practically the same exact steps. (Same case, motherboard, similar CPU). One modification I would recommend is updating the supermicro firmware instructions to use version 20, since that's what the driver in FreeNAS 9.3 uses. I had a very similar problem with routing the power cord, but I managed to squeeze it into the large hole on the right near the top, so it's mostly behind the motherboard. I gotta say, your cables in the back look a lot nicer than mine, but that could partly be because that picture is only with 6 drives. Again, very nice post.

I'm glad you liked the writeup.

FreeNAS 9.3 uses the v16 LSI driver. FreeNAS 9.3.1 updated to v20. It's unclear if the SuperMicro version that was out there (20.00.02) is stable, so I've been holding off on any updates. SuperMicro just released their 20.00.04 firmware version - literally today, so I'll be updating to FreeNAS 9.3.1 and the new firmware pretty soon. See this thread for a lot more info.
 

ondjultomte

Contributor
Joined
Aug 10, 2015
Messages
117
Do you have a windows 10 box there? Could you try the Chelsio S310E-CR NIC in that box? I would love to know if its possible to get it up and running in W10.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Yeah, but a lot of the drivers from Vista and newer are forward compatible ...
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Excluding graphics (every version of Windows since Vista has had a new graphics driver model, with 10's being a major change) and biometrics, all drivers written for Vista should work on any version, if proper practices were followed. Oh, and printers. Windows 8 made printers actually not a pain to use. Windows 10 "fixed" that and printers are back to being unreliable pieces of crap. Unicorn crap, judging by ink prices.
 

Brosif_My_Nif

Explorer
Joined
Jan 12, 2012
Messages
62
BigDave, I used the NIC-TOE driver because it appears to be more recent than the Unified Wire version. But I managed to scrounge a different 10G NIC so I'll probably be pulling the Chelsio out of my Windows box pretty soon in an attempt to solve the no display after sleep problem I described in an earlier post.

Pheran, what NIC did you end up going with on your Windows Box?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
On the other hand, it pays to remember that the Chelsio card is probably intended for server use, and therefore there could be some sort of suboptimal driver interaction with Windows.
 

BigDaddy8192

Cadet
Joined
Oct 25, 2015
Messages
1
Thanks for the great build post - I used your build except I used 2 x 120 GB SanDisk Ultra II SSD for boot.

I did want to leave a warning for anyone thinking about SanDisk Ultra II SSDs - they do not fit in the R5 SSD slots - they are slightly too wide. I had to put them in unused 3.5" bays.
 

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
Thanks for the great build post - I used your build except I used 2 x 120 GB SanDisk Ultra II SSD for boot.

I did want to leave a warning for anyone thinking about SanDisk Ultra II SSDs - they do not fit in the R5 SSD slots - they are slightly too wide. I had to put them in unused 3.5" bays.

I'm struggling to understand how that's possible. 2.5" drives have a standard form factor, it's not as if the manufacturers can just randomly alter the drives and expect them to fit into anything.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
I'm struggling to understand how that's possible. 2.5" drives have a standard form factor, it's not as if the manufacturers can just randomly alter the drives and expect them to fit into anything.
Well, there are several heights.
 

ondjultomte

Contributor
Joined
Aug 10, 2015
Messages
117
There are 8 in my R5 now, and although 1 is a cold spare it's cabled for 8. Power will route around the back of the motherboard and only 2 cable ties :D

Think the CPU fan is on a different jumper now too, and the rear fan too. I've also installed a 2nd fan at the front to keep the drives cool. Usually around the 30 degree mark.

freenas1-front.jpg
How do you mirror the USB?
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Please read the manual, link is in the top of the forum (the yellow one) ;)
 

adrianwi

Guru
Joined
Oct 15, 2013
Messages
1,231
How do you mirror the USB?

If you mean physically, I've got 2 USB Sandisk Cruzers attached on the bottom of the motherboard using a small adapter where the front plugs would usually be connected too. If you looks closely between the red led and SATA cables you'll see them :D
 

ondjultomte

Contributor
Joined
Aug 10, 2015
Messages
117
If you mean physically, I've got 2 USB Sandisk Cruzers attached on the motherboard using a small adapter when the front connections would usually be connected too. If you looks closely between the red led and SATA cables you'll see them
Yes i know Internal or externa doesnt matter. How do you mirror them? Is there a bios option to mirror usb drives? Or do u manually mirror them?

Skickat från min HTC One via Tapatalk
 

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
Pheran, what NIC did you end up going with on your Windows Box?

I finally made progress on the 10Gbps NIC front. Some other issues caused me to reinstall my Windows PC from scratch, and I decided to go to Windows 10. I tested the Chelsio card there. It mostly worked, but the whole system did crash (Win10 equivalent of a bluescreen I think) shortly after I installed the drivers. After that the networking did work, but I still had the no display after sleep problem on a freshly installed system. So I ditched the Chelsio from my PC and installed a Myricom 10G-PCIE2-8B2-2S that I had scrounged up. The Myricom has much more recent drivers and doesn't cause any problems coming back from sleep that I've seen. The performance seems to be about the same as before (500MB/s), but it may be limited by my SSD anyway. So overall, the Chelsio works fine in a FreeNAS box but I cannot recommend it for a Windows client.
 

Brosif_My_Nif

Explorer
Joined
Jan 12, 2012
Messages
62
I finally made progress on the 10Gbps NIC front. Some other issues caused me to reinstall my Windows PC from scratch, and I decided to go to Windows 10. I tested the Chelsio card there. It mostly worked, but the whole system did crash (Win10 equivalent of a bluescreen I think) shortly after I installed the drivers. After that the networking did work, but I still had the no display after sleep problem on a freshly installed system. So I ditched the Chelsio from my PC and installed a Myricom 10G-PCIE2-8B2-2S that I had scrounged up. The Myricom has much more recent drivers and doesn't cause any problems coming back from sleep that I've seen. The performance seems to be about the same as before (500MB/s), but it may be limited by my SSD anyway. So overall, the Chelsio works fine in a FreeNAS box but I cannot recommend it for a Windows client.
Thanks for the follow up!
 

Vivan

Dabbler
Joined
Aug 11, 2015
Messages
24
I just finished my (quite similar) build and I actually find the case to be very loud with all 8 drives spinning...
 
Top