Pheran's 32TB FreeNAS build with photos

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
Well, the 10 Gbps addition wasn't quite as clean as I thought. There's no problem with FreeNAS, but my Windows 7 PC fails to activate the display when waking up from sleep since installing the Chelsio card. This is really annoying. I'll try to find a workaround but if not I'm going to look into an alternate 10G NIC for the PC side. I may even be able to scrounge one from work.
 

TheDubiousDubber

Contributor
Joined
Sep 11, 2014
Messages
193
Glad I came across this. Awesome write up. I've been looking into (cheap as possible) ways of getting 10GbE for backups between two FreeNAS servers, freeing up the main network for serving content, etc. Seems your approach may be the best way to go.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
It might be, but it complicates stuff. Check out the 10Gig primer where we discuss the Dell 5524. The Dell 5524 and two single port SFP+ 10G cards is a pretty cheap way to go AND on top of it you get full gigE capabilities to multiple clients simultaneously.
 

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
It might be, but it complicates stuff. Check out the 10Gig primer where we discuss the Dell 5524. The Dell 5524 and two single port SFP+ 10G cards is a pretty cheap way to go AND on top of it you get full gigE capabilities to multiple clients simultaneously.

These aren't mutually exclusive - in fact you could look at my solution as a stepping stone to this configuration, because you get the two NICs you need first. Then you can add the 5524 (or another switch) later if you like. You have to do a bit of network reconfiguration to get rid of the extra subnet, but otherwise it's a simple upgrade path.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Based on the networking prowess of many of the posters who have tried multiple interface configurations in the past, they do not find it a "simple upgrade path" or "a bit of network reconfiguration" (that they understand, at least). Most of the posters here don't find networking to be trivial, even if I do.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Based on the networking prowess of many of the posters who have tried multiple interface configurations in the past, they do not find it a "simple upgrade path" or "a bit of network reconfiguration" (that they understand, at least). Most of the posters here don't find networking to be trivial, even if I do.

+1

Most people's minds lock up when the consider two subnets, side-by-side.

"That's even possible?!"

"Uh... how does it know how to route traffic!?"

"Are you sure it actually works!?"
 

TheDubiousDubber

Contributor
Joined
Sep 11, 2014
Messages
193
I saw a 5524 on eBay that went for $200 the other day. Wish I could have picked it up, but my wife would have killed me for dumping any more money into this project of mine (at least for now). I considered taking this approach, but may hold off on everything altogether. Pheran is mostly right, though I wouldn't say simple upgrade. Perhaps better to say reasonable upgrade path. I don't find networking to be trivial at all, quite difficult in fact, but I am eager to learn more and understand why. If that isn't your attitude and you have the same or even less knowledge than I, then why would you be spending hundreds of dollars on a faster setup that you have no idea how to manage? Maybe that's not the point and maybe it's merely Pheran's wording, but I think its a great solution for an introduction into 10GbE without having to blow a stack of money on a switch.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I just bought a 6224 for $175 on ebay last week. :D
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,458

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I think that was me. The shipping was a bit high,but I still figured it was worth it.

For $200? Fsck. I'd have picked one up for that, can always find a use for another good switch.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Please can you post the online shop links to the fan and screws you used for this.
@trumee, I cut down some screws from an old heat sink I had lying around my shop, so I'm sorry to say that I can't help you with part numbers.
 

Vivan

Dabbler
Joined
Aug 11, 2015
Messages
24
I'm really struggling to pick a CPU cooler for my build... all I want is something silent and better than the stock intel fan :[
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
The NICs arrived today so I installed them into the FreeNAS server and my Windows 7 box. The install was practically effortless. FreeNAS immediately recognized the card and gave me a new cxgb0 interface. I downloaded the Windows drivers from Chelsio's support site and installed them with no trouble.
@Pheran
I was wondering which driver for Windows 7 that you used. There seems to be more than one...
Here's a screen shot (click the tumbnail) of the websites choices
Chelsio-Driver.jpg
Thanks for posting this, I was inspired and just purchased two cards complete with tranceivers today :D
 

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
To respond to the comments about using two subnets, yeah, I understand that it may be confusing for network novices. I'm a networking expert so unfortunately I don't have a very good perspective when it comes to understanding how difficult a networking task may be for someone else. In the case of the direct-linked 10G NICs, it's merely a matter of adding two IP addresses that aren't on the same subnet as your main network, then making sure that your 10G client actually uses the 10G IP rather than the regular one to connect. But you are absolutely right that it does add a complication to the network that is eliminated by having a switch with a few 10G ports. It always comes down to personal choice - are you willing to spend the few hundred dollars it will cost to eliminate that complication? In my case it made little sense but it might make sense for someone else.
 

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
@Pheran
I was wondering which driver for Windows 7 that you used. There seems to be more than one...
Here's a screen shot (click the tumbnail) of the websites choices
View attachment 9016
Thanks for posting this, I was inspired and just purchased two cards complete with tranceivers today :D

BigDave, I used the NIC-TOE driver because it appears to be more recent than the Unified Wire version. But I managed to scrounge a different 10G NIC so I'll probably be pulling the Chelsio out of my Windows box pretty soon in an attempt to solve the no display after sleep problem I described in an earlier post.
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
BigDave, I used the NIC-TOE driver because it appears to be more recent than the Unified Wire version. But I managed to scrounge a different 10G NIC so I'll probably be pulling the Chelsio out of my Windows box pretty soon in an attempt to solve the no display after sleep problem I described in an earlier post.
You're not kidding, the unified driver is from 2009 :eek:
When I purchase a managed switch (next purchase) with some SFP+ ports, my goal is to hook my desktop and the FreeNAS server with fiber cable and soon add my sons desktop after
getting mine all configured. Thanks for the driver heads up, I'm just trying to have everything I need (driver) when the hardware is all purchased. I hate waiting to the last minute ;)
 

jab416171

Cadet
Joined
Sep 26, 2015
Messages
9
This writeup is incredible. Bravo.

I wish I had found it earlier. I just went through practically the same exact steps. (Same case, motherboard, similar CPU). One modification I would recommend is updating the supermicro firmware instructions to use version 20, since that's what the driver in FreeNAS 9.3 uses. I had a very similar problem with routing the power cord, but I managed to squeeze it into the large hole on the right near the top, so it's mostly behind the motherboard. I gotta say, your cables in the back look a lot nicer than mine, but that could partly be because that picture is only with 6 drives. Again, very nice post.
 
Top