Tuning for 10Gbe, 256GB RAM, and 32 cores?

Status
Not open for further replies.

jasonlitka

Dabbler
Joined
Aug 30, 2012
Messages
25
Anyone have any suggestions for tuning an install for a high-end box? I'm repurposing an IBM server with 2x 10Gbe, 32 cores (16 physical, 16 logical), and 256 GB of RAM as the control unit for a new storage box (42x 4TB SAS drives in mirrored pairs, 3x spares, 2x mirrored 100GB eMLC SSDs for Log, 2x striped 240GB SSDs for L2ARC).

My client network is wired in such a way that it is possible for all clients to connect at once and all to be able to pull a full Gig-E without contention assuming the source supports it, so it's real likely that this box will get hammered.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
Perhaps other people can pipe in about the ZIL and L2ARC, but I wouldn't add those until you think you are hitting a bottleneck.

The L2ARC will steal RAM from the ARC(stored in RAM) to keep the L2ARC organization in RAM. So you're trading RAM for L2ARC. It's small, but at your scale it may matter.

Also, ZIL typically doesn't need to be very big. Somewhere I had read that just 16GB was a huge overkill for most installations.

Have you actually installed FreeNAS yet on the machine? Is all of the hardware detected and functional? Someone had mentioned in another thread that 10Gbe wasn't supported for most brands aside from Intel. Also you didn't mention what SAS controller you are using, but that could be a no-go too.

It seems if you spend too little money you get hardware that isn't going to work well, and if you spend too much it won't work either. It has to be "just right". Like the children's story The Three Little Bears. :P
 

survive

Behold the Wumpus
Moderator
Joined
May 28, 2011
Messages
875
Hi jasonlitka,

If there was ever somebody who ought to contact ix Systems, the company behind FreeNAS & TrueNAS, and get a quote on a TrueNAS system I think you are it.

Give'em a call, I'm sure thet will set you up with something nice.

-Will
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
I think he already owns the hardware since he said he's repurposing it. If he doesn't own the hardware, I agree that ix systems should be contacted.
 

jasonlitka

Dabbler
Joined
Aug 30, 2012
Messages
25
Also, ZIL typically doesn't need to be very big. Somewhere I had read that just 16GB was a huge overkill for most installations.

Have you actually installed FreeNAS yet on the machine? Is all of the hardware detected and functional? Someone had mentioned in another thread that 10Gbe wasn't supported for most brands aside from Intel. Also you didn't mention what SAS controller you are using, but that could be a no-go too.

A pair of 8GB ZeusRAMs weren't in the budget for this project, at least not at this point (I may plug them in for next year and see if it gets approved). I went with the OWC Enterprise 100GB SSDs because of their TBW ratings of 1433TB, not because of the usable size.

All the hardware is supported. The NIC is the Intel X520-DA2, SAS controller the LSI 9200-8e.

Hi jasonlitka,

If there was ever somebody who ought to contact ix Systems, the company behind FreeNAS & TrueNAS, and get a quote on a TrueNAS system I think you are it.

Give'em a call, I'm sure thet will set you up with something nice.

-Will

I've got one of their systems for my backup target and I'm a little torn on them. They sold me a system with two DOMs for the OS which they said could be used in RAID 1. This turned out to not be true, and while they gave a refund for the hardware that couldn't be used, plus a little more on top, it still left a bad feeling behind. If I had bothered to install FreeNAS myself in advance I would have seen that that wasn't possible, and I really think that someone selling a product should know enough to be able to speak to an issue identified with 5 minutes of turning on the box. They also shipped the DOM with a 32-bit install even though the box they sold me had 48GB of RAM.

I haven't needed any significant post-sales tech support yet so I can't speak to that.

This is an in-house effort because aside from the drives and the JBOD chassis I already had the hardware sitting here unused.

I think he already owns the hardware since he said he's repurposing it. If he doesn't own the hardware, I agree that ix systems should be contacted.

See above.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
WOW. That sounds like a semi-crappy experience. I've never interacted with them at all, nor heard anyone with experience with them. That is a little disappointing though.
 
Status
Not open for further replies.
Top