124 bay 12Gbps SAS3 setup

[Mikkel]

Hey all,

I've been wondering about pool setup in a large NAS. I'm about to order a NAS for FreeNAS that uses:

1x SuperStorage Server 6048R-E1CR36L
2x SuperChassis 847E1C-R1K28JBOD
2x Intel® Xeon® Processor E5-2660 v3
16x 32GB DDR4 1.2V-2133 Registered ECC
124x 6TB Seagate SAS3 12Gbps 4KN
Dual 40Gbit uplinks

The goal is to have 300TB of high performance sequential space as it is used to store/serve master copies of digital video (120-400Mbps video).

Right now I'm thinking 11x Z3 (8+3) but 20x Z2 (4+2) could be an alternative - I'm just not a 100% sure what is best on paper in terms of performance and stays above 300TB usable space.

Any input is very welcome :)

 

[zambanini]

the lsi sas3 driver are not labeled as stable, I guess you might be one of the first big users with that chips and freenas ; )

 

[Mikkel]

the lsi sas3 driver are not labeled as stable, I guess you might be one of the first big users with that chips and freenas ; )

Where do I see what is considered stable in terms of LSI chips? :)

 

[zambanini]

read the sticky posts in the forum. basically you should be clear, that freenas is the beta test pool for the commercial TRUENas. maybe you take a look at it.

https://forums.freenas.org/index.php?threads/confused-about-that-lsi-card-join-the-crowd.11901/

 

[Mikkel]

They won't sell me TrueNAS as a software package, I've tried that, and I'm not interested in paying an obscene markup for SuperMicro hardware that I can get much cheaper/faster locally.

Anyway mpr(4) for LSI SAS3 products was included in FreeBSD 9.3-RELEASE so I can only assume it's considered stable by FreeBSD.

 

[jgreco]

Having a driver included in FreeBSD doesn't guarantee stability of the sort we're talking about. There's a difference between "the device driver author tested it extensively and encountered no problems" and "the installed FreeNAS base has pushed through ten thousand server-years of data on it without incident."

We tend to trust the latter. The former is an example of the bleeding edge.

 

[Ericloewe]

LSI SAS2 (SAS 2008/2308) is quite stable these days. SAS3 is scheduled to get a major update soon that will improve stability, but it's still a bit experimental.

 

[L]

I agree with a comment above. If you are going to build something that large out of commodity components, I would at least get a quote from IX on an identically configured hardware + software platform. They know where the outlayers/gothchas/caveats are..and they can help if you have problems. I have started to resell their products and have been amazed at how reasonably priced they are..

That being said... You have 744TB of raw storage. If you are looking to get 300TB usable, you can easily use either mirrored pairs or raidz 2+1 sets. They key better performance is ..more vdev's better performance. If you want to diy, read this before you proceed. http://blog.delphix.com/matt/2014/06/

Driving dual 40gbe's isn't easy. The oracle zfs appliance runs with 8-10gbe's but that is not easier. That's a lot of network to drive. You will have to test to see how it will drive and what combo is the sweet spot for your configuration.

Most of the big boxes I work with run raidz or raidz2 set of 9-12 disks, but aren't driving that kind of uplinks.. You may also want to check to make sure you have a driver for those 40gbe's. Check the freebsd 9.3 hardware compat lists. The boxes I have designed that run that kind of network(haven't done 40gbe, but have run as many as 10-10gbe, but never all at once) usually have a lot of vdev's.

 

[cyberjock]

They won't sell me TrueNAS as a software package, I've tried that, and I'm not interested in paying an obscene markup for SuperMicro hardware that I can get much cheaper/faster locally.

Anyway mpr(4) for LSI SAS3 products was included in FreeBSD 9.3-RELEASE so I can only assume it's considered stable by FreeBSD.

Good, because the new TrueNAS isn't build on Supermicro.

And the markup is because iX is handling things like compatibility with hardware, drivers, etc. You have a problem with a TrueNAS under contract, you call iX and its their problem to fix. You have a problem with your LSI controller, you are on your own to resolve the problem. So it's not just obscene markup. And if you think iXsystem's has "obscene markup" you should look at what other companies like NetApp do.

Good luck to you! Large servers have very unique challenges that almost nobody here will have experience to assist you with.

 

[diehard]

Is there really any point to using SAS3 ATM, with mechanical drives?

 

[cyberjock]

Is there really any point to using SAS3 ATM, with mechanical drives?

Honestly, not really. SAS3 is great for SSD, but with the whole advent (and support) for NVMe SAS3 is something that is kinda-sorta obsolete right from the get-go. It'll be nice for those of us that want physically small systems that have few PCIe slots or can't afford NVMe. But for anyone wanting any kind of "real" workload in a company, its a tech that is basically dead thanks to NVMe.

 

[diehard]

Honestly, not really. SAS3 is great for SSD, but with the whole advent (and support) for NVMe SAS3 is something that is kinda-sorta obsolete right from the get-go. It'll be nice for those of us that want physically small systems that have few PCIe slots or can't afford NVMe. But for anyone wanting any kind of "real" workload in a company, its a tech that is basically dead thanks to NVMe.

I wonder if we will start to see SFF-8639 backplanes from OEM's anytime soon.

 

[HeloJunkie]

Good, because the new TrueNAS isn't build on Supermicro.

And the markup is because iX is handling things like compatibility with hardware, drivers, etc. You have a problem with a TrueNAS under contract, you call iX and its their problem to fix. You have a problem with your LSI controller, you are on your own to resolve the problem. So it's not just obscene markup. And if you think iXsystem's has "obscene markup" you should look at what other companies like NetApp do.

Good luck to you! Large servers have very unique challenges that almost nobody here will have experience to assist you with.

@cyberjock when did this change? I am looking at their Neptune system and they told me it was a supermicro solution:

http://www.ixsystems.com/servers/families/?family=Neptune 2200

 

[cyberjock]

@cyberjock when did this change? I am looking at their Neptune system and they told me it was a supermicro solution:

http://www.ixsystems.com/servers/families/?family=Neptune 2200

That's not for TrueNAS...

http://www.ixsystems.com/storage/truenas/

 

[HeloJunkie]

Ah...what is the difference between their product lines (say the truenas and the neptune)..? Just different hardware platforms?

 

[cyberjock]

Ah...what is the difference between their product lines (say the truenas and the neptune)..? Just different hardware platforms?

Yeah. iXsystems sells lots of servers besides just file servers. You can call for Linux, FreeBSD, Windows, FreeNAS Certified, etc. But TrueNAS has it's own very special and very well supported platform. If there's a problem with a driver that is found that is on TrueNAS hardware devs are thrown at it to fix it, pronto. If it's something else it's fixed depending on how big of a deal it is and how much effort iX feels should be spent on it. Some stuff is given relatively high importance if it has a potentially catastrophic bug that affects a wide audience. But the fix may not be released immediately. TrueNAS is iX's special software, and if some really ugly bug was found, you can bet a release would be pushed out immediately.

So to a certain extent, self-supported is cheaper, but things can (and for an unlucky bunch, they do) go wrong.

You will find that even at the 10Gb networking, supported hardware is very limited. It's even more limited at higher speeds, and you mentioned 40Gb. So it will either be a slam-dunk and work perfectly if everything for 40Gb is on FreeNAS (iX sells no 40Gb FreeNAS servers as far as I know); it will work, but will be buggy and fraught with problems; or its something that never works right and you have little to no chance of ever getting a solution.

There is a bunch of wise anecdotes just in this forum alone that say that when building for large scale and/or high speed, you are better off making your reliability and performance someone else's problems. It's one thing when you buy it from iX and when it performs poorly (or not at all) you call iX and make them fix it. It's another when it performs poorly (or not at all) and your boss calls you and expects you to fix it, even if its a software code or driver problem. You might have saved the company a bunch of money, but when you go to work and the server is too slow, that money saved doesn't absolve you of the responsibility of keeping the server up and fast. Your boss will without a doubt expect you to keep that server running at peak efficiency.

So like I said above. I wish you luck (this is not sarcasm either). I've seen both sides of the story, but too many of them ended with a customer saying they will buy TrueNAS but they will require some assistance to help with data migration. And that's often code for "Our previous IT guy is no longer employed by this institution".

 

[HeloJunkie]

Hey @cyberjock - thanks for the intel. I am not the OP with the huge system, I was just reading the thread and saw your comment about the hardware being different and was puzzled as when I called them they quoted me a supermicro platform. I had no idea they had different levels of hardware that they sold and supported, this is great information.

Thanks again for the info!

 

[cyberjock]

Yeah, I hit submit before I realized it. I'm an hour past bedtime. :P

Hopefully the info helps someone else. :)

I had no idea when I first started working at iX that TrueNAS is only a small part of their business. They sell a cubic crapload of servers that aren't file servers at all. They also sell a crapload of servers that are file servers for Windows, Linux, whatever you need. But TrueNAS is really the product that iXsystems is really proud of and supports very well.

 

[HeloJunkie]

That is good to know, I typically only buy Supermicro and if they sell servers I will give them a try next time I need a new box. I just spent almost $35K on a quad node supermicro system for some new domain controllers!

 

[fullspeed]

Hey all,

I've been wondering about pool setup in a large NAS. I'm about to order a NAS for FreeNAS that uses:

1x SuperStorage Server 6048R-E1CR36L
2x SuperChassis 847E1C-R1K28JBOD
2x Intel® Xeon® Processor E5-2660 v3
16x 32GB DDR4 1.2V-2133 Registered ECC
124x 6TB Seagate SAS3 12Gbps 4KN
Dual 40Gbit uplinks

The goal is to have 300TB of high performance sequential space as it is used to store/serve master copies of digital video (120-400Mbps video).

Right now I'm thinking 11x Z3 (8+3) but 20x Z2 (4+2) could be an alternative - I'm just not a 100% sure what is best on paper in terms of performance and stays above 300TB usable space.

Any input is very welcome :)

I am running a similar setup, although not quite as beefy.. I have two of these @ 326TB RAW, 7 x 12 disk raidz3s (mix of 4TB / 6TB )

Chassis - Dell R510 + 2x SuperMicro SC847J
OS - FreeNas 9.3-STABLE
Network - 10GBe Failover
RAM - 128GB
HBA - LSI 9300E
zil ~ raidz1 ~ 2x200GB SSD (Write Cache)
l2arc ~ JBOD ~ 4x512GB SSD (Read Cache)

I would suggest not using 12G/s, although it "works" i've had lots of good drives error out with this controller and i'm not the only one. Not to mention performance has not been stellar, i'll probably be downgrading.