BUILD Configuration Recommendation

[Chris Tobey]

Hi Everyone,

I have a new SuperStorage Server 6048R-E1CR24N from SuperMicro (https://www.supermicro.com/products/system/4U/6048/SSG-6048R-E1CR24N.cfm) that I would like to use a new FreeNAS build and would like some input as to the optimal configuration I can make, as well as any hardware I might want to reconsider.

So here is the parts breakdown:

Chassis: SuperMicro 6048R-E1CR24N
Motherboard: SuperMicro x10DRi-T4+
CPU: 2 x Intel E5-2620v3 (6 core/12 thread @ 2.4GHz per)
RAM: 12 x SuperMicro DR416L-SL01-ER21 (Samsung, 16GB DDR4 2133 ECC Registered)
HBA: SuperMicro AOC-S3108L-H8IR-O-P (LSI MegaRAID 8-port SAS 12Gb/s)
HDDs: 12 x Seagate ST4000NM0023 (4TB SAS 6Gb/s)
SSDs: 2 x Samsung 840 Pro 128GB
Network: ChelsIO T420-LL-CR (2 x SFP+ for 10G Fibre)
UPS: APC SMT3000RMT2U

So when I went to set this up I ran into my first problem, being that the SuperMicro HBA was a MegaRAID card with the "SAS-MFI BIOS". I scrapped that and ordered an LSI SAS 9300-8i (with the 3008 chip) which I believe should give me direct drive access.

The server will be used primarily for NFS exports with one SFP+ connected with 10G fibre to my main switch as the only network connection.

So my questions are:
1. Are there any problems with this proposed setup?
2. What do I need to do to flash / configure the LSI SAS 9300-8i?
3. What is the best configuration of the 12 HDDs and 2 SSDs? (2 x 6 RAIDZ-2 + Mirrored ZIL?)
4. Does anyone know which driver option to use for the APC SMT3000RMT2U (upgraded to latest firmware)

Thanks everyone!

 

[danb35]

I scrapped that and ordered an LSI SAS 9300-8i (with the 3008 chip) which I believe should give me direct drive access.

It should, but I'm not certain that HBA is quite stable with FreeNAS. @cyberjock?

 

[Fuganater]

You need that much power for a NFS ??

 

[Ericloewe]

LSI SAS3 stuff is still a bit experimental. Not something I'd recommend for production use.

2. What do I need to do to flash / configure the LSI SAS 9300-8i?

sas3flash, included with FreeNAS or available from LSI for DOS, UEFI shell, Windows, Linux, FreeBSD and who knows what else.

 

[Chris Tobey]

That's too bad, it just came in and I was hoping to have the future option of SAS3 devices (even though the ones I have right now are only SAS2).

I meant is there a specific firmware version for 9.3.1 I should use (v9?) and is there special "IT mode" firmware?

Does a 2 x 6 drive RAIDZ-2 with a mirrored SSD ZIL make sense?

 

[Chris Tobey]

You need that much power for a NFS ??

The company I work for wanted to buy a 20k server, with 4TB of memory, I decided to build one 8x bigger for half the cost, so lots of power was easy to get :)

 

[Chris Tobey]

What's the best HBA option? I have the SuperMicro BPN-SAS3-846EL backplane expander that supports the 24 SAS3 drives with two SFF-8643 (miniSAS-HD) cables.
Preferably I would use the SAS3 HBA.
I could use a SAS2 HBA and then buy new miniSAS->miniSAS-HD cables if the SAS3 is really, Really, REALLY not recommended. If so, which one?

 

[cyberjock]

LSI SAS3 stuff is still a bit experimental. Not something I'd recommend for production use.

+1. Keep the card since its bought and paid for. It's virtually guaranteed to be well supported in a year or two (see what i did there since it's still 2015 where I am?) but definitely don't use it right now if you plan to use it for storing real data.

 

[Chris Tobey]

Does a 2 x 6 drive RAIDZ-2 with a mirrored SSD ZIL make sense?

The drives will primarily be NFS with some CIFS.

 

[Chris Tobey]

So I set this up today.

I flashed the LSI SAS 9300-8i with the "SAS9300_8i_IT.bin" P9 via UEFI (firmware from the 9300_8i_Package_P9_IR_IT_Firmware_BIOS_for_MSDOS_Windows.zip package) and it seems to be working correctly. I have tried searching to find out what the issues with the SAS3 HBAs are and did not find much recently. Most people seems to not be able to see their drives, but mine are ok. @cyberjock do you know of any issues I should look for?

I ordered an IBM M1015 per the general consensus on the forum, but if I can use the SAS3 controller that would be better :).

I set up my pool with 1 x 6 drive RAIDZ-2 with a mirrored SSD ZIL and it works well. Does it make sense to extend the pool with a second VDev of 6 drives in RAIDZ-2 or is there a better configuration?

 

[cyberjock]

I know there "are" issues. But as I've never had a 9300 to play with before, I can't really say. Unfortunately when I read stuff on the forum its hard to tell if the problems are because the end-user doesn't know what they are doing or if they know their shiz and there is actually a hardware problem of some sort. Without having someone that I know and trust that has played with the hardware first-hand its tough to discuss the exact problem.

Virtually all of the experienced users around here haven't bought one yet (they aren't worth the investment right now) and iXsystems doesn't use them yet. I definitely do not need to upgrade and so spending money on one right now is just a waste of my money (doubly so if it doesn't work).

 

[Chris Tobey]

Well, I guess unless you see me here in a few days yelling "HELP", then the SAS3 controller is working well :)
If that does become the case, my replication backup should come in handy along side the M1015.

 

[Chris Tobey]

Two months in and my LSI SAS 9300-8i is working well :)

 

[Chris Tobey]

Five months strong! Even added an additional 6-drive RAIDZ-2 and all is well.

 

[Chris Tobey]

8+ Months with the LSI SAS 9300-8i. No issues. :)

 

[Chris Tobey]

12+ Months with the LSI SAS 9300-8i. No issues.

 

[ewhac]

12+ Months with the LSI SAS 9300-8i. No issues.

https://youtu.be/etb2LGjY3i0

 

[Darren David]

12+ Months with the LSI SAS 9300-8i. No issues.

Belated question - can I ask what firmware you're running? I've got the same card running v16.00.01.00 of the firmware (latest available) on FreeNAS 11.2 and I'm seeing intermittent checksum errors that degrade my zpool weekly on brand new drives.

 

[MikeyG]

@Darren David I have a 3008 running 16.00.00.00 and was getting checksum/cam status errors recently until I replaced one of my host swap cages. Also did not show up until I installed 11.2, but as it's been stable about 3 weeks since the replacement I'm thinking it was flaky connectors or cabling.

 

[Darren David]

@Darren David I have a 3008 running 16.00.00.00 and was getting checksum/cam status errors recently until I replaced one of my host swap cages. Also did not show up until I installed 11.2, but as it's been stable about 3 weeks since the replacement I'm thinking it was flaky connectors or cabling.

Fingers crossed for flaky cables, then. Although I have not found any potential replacement for these...