Trying to right a wrong and build a "big" nas. 70+ disks

[EnragedZeus05]

hello,

I have been lurking on the TrueNas Forum and on reddit for a few months now, mostly just reading up to prep myself to correct a massive wrong that i have frakensteined over the last 10-15 years. I currently have a server in a massive hardware raid 60 w/ 24 6tb disks and some SSDs in a r620XD im currently using. Yes, i know how bad it is but it sort of grew into this monster from only 2 2tb disks. i have a total of 110tb of data that i am storing.

I am finally ready to fix this and have "most" of the hardware but i have a few questions that i cannot locate on this forum, reddit, google, or i just cant understand it for some reason.

My case use is file storage only. Plex storage, pictures, personal cloud, etc. No VM's but i might get a few SSDs and try to make a pool out of those for games.

The hard drives i am aiming to get is HGST 6tb enterprise helium drives (used/ebay) which will be stored in a NetApp DS4486 (48 bay drive enclosure). I do have two of these enclosures that i will eventually put more 6tb drives in or possibly 8tb drives in down the road for growth.

I also have a NetApp DS2246 which holds 24 2.5 SSDs which i will be using to multiple pools to host my plex DB, stuff i need to either archive/unpack, and lastly stuff that i need speed for transferring over the network.

i have acquired a Dell R730XD barebones (only missing ram/CPU/storage cards).

So, as it sit i only need the CPUs, RAM, NIC, and the HBA's. I mostly need help hammering out which HBA's ill need to connect to the diskshelf and to the local drives in the R730XD, CPU, and 10gb NIC.

I have been eye balling the Intel XEON E5-2640 2.6Ghz 8 core cpu but i can also get a 4 core 3.0 Ghz cpu if that is the better route. I planned on getting 1 CPU 2/ 64gb of ram to start. also, would 2 CPU's be better/needed for performance?

The HBA's i am getting confused a lot. Do i need card with more ports? do i need a sas expander?

10gb NICs seem to be all over the place while some have issues and some dont. Was hoping to get feed back on the "best" card that works out of the box.

Thanks for any replies and advice to get me going in the right direction. i really need to do this right the first time because i am not looking forward to moving 110tb :/

 

[sretalla]

Do i need card with more ports? do i need a sas expander?

Depends on the enclosures you have as the backplanes may already be effectively acting as expanders (I'm not directly familiar with the model you mention, but have a look at how many Mini-SAS ports it has on it, that will be the clue to work with.

Expanders would help if you have more ports to connect to on the backplane than you have on the HBA (which can typically support hundreds more disks than the number of ports would indicate via expanders).

would 2 CPU's be better/needed for performance?

Only for running VMs or Transcoding plex media.

Was hoping to get feed back on the "best" card that works out of the box.

I see a lot of people saying the Chelsio cards work well.

I would normally be looking at Intel if I was going to 10G, but watch out for the different chipsets as there's good support for one and not so much for the other.

i really need to do this right the first time because i am not looking forward to moving 110tb

Make sure you plan carefully with your pool layout to account for how you want to expand later... you can't just add disks one by one with ZFS like you might have with hardware RAID.

 

[blanchet]

I am not sure if TrueNAS Core will work well on your Dell PowerEdge R730XD. It really depends on the model of PERC and NIC you have. On the other hand, TrueNAS Scale should work

Normally, you need only one LSI 9300-8e HBA to connect your NetApp disk shelves.

Host Bus Adapters

Broadcom Host Bus Adapter (HBA) cards can enable an easy, long-term storage growth strategy in practically any direct-attached storage scenario.

www.broadcom.com

If you have more than 2 disks shelves to connect, you can try to daisy-chain them like NetApp does, or else purchase another LSI 9300-8e.

Warning: your Netapp DS4486 contains two SATA disks per carrier. For such disk shelves, OnTap supports hot-swapping by copying automatically the good disk to a spare before pulling the tray, but there is nothing similar on TrueNAS.
So you should not hot-swap the disks with TrueNAS, otherwise you will lose two disks at once.

https://library.netapp.com/ecm/ecm_download_file/ECMP1113327

 

[NugentS]

That's a hell of a lot of storage for personal use.
You need LSI Cards (not MegaRAID) running in IT mode for this or equivalents. The most important bit is IT mode. How many and of what kind depends on as @sretalla says the makeup of your enclosures. I would be looking at the 9300's as they have a much better throughput than the 9200 and for the number of disks you have with the expanders..... 9300's are well tested as well.

For NIC's (SFP+)- Chelsio T520, Intel 720, Intel 540, Chelsio T420. Some people have had good luck with Solarflare - but I have never used one, and I am not sure which models
For 10GBe Chelsio T520, Intel X550, Chelsio T420, Intel X540

Top Picks for FreeNAS NICs (Networking)

We have 1GbE, 10GbE (SFP+ and 10Gbase-T) and 40GbE networking options for the top FreeNAS NICs as storage performance is network dependent.

www.servethehome.com

Oh and as @sretella says be very careful with your pool designs - getting that wrong will be very expensive in time and potentially data.

How are you backing up 110TB?

 

[Redcoat]

Some people have had good luck with Solarflare - but I have never used one, and I am not sure which models

I have good experience with SN6122F (used on ebay, about $50 each). Others have reported similarly.

 

[EnragedZeus05]

Depends on the enclosures you have as the backplanes may already be effectively acting as expanders (I'm not directly familiar with the model you mention, but have a look at how many Mini-SAS ports it has on it, that will be the clue to work with.

Expanders would help if you have more ports to connect to on the backplane than you have on the HBA (which can typically support hundreds more disks than the number of ports would indicate via expanders).

Make sure you plan carefully with your pool layout to account for how you want to expand later... you can't just add disks one by one with ZFS like you might have with hardware RAID.

-Ah, quick google search and i think the IOM's do work as a sas expander. so, i think im safe and do not have to deal with SAS expanders.

When im ready to build out i usually only add 24+ disks at a time so no worries for a "single disk" add for me.

I am not sure if TrueNAS Core will work well on your Dell PowerEdge R730XD. It really depends on the model of PERC and NIC you have. On the other hand, TrueNAS Scale should work

Normally, you need only one LSI 9300-8e HBA to connect your NetApp disk shelves.

Host Bus Adapters

Broadcom Host Bus Adapter (HBA) cards can enable an easy, long-term storage growth strategy in practically any direct-attached storage scenario.

www.broadcom.com

If you have more than 2 disks shelves to connect, you can try to daisy-chain them like NetApp does, or else purchase another LSI 9300-8e.

Warning: your Netapp DS4486 contains two SATA disks per carrier. For such disk shelves, OnTap supports hot-swapping by copying automatically the good disk to a spare before pulling the tray, but there is nothing similar on TrueNAS.
So you should not hot-swap the disks with TrueNAS, otherwise you will lose two disks at once.

https://library.netapp.com/ecm/ecm_download_file/ECMP1113327

I do plan on doing lots of testing with truenas core before i decom my current set up and sell off the drives and disk shelf but im hoping core works as i have not looked into scale at all. I currently am running multiple shelves daisy chained and it does work w/o issue so im hoping i dont have issues with core with that.

I will have to look more into this dual caddy set up. i anticipated just using all 48 disks in total and keeping a few spares on hand for when one dies. i anticipated having to power down the NAS to replace the disk since they are dual caddy though.

That's a hell of a lot of storage for personal use.
You need LSI Cards (not MegaRAID) running in IT mode for this or equivalents. The most important bit is IT mode. How many and of what kind depends on as @sretalla says the makeup of your enclosures. I would be looking at the 9300's as they have a much better throughput than the 9200 and for the number of disks you have with the expanders..... 9300's are well tested as well.

For NIC's (SFP+)- Chelsio T520, Intel 720, Intel 540, Chelsio T420. Some people have had good luck with Solarflare - but I have never used one, and I am not sure which models
For 10GBe Chelsio T520, Intel X550, Chelsio T420, Intel X540

How are you backing up 110TB?

Do you have any recommendations of a 9300 card? someone suggested a 9300-8e. Thanks for tossing up the model numbers on the NIC's, ill be searching ebay today to grab one.

I am not backing up all 110tb of storage as that would make me go broke. i am backing up though what is personal to me. IE pictures etc in the cloud and i have a small computer at my sisters that i use as well.