Recommendation on HBA for Massive 700TB TrueNAS Build

[j05iah]

Hi! I am new to TrueNAS and excited to get my first 40 drive build done. Also, this is my first time ever posting in a forum. I ran into trouble getting all the drives to show up and it looks like the problem was cheap HBA's. Would anyone be willing to offer some advice on a specific HBA that you think would work for my build? I've been reading and researching and it's not entirely clear to me whether I can use an LSI HBA expander card without a controller. The hardware for my build is:

Hardware:
- Asus WS X299 SAGE/10G motherboard
- Intel Core i9-10980XE Extreme Edition Processor
- 128 GB of RAM
- 40 (forty) Seagate 18TB IronWolf Pro NAS drives [ST18000NE000] for storage; I will choose whatever zfs is closest to RAID6 which I think should give me somewhere around 650-700TB of storage
- 1 (one) Sabrent Q NVMe PCIe M.2 2280 SSD for the boot drive

My Question:
Can you recommend an HBA setup that you think might make this work? I've tried several cheap no name ones that did not work, then I found the forum and realized the LSI ones were more what I need. But, can you recommend a specific one and do I need expander cards plus a controller?

Thanks for your help!

Josiah

 

[HoneyBadger]

Hold up a moment before you get too far. You've got an "extreme performance" high-end desktop motherboard and processor here, but what you're building is a server. You don't need an 18-core 165W TDP monster (that doesn't even support ECC, for that matter) - while overkill is always fun to build and fun to have, it's better to spend your money in a targeted manner. A weaker processor with more RAM is likely going to be an overall better solution.

What's the goal here, besides "lots of TB's in a box" - are you going to be serving video (HD/4K repository) or smaller files? Is this for home or business?

 

[Chris Moore]

I will choose whatever zfs is closest to RAID6

That depends on how many vdevs you are going to make in your pool of drives. We need to talk about your performance expectations so we can make some informed recommendations.
Depending on what you want to do with this, it might be better to have more drives with a smaller per-drive capacity so you can have more vdevs. Vdevs give you IOPS.

I ran into trouble getting all the drives to show up and it looks like the problem was cheap HBA's.

Exactly what hardware are you using?

Please read:

Forum Rules

Welcome to the TrueNAS Community! Updated 11/28/2023 First and foremost I want to thank you for joining the TrueNAS community! My name is JoshDW19 and I'm the Community Manager here at TrueNAS. The TrueNAS Community is a friendly place to discuss open source software, technology, and products...

www.truenas.com

 

[j05iah]

Hold up a moment before you get too far. You've got an "extreme performance" high-end desktop motherboard and processor here, but what you're building is a server. You don't need an 18-core 165W TDP monster (that doesn't even support ECC, for that matter) - while overkill is always fun to build and fun to have, it's better to spend your money in a targeted manner. A weaker processor with more RAM is likely going to be an overall better solution.

What's the goal here, besides "lots of TB's in a box" - are you going to be serving video (HD/4K repository) or smaller files? Is this for home or business?

Hi, thanks for your reply! Much appreciated. My brothers and I are building several 'servers', storage, and processing machines. The main purpose of this particular rig will be to temporarily store massive image files (aerial imagery in this case) while it is being processed into another format. The file sizes can be quite large. Part of the reason for overkill is the difficulty in getting PC parts lately so we're working with what we have, but we also intend to take parts from the multiple rigs we're building to repurpose them. So, we aren't after long term storage with this rig, performance is more important because even if we lose all the data we would just reload it again (not that we want that to happen lol). But, thank you for pointing that out because we do want to build a rig later that will be used for long term storage and I see what you're saying that it would be better to go with parts that are built for that purpose.

 

[j05iah]

That depends on how many vdevs you are going to make in your pool of drives. We need to talk about your performance expectations so we can make some informed recommendations.
Depending on what you want to do with this, it might be better to have more drives with a smaller per-drive capacity so you can have more vdevs. Vdevs give you IOPS.

Exactly what hardware are you using?

Please read:

Forum Rules

Welcome to the TrueNAS Community! Updated 11/28/2023 First and foremost I want to thank you for joining the TrueNAS community! My name is JoshDW19 and I'm the Community Manager here at TrueNAS. The TrueNAS Community is a friendly place to discuss open source software, technology, and products...

www.truenas.com

Chris, thank you for your response. We were thinking we would get close to the performance of what a single drive is listed as by the manufacturer, but if that is wishful thinking I think it would still be ok. It's just for temporary storage of large aerial image files. We load those image files via external hard drives or downloaded. Once they are on the NAS the image files will then be processed (or I would say 'combined' into a special format); then sent to the final server from there. I think the files would generally be on the NAS for a week or two before new files come in and overwrite the old stuff. This rig will be temporary until we can get it built the right way, and we will need more storage. I was originally thinking we could just make one big storage pool, but I'm reading the TrueNAS documents and forum to understand the best way to do this. As a side note, this rig won't be 'mission critical', our little brains are just trying to think of it as a giant hard drive haha.

The SATA card we tried is this one, which we mistakenly thought would work:

ASHATA PCIE SATA Card,PCI-E to SATA3.0 Controller Expansion Board 8-Port SATA3.0 Interface Expansion Card

Amazon.com link: https://www.amazon.com/ASHATA-SATA3-0-Controller-Expansion-Interface/dp/B082Q1SPW9

 

[j05iah]

I guess from what I'm reading on the forum (and what's available to buy out there) this looks like it would be the best choice to get up and running with something that is supported and other people have gotten to work? I would need 3 of them: LSI Logic 9201-16i

 

[HoneyBadger]

Hi, thanks for your reply! Much appreciated. My brothers and I are building several 'servers', storage, and processing machines. The main purpose of this particular rig will be to temporarily store massive image files (aerial imagery in this case) while it is being processed into another format. The file sizes can be quite large. Part of the reason for overkill is the difficulty in getting PC parts lately so we're working with what we have, but we also intend to take parts from the multiple rigs we're building to repurpose them. So, we aren't after long term storage with this rig, performance is more important because even if we lose all the data we would just reload it again (not that we want that to happen lol). But, thank you for pointing that out because we do want to build a rig later that will be used for long term storage and I see what you're saying that it would be better to go with parts that are built for that purpose.

As an on-and-off gamer who's currently "on" I certainly feel you on the challenges getting hardware; but getting server hardware is actually much easier than trying to build a Ryzen/RTX setup right now.

How much of the hardware do you already have? My temptation would be to stick with Supermicro X9 or Sandy Bridge/Ivy Bridge generation hardware so that you can buy huge amounts of ECC DDR3 RDIMM for next to nothing, and stuff the system to the gills with memory.

Performance wise, if you will be doing huge loads of data sequentially and then reading it off later in the same manner; a multi-vdev RAIDZ2 is probably the right way to set it up. With 40 drives that lends itself to something like a 5x 8-drive or 4x 10-drive configuration fairly easily.

The SATA card we tried is this one, which we mistakenly thought would work:

ASHATA PCIE SATA Card,PCI-E to SATA3.0 Controller Expansion Board 8-Port SATA3.0 Interface Expansion Card

Amazon.com link: https://www.amazon.com/ASHATA-SATA3-0-Controller-Expansion-Interface/dp/B082Q1SPW9

Definitely the source of your issues. The LSI SAS2008 and SAS2308 based cards are the best supported under FreeBSD (which FreeNAS is based on) - as to whether or not you will need a SAS expander, that will depend on your base and drive backplane. Have you already purchased a case, or is that still up for discussion?

 

[Chris Moore]

If you are looking for a system that works well, these are what we have been buying at work and they do the job:

6049P-E1CR60L | SuperStorage | Products | Super Micro Computer, Inc.

 

[no_connection]

What is your actual use? I mean I get the large image part but that does not say much about the pipeline or workflow you expect data to take.
Do you load it up with a few hundred TB of raw images and then a renderfarm pulls data randomly and stitch it or is it more linear?
Or mixes reads and writes and so on.

When doing weird uses you can do weird solutions. Since you have the raw data somewhere to easy replace files, you could use ZFS to checksum and detect corruption, but use RAID0 so you don't have redundancy to fix it. I have never ever seen a checksum error on my NAS for all the years it has been running so for this type of temporary work it could be fine to know about corruption and manually fix it. It really depends on what you do and how you work with it. Or do a mix.
I don't see 4 vdev with 10 drive Z2 being super fun to rebuild anyway. And net you 576TB
5x8Z2 would be 540TB
R10 would be 360TB

20 Z0(RAID0) would be 360TB + 20 Z10 (RAID10) 180TB for a total of 540TB and you can choose where to store some files. Say the RAID0 would be source files for the batch and RAID10 would be the destination.

Or split up into 4 pools of RAID0 for total of 720TB

Normally you want the NAS to guarantee no errors no matter what, but that kinda needs more Server type with ECC and other safety. And since you already are below that level of hardware you already have to think about error handling in the pipeline when it comes to the processing software you use. And error detection might be acceptable enough rather than error correction at the file level. Just know that you have to make that decision at some point.

I would still take a long look at what HoneyBadger suggested and "do it right".