Compatibility of a 9400-16i HBA with desktop/NAS drives

[Le.Patator]

Dear all,

struggling with my current NAS (a small HP NL40 customized to host 6 drives running FreeNAS - I need to update... ;-) ), I would like to design my new home server. I noticed that a "normal" high-capacity server (16 drives for example) can be built using the XL series of Fractal Design cases (Meshify 2 XL for example).

I have the idea of buying an HBA such as the Broadcom 9400-16i to manage most of the drives (my first idea was to buy older HBAs from Art Of Server, but being in Europe, the overhead cost may be high).

Simple question: is the 9400-16i compatible with desktop/NAS drives? (WD Red or Green to begin the Vdevs) - I can't find any relevant information on Internet, except the compatbility with entreprise drives (expensive and I already have the Red drives).

Any advice for another HBA is welcome if other proposals are more relevant! :)

Thanks in advance.

 

[sretalla]

is the 9400-16i compatible with desktop/NAS drives? (WD Red or Green to begin the Vdevs)

Compatible, probably (almost certainly), but should you use them together for TrueNAS... absolutely not.

WD Red (desktop version) could be SMR, so entirely inappropriate for ZFS/TrueNAS (https://www.truenas.com/community/resources/list-of-known-smr-drives.141/). If those are Red Plus (NAS drives), fine and certainly compatible.

WD Green are not really suitable for NAS use either and likely to fail under the kind of workload a NAS will put on them.

 

[Ericloewe]

Any advice for another HBA is welcome if other proposals are more relevant! :)

Nope, you're on the right track. SAS 9300s are also fine, as should be SAS 9500s.

 

[Le.Patator]

@sretalla:
Thanks a lot for your answer. No, my target WD Red drives are 8TB CMR drives (NAS), fortunately. I also have 8TB NAS N300 Toshiba and Seagate Ironwolf available. And also some 6TB EFRX (so CMR) if I am right.

I followed this SMR/CMR thing a few years ago and I was quite happy to have these 8TB drives...

The (small) Green drives are just to start building the pools, then, they will be replaced by the Red drives once they are empty (waiting for the RAIDz expansion feature? ).

@Ericloewe
Thanks also! I considered 9500 but they are quite expansive for a home server (if 9400 is sufficient! - 400€ for a brand-new one). I also considered 9300 but they are out of stock of several e-retailers (too old?). Second-hand should be fine, but it is quite difficult to find reliable vendors in Europe. :)

Next topic to think about: ECC vs. non ECC => CPU and motherboard. :)

 

[sretalla]

6TB EFRX (so CMR) if I am right.

Yes, CMR.

The (small) Green drives are just to start building the pools, then, they will be replaced by the Red drives once they are empty

Short-term use would probably not be much of a risk (but a risk nonetheless... so take note).

waiting for the RAIDz expansion feature?

You can replace the disks in-place in the pool with the larger Reds, so no need.

Don't hold your breath for that feature... late 2024 was the last estimate I saw.

 

[Etorix]

Notwithstanding that the 9400 series does not yet have enough field testing for the taste of our most conservative experts, it is ridiculous overkill for hard drives. Go for a 9200 or 9300, the latter already being somewhat overkill for spinning platters.
It should be possible to find second-hand 9200/9300 HBAs within the EU.

 

[Le.Patator]

Short-term use would probably not be much of a risk (but a risk nonetheless... so take note).

You can replace the disks in-place in the pool with the larger Reds, so no need.

Don't hold your breath for that feature... late 2024 was the last estimate I saw.

I have backups so no problem if one Green drive fail in the process. I plan RAIDZ2 at minimum, perhaps RAIDZ3 for the large number of drives. I may also divide the problem: start with a 8-drive RAIDZ2 vdev, then add a second 8-drive RAIDZ2 vdev in the pool (or something like this) instead of going directly to a 16 RAIDZ3 vdev... I need to think about that "upgrading process".
If I need more space in the future, I can also buy another case and link both cases with external adapters.

Notwithstanding that the 9400 series does not yet have enough field testing for the taste of our most conservative experts, it is ridiculous overkill for hard drives. Go for a 9200 or 9300, the latter already being somewhat overkill for spinning platters.
It should be possible to find second-hand 9200/9300 HBAs within the EU.

Thanks. This was my first idea, trying to compare new/second-hand prices, trying to find reliable vendors, etc. 9300 should be fine, but only second-hand (no problem from a reliable vendor... to find... ).

 

[Etorix]

16-wide is too wide for any geometry. 2 * 8-wide is much better, and does not require 16 drives upfront.

 

[Le.Patator]

Thanks. Your remark is in line with what I read several years ago when I discovered ZFS (OpenIndiana, then OmniOS + Napp-IT).

So another idea could be to stick with "old" but rock-solid HBAs (for HDDs) such as buying two of these beauties (~90/100€ from the US instead of >400€ for a more modern new 9400-16i) - depending on the SATA ports on the motherboard?

HP H220 (=LSI 9207-8i) 6Gbps SAS PCIe 3.0 HBA P20 IT Mode ZFS FreeNAS unRAID | eBay

I have fully tested them using all 8 SAS lanes. I've used ZFS for many years. I'm familiar with ZFS on Linux, FreeNAS, and many of the technologies commonly used by other enthusiasts. They have been flashed with Avago / LSI 9207-8i IT (Initiator Target) firmware version P20 (specifically...

www.ebay.com

Then setup a first 8-drive vdev (RAIDZ2) pool, then extending it with a second RAIDZ-2 vdev...

One potential issue if I need two HBAs would be the case where I need to expand the first server with a third external HBA. I need to consider an ATX motherboard. Another issue is that I planned to cool the HBA using a "PCIe fan with bracket". I need to consider that.

For the motherboard and CPU, the not expensive way could be desktop CPUs and motherboards, but without ECC... :-/
Otherwise, go for AMD, with potential compatibility issues with TrueNAS core (as stated in the HW webpage).
Going to workstation motherboard and desktop CPU such as the 13500 could be interesting, with the price of the motherboard, and not-so-cheap DDR5 ECC.
Or going to full server processor (Atom/Xeon) and according MB (Supermicro?)...

A lot of thinking... Reminds me years ago... :-D

 

[Ericloewe]

Sure, SAS2 is plenty good for spinning rust.

One potential issue if I need two HBAs

Or you can use a SAS expander instead.

 

[Etorix]

In an XL case accommodating 16 drives, there's plenty of space for an (E-)ATX server motherboard with 8-12 SATA ports. A single -8i HBA will do the rest when adding the second vdev.
Issues will be cooling the drives, and possibly drive noise.

What's the purpose of the NAS? Plain bulk storage, or more sophisticated tasks?

 

[sretalla]

I may also divide the problem: start with a 8-drive RAIDZ2 vdev, then add a second 8-drive RAIDZ2 vdev

If you start with a RAIDZ2 VDEV that contains disks that aren't great for ZFS, you can't remove that VDEV from the pool... ever... since the pool contains RAIDZ VDEVs.

That means that if you add a second VDEV later, you still have the not-great disks in your pool and your only way out would be 1-by-1 replacement of the disks in that VDEV.

 

[Le.Patator]

Sure, SAS2 is plenty good for spinning rust.

Or you can use a SAS expander instead.

Thanks for confirming. A lot of possibilities. :)

In an XL case accommodating 16 drives, there's plenty of space for an (E-)ATX server motherboard with 8-12 SATA ports. A single -8i HBA will do the rest when adding the second vdev.
Issues will be cooling the drives, and possibly drive noise.

What's the purpose of the NAS? Plain bulk storage, or more sophisticated tasks?

My current issue is finding the correct CPU and motherboard. If I am right, I will need to flash the possible onboard RAID controller with an IT firmware. I do not want to put "a lot" of money in the CPU and motherboard ("a lot" is like 1500€ for both for example), and a real issue will be availability of the parts (not too new to have good SW support, not too old to have good perf/watt and functionalities - while being able to buy "locally").
I noticed potentially Xeon E-23xx and so on, or E3-12xxv6 processors as potential candidates (not too old). I would like to have some server board such as Supermicro ones (I really like IPMI!). I noticed some integrated CPU/MB for 1U servers (Xeon-D for example), but I currently use such an MB at work with cooling issues in a standard case (extreme conditions, though).
For the cooling, I plan to put as many fans as I can in the front (14cms). In a Fractal Meshify 2 XL, this will do the trick, at least for the "top" drives. I currently have 6 HDDs in my "workstation" and no noise. It may not be the case with 16 or 18... :)
The main purpose is clearly bulk storage currently (I need a big unified space), but I would like to a little room for one or two VMs if needed.

If you start with a RAIDZ2 VDEV that contains disks that aren't great for ZFS, you can't remove that VDEV from the pool... ever... since the pool contains RAIDZ VDEVs.

That means that if you add a second VDEV later, you still have the not-great disks in your pool and your only way out would be 1-by-1 replacement of the disks in that VDEV.

Thanks! My initial plan was indeed to replace the "small" drives one by one with the Red drives. Time consuming, potentially rude for the Green drives.

 

[Etorix]

My current issue is finding the correct CPU and motherboard. If I am right, I will need to flash the possible onboard RAID controller with an IT firmware. I do not want to put "a lot" of money in the CPU and motherboard ("a lot" is like 1500€ for both for example), and a real issue will be availability of the parts (not too new to have good SW support, not too old to have good perf/watt and functionalities - while being able to buy "locally").
I noticed potentially Xeon E-23xx and so on, or E3-12xxv6 processors as potential candidates (not too old). I would like to have some server board such as Supermicro ones (I really like IPMI!). I noticed some integrated CPU/MB for 1U servers (Xeon-D for example), but I currently use such an MB at work with cooling issues in a standard case (extreme conditions, though).

Xeon-D1500 do great in standard cases with a small fan on the heatsink (Noctua NF-A6x25, held by two strips of gaffer tape on the sides).
If you go as old as E3v5/6, rather look for Xeon E5v3/4—to support more RAM and, most importantly, as RDIMM, which comes nicely cheap second-hand.
Mostly storage "with room for a VM or two" can do with few cores but high clocks whenever possible. Xeon D-1500 could do; Xeon E5 for more computing power, or Xeon W-2000 if you can find a X11SRL-F. Atom C3000 (A2SDi) if you buy new.
All of these should be possible for less than 1500 E, including RAM. Issues are availability, for second-hand refurbished boards, and how much is "not a lot".

 

[Le.Patator]

Thanks for your answer. For the Xeon-D 1700 I use at work, I built a "custom" air shroud with a 12cm-fan: 20°C lower temperature.

I will need to dig a bit better or ask to the "french" part of the forum (I live in France): finding a Xeon/Xeon-D motherboard or even a Xeon processor is really hard for non-professional customers... I will perhaps have better luck with "workstation" parts instead of "server" parts. Thanks again!

 

[Le.Patator]

Maybe an ASUS P12R-M motherboard with a Xeon E may do the trick (available on "french" Amazon)...

 

[Etorix]

Maybe you're not looking at the right places?
Here are X10SDV second-hand boards from France.
If you really can't find a Supermicro retailer in France, or they all are B2B-only, have a look at Germany. Or go straight to Anafra in the Czech Republic: Highly commendable shop.

 

[Ericloewe]

X10SDVs have not had many great deals on eBay recently, that I've seen. They're fairly seasonal, so check back regularly.

 

[Le.Patator]

Maybe you're not looking at the right places?
Here are X10SDV second-hand boards from France.
If you really can't find a Supermicro retailer in France, or they all are B2B-only, have a look at Germany. Or go straight to Anafra in the Czech Republic: Highly commendable shop.

I think it is the case. After my last message, I spent some time to search deeper and it really seems that all "suggested" retailers in France only do B2B. Or they "consider you as professionals" as wrote in the ASInfo (for example) description (they perhaps may sell to you but with some restrictions (?) and their stock is at low level). There is one website which is selling to individuals but with some bad comments, so... (dropshipping-like or something like this...).
The other known retailers only target gamers and so on. Germany could be a good place to search, but I only ordered from Caseking, which does not distribute Xeon or Supermicro.
Finally, I had some links to the Anafra website on Google, and it seemed to be reliable as you wrote (and thanks again for confirming this!), with stock and not-so-high prices and they seem to sell to individuals. After some search, they also offer free shipping to Europe for orders over 400€. I guess I found my retailer. :)

Now, I need to chose my weapon.
Xeon-D1500 could be nice indeed. I think it would be nice to have at least two PCIe: one for an HBA, the other one in case I would like to extend the storage with an "external" HBA (no experience on that). Anafra seems to have stock on Mini-ITX with only one PCIe or low stock on some possible interesting models such as this one (?):

Supermicro MBD-X10SDV-4C-7TP4F-B

Intel® Xeon® processor D-1518, Single socket FCBGA 1667; 4-Core, 8 Threads, 35W Up to 128GB ECC RDIMM DDR4 2133MHz or 64GB ECC/non-ECC UDIMM in 4 sockets 2 PCI-E 3.0 x8, Mini PCI-E (mSATA support), M.2 PCI-E 3.0 x4 (SATA support), M Key 2242/2280/22110 2 10G SFP+ and 2 GbE LAN 4 SATA3 (6Gbps)...

smicro.eu

The Xeon E3v5/6 option was really based on "french" stocks or retailers, which is no longer an issue. With a Xeon W-2000 (4 cores/8 threads) and a X11SRL-F (available at Anafra: https://smicro.eu/search?search=X11SRL-F - external stock), this would be interesting for extension (lot of PCIe).
Concerning the Atom C3000, motherboards seems to be mainly Mini-ITX, wich is great for a small NAS but maybe a bit small for 16+ drives.

A least, I have some material to think about with Anafra (and I was in Prague a few months ago... :) ).

X10SDVs have not had many great deals on eBay recently, that I've seen. They're fairly seasonal, so check back regularly.

Thanks!

 

[Etorix]

The other known retailers only target gamers and so on. Germany could be a good place to search, but I only ordered from Caseking, which does not distribute Xeon or Supermicro.

Caseking is (surprisingly) good for cases. For what goes into the case, look into Jakob, Barax and the like.

Now, I need to chose my weapon.
Xeon-D1500 could be nice indeed. I think it would be nice to have at least two PCIe: one for an HBA, the other one in case I would like to extend the storage with an "external" HBA (no experience on that). Anafra seems to have stock on Mini-ITX with only one PCIe or low stock on some possible interesting models such as this one (?):

Supermicro MBD-X10SDV-4C-7TP4F-B

Intel® Xeon® processor D-1518, Single socket FCBGA 1667; 4-Core, 8 Threads, 35W Up to 128GB ECC RDIMM DDR4 2133MHz or 64GB ECC/non-ECC UDIMM in 4 sockets 2 PCI-E 3.0 x8, Mini PCI-E (mSATA support), M.2 PCI-E 3.0 x4 (SATA support), M Key 2242/2280/22110 2 10G SFP+ and 2 GbE LAN 4 SATA3 (6Gbps)...

smicro.eu

A -7TPnF board indeed comes with everything on-board. But it also approaches the price point at which you may look into the lower end of X12SDV, with on-board 25 GbE.

The Xeon E3v5/6 option was really based on "french" stocks or retailers, which is no longer an issue. With a Xeon W-2000 (4 cores/8 threads) and a X11SRL-F (available at Anafra: https://smicro.eu/search?search=X11SRL-F - external stock), this would be interesting for extension (lot of PCIe).
Concerning the Atom C3000, motherboards seems to be mainly Mini-ITX, wich is great for a small NAS but maybe a bit small for 16+ drives.

An A2SDi-H-TF supports 12 drives out of the box; with a 9300-8i in the (open) x4 slot, it could do 20 drives, that is all 16 in your case with some spare ports for drive replacements or a SATA SSD pool for VMs. A lot of punch for its size!

Now it's a matter of use case. For pure NAS duty, an embedded board with 2-4 threads is enough. Any VM would require cores on top of that—cannot oversubscribe.