Help with server build?

[serverboy]

Hi

I have been using Truenas Core quite for some time virtualised in my homelab to present datastores to my vmware hosts which has been working fine for years.

I now wanted to build my own storage NAS. I have been using synology quite for some time now. But though lets bite the bullet and build a truenas server.

This server will be used to run PLEX, Backup repository for VEEAM, VMs

I plan to use a 24 bay case - 12 bays with 20TB Seagate EXOS, this will hold my media for PLEX, veeam backups etc and general files. I was going to populate the other 12 with SSD 870 EVO at some point and then some NVME quad cards. Some of the SATA SSDs could be used as cache for the spinning drives.

Below is my parts list as of today - still gathering bits

CASE - CASE
MB - SuperMicro - X10DRi
CPU - 2x Xeon E5 2667v4
RAM - 8x 2133Mhz ECC 32GB Modules
GPU - Quadro P2200
HBA - Saw this - HBA -LSI Logic Controller Card IT Mode 9305-24i 24-Port SAS 12Gb/s PCI-E 3.0
DRIVES - 12x 20TB Seagate EXOS start with then probably add some Samsung EVO 870
NIC - Mellanox Connectx4 25GBe dual sfp+
PCI - 4 slot NVME Card
PSU - Corsair AX1500i
CPU COOLER - Noctua NH-D9DX i4 3U

I noticed this motherboard has 2x EPS connectors so I got a bigger PSU which supported this and was on sale.

I went with this build as I wanted the PCIe Lanes and the board supports bifurcation which is good.
My knowlege around HBA cards is very limited but I saw this card mentioned in a few forums so thought check it out.
Would the following HBA card work with my setup straight out the box? It does state it is in IT mode which i know that is required to work with Truenas.

Any help is much appreciated.

Thanks

 

[artlessknave]

a

MB - SuperMicro - X10DRi
CPU - 2x Xeon E5 2667v4
RAM - 8x 2133Mhz ECC 32GB Modules

tad aged but should still be solid.

HBA - Saw this - HBA -LSI Logic Controller Card IT Mode 9305-24i 24-Port SAS 12Gb/s PCI-E 3.0

woa there big spender. solid card but pricier than you need. the 92XX cards will be more than enough.
this is a pure HBA. there IS no IR mode.

NIC - Mellanox Connectx4 25GBe dual sfp+

blingy.

PCI - 4 slot NVME Card

many of these are ewaste.

PSU - Corsair AX1500i

i usually wouldnt comment on your size but this is like 3x overkill. certainly not gonna hurt anything though.

 

[serverboy]

woa there big spender. solid card but pricier than you need. the 92XX cards will be more than enough.
this is a pure HBA. there IS no IR mode.

The 24 bay slots will all be populated at some point maybe 12HDD and 12SSD, so would I need the 12Gbps? What is IR mode? Im quite new to these HBA cards and server kit.

many of these are ewaste.

How come? I got a dual syba iocrest one which runs pretty great.

i usually wouldnt comment on your size but this is like 3x overkill. certainly not gonna hurt anything though.

Yeh it was on sale so thought bettet to be safer than sorry. I do have a AX850 in my PC which I could use instead... Just thought with the amount kit of going inside and the CPUs rated at 130w each on intel Ark along with a ton of drives probably best to go big.

 

[artlessknave]

The 24 bay slots will all be populated at some point maybe 12HDD and 12SSD, so would I need the 12Gbps? What is IR mode? Im quite new to these HBA cards and server kit.

not for HDD. a 9211 is overkill on spinners. spinners barely do sas1/sataI most of the time. if you are adding SSD's though that's different, though SSDs still max out at sataII/SAS2 speeds (600MB/s iirc) unless you get SAS3 SSD's which are typically costly. if it's within the budget, its completely fine. I have one. when I first was getting into this I had unknowlingly gotten some fake LSI cards and bought it cuz I was tired of having issues (one died, the other would show disks but none we accessible). it's completely wasted on my spinning disks.
RAID=IR mode
HBA=IT mode

How come? I got a dual syba iocrest one which runs pretty great.

many of these, as well as silly things like sata port replicators (shudders) are typically designed for windows use, with drives only being used one a time. ZFS very often pushes hardware to it's full capabilities and usually finds the weakest points. the same problem with using USB disks. the cheap SATA cards and USB access disks round robin, but ZFS expects the disks to be available any time it needs to access them, which is usually constantly.

Yeh it was on sale so thought bettet to be safer than sorry. I do have a AX850 in my PC which I could use instead... Just thought with the amount kit of going inside and the CPUs rated at 130w each on intel Ark along with a ton of drives probably best to go big.

nothing wrong with it; some cant reach their full spec unless they run on 240v though

 

[serverboy]

not for HDD. a 9211 is overkill on spinners. spinners barely do sas1/sataI most of the time. if you are adding SSD's though that's different, though SSDs still max out at sataII/SAS2 speeds (600MB/s iirc) unless you get SAS3 SSD's which are typically costly. if it's within the budget, its completely fine. I have one. when I first was getting into this I had unknowlingly gotten some fake LSI cards and bought it cuz I was tired of having issues (one died, the other would show disks but none we accessible). it's completely wasted on my spinning disks.
RAID=IR mode
HBA=IT mode

Would I need IR mode. I thought with Truenas you need to stay away from HW RAID cards and only use cards which present disks to the OS so the Truenas system can perform Software RAID.

Which card would you recommended for what I want to achieve as mentiond above in the specification? I am from the UK so I will most likely be looking on UK Amazon or Ebay.

nothing wrong with it; some cant reach their full spec unless they run on 240v though

Ok that is good to know. I will just use this 1500 Watt then.

Thank you for the advice so far.

 

[Davvo]

like sata port replicators

Multipliers should be the term you want to avoid.

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

Would I need IR mode.

Big NO.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[serverboy]

Hi Davvo

Big NO.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

What would you recommend for a HBA

 

[Ericloewe]

sas1/sataI most of the time. if you are adding SSD's though that's different, though SSDs still max out at sataII/SAS2 speed

SAS1 does 3 Gb/s, SAS2 does 6Gb/s, SAS3 12 Gb/s. If you're crazy enough, SAS4 does 24 Gb/s.
First-generation SATA does 1.5 Gb/s, newer stuff is properly know as SATA 3 Gb/s and SATA 6 Gb/s.

As a sidenote to the sidenote, SAS SSDs are pretty terrible value and to be avoided as much as possible, in a world where SATA SSDs occupy the low-end and NVMe SSDs are actually trying to be price competitive.

 

[serverboy]

SAS1 does 3 Gb/s, SAS2 does 6Gb/s, SAS3 12 Gb/s. If you're crazy enough, SAS4 does 24 Gb/s.
First-generation SATA does 1.5 Gb/s, newer stuff is properly know as SATA 3 Gb/s and SATA 6 Gb/s.

As a sidenote to the sidenote, SAS SSDs are pretty terrible value and to be avoided as much as possible, in a world where SATA SSDs occupy the low-end and NVMe SSDs are actually trying to be price competitive.

I will probably be purchasing spinning SATA drives for the first 12 bays.

Although I believe the SAS versions of the Seagate EXOS are cheaper than the SATA. So could get those. For my SSDs I will be using SATA 870 EVOs

But the whole HBA thing is new to me so not sure what to get. All I know it has to be in IT Mode right?

 

[Davvo]

What would you recommend for a HBA

You should check the bandwith, but I agree with @artlessknave that a 92XX (-24i or less if you want to use your motherboard's ports) series card could be worth the savings.

Really depends on your budget and needs... Also, are you using a backplane? I don't have much experience regarding servers, but quite a lot seem to use one. The presence or absence of one might change your requirements.

 

[serverboy]

You should check the bandwith, but I agree with @artlessknave that a 92XX (-24i or less if you want to use your motherboard's ports) series card could be worth the savings.

Really depends on your budget and needs... Also, are you using a backplane? I don't have much experience regarding servers, but quite a lot seem to use one. The presence or absence of one might change your requirements.

Yes the case i am getting has a backplane.
"Six internal SFF-8643 HD Mini SAS connectors support up to twenty-four 6Gb/s 3.5" or 2.5" SATA (II or III) or SAS 6 or 12Gb/s hard drives"

 

[Davvo]

In order not to bottleneck your backplane the HBA you posted on the first post of this thread is appropriate.

 

[serverboy]

In order not to bottleneck your backplane the HBA you posted on the first post of this thread is appropriate.

Yeh i was looking for the 92xx 24i version but couldn't find one on ebay. To be honest not even sure what model number i should be looking etc.

I have seen some people get a 8i version then use a sas expander card.

But this take up a PCIE slot which I want to keep for other stuff.

So as per the comment from @artlessknave IR mode is irrelevant for my setup right. As Truenas focuses on software RAID right?
So the proposed card is fine to go with?

 

[artlessknave]

the HBA is perfectly fine. I missed the part about adding SSDs. I wouldn't have recommended any 92XX cards if I hadn't missed that.

it will cheerfully run a hundred or so SSDs at excellent speeds.

 

[Ericloewe]

Yeh i was looking for the 92xx 24i version but couldn't find one on ebay. To be honest not even sure what model number i should be looking etc.

There is no such thing. There is the SAS 9201, which has 16 channels, but anything larger is just multiple controllers behind a PCIe switch. For SAS3, there are a few larger models, depending on the exact generation.

 

[Davvo]

So as per the comment from @artlessknave IR mode is irrelevant for my setup right.

Yup he was just telling you that it has to be in IT mode.

 

[serverboy]

Ok then that is good to know.

@Davvo I believe the card comes flashed in IT MODE already from looking at the advertisement so should just work. I have ordered Mellanox Cards off this seller before and they were fine. So I have some faith in the seller.

I will pull the trigger on the card. Thanks for the guidance on this. Much appreciated.

 

[serverboy]

Hi Guys, I had another question in regards to the SATA drives.

What is the recommended size to go with. I always see people using smaller sizes like 4 or 6TB.

Is it fine to use 20TB Seagate EXOS with Truenas or should I aim a little lower round the 14TB mark?

 

[serverboy]

the HBA is perfectly fine. I missed the part about adding SSDs. I wouldn't have recommended any 92XX cards if I hadn't missed that.

it will cheerfully run a hundred or so SSDs at excellent speeds.

That is great. I plan to use Samsung 870 EVO SSD at 2TB each. So this should be fine right.

If I ever wanted to expand I could just buy another card with external ports right? I will have spare x8 PCIe slots.

 

[artlessknave]

I always see people using smaller sizes like 4 or 6TB.

these are older disks genreally and thus have been around for debugging far longer, giving them a higher chance of being stable.
the larger the drive the newer their build generally is, and thus, the less testing and debugging, and thus the higher the potential risk of failure.
plus, more, smaller drives can be setup with more rendundancy, faster resilvers, etc. a big drive ultimately means bigger impact if it fails.

If I ever wanted to expand I could just buy another card with external ports right? I will have spare x8 PCIe slots.

off of that monstrous HBA you can just use expanders. up to 256 disks. expanders only need power and SAS connections to a controller. essentially a network for disks.
something like one of these, for example

Lenovo Adaptec AEC-82885T 36-port 12Gbps SAS/SATA Expansion Card RAID 8643​

Lenovo Adaptec AEC-82885T 36-port 12Gbps SAS/SATA Expansion Card RAID 8643 | eBay

Find many great new & used options and get the best deals for Lenovo Adaptec AEC-82885T 36-port 12Gbps SAS/SATA Expansion Card RAID 8643 at the best online prices at eBay! Free shipping for many products!

www.ebay.ca