2nd HBA, SAS expander or cheap SATA PCIe card

[jyavenard]

My current TrueNAS server is made a of a 12x4TB SATA disks zpool.
The motherboard has a 9300-8i (well, broadcom 3008 it's called these days).
So I use the HBA (in IT mode) for driving 8 disks, and then 4 disks are using the motherboard's 4 SATAs.

I'm looking at virtualising this TrueNAS box (so it runs on top of proxmox), so I would need to do a passthrough of the SATA controller, but this would limit future upgradability.

I'm wondering what is the best or most economical way to go about freeing the 4s system SATA:
Add a new HBA (like another 9300-8i) or to use a SAS expander?
Alternatively, could add a SATA PCIe controller, they are very cheap.

Any recommendations?
new 9300-8i HBA or expander are similar cost wise, but the expander would give me even more disk ports should I need it in the future.

TIA

 

[sretalla]

Add a new HBA (like another 9300-8i) or to use a SAS expander?

Yes, always the best answer.

Alternatively, could add a SATA PCIe controller, they are very cheap.

Not great and you will possibly run into this:

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

 

[jyavenard]

Yes, always the best answer.

I'm guessing here you're referring to new HBA, a bit unclear :)

Not great and you will possibly run into this:

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

Does SAS expander fall into this category? not really a sata port multiplier

 

[sretalla]

Does SAS expander fall into this category? not really a sata port multiplier

It's a SAS "switch", not a port multiplier.

I'm guessing here you're referring to new HBA, a bit unclear :)

Both options are fine (Expander for your existing HBA or new HBA), just comparing them to SATA controller cards... not always fine.

 

[jyavenard]

Turned out, I can just replace the backplane of my supermicro chassis for a SAS one that includes the expander already (the chassis ony had a SAS1 backplane).

Now the SAS port on my HBA is SAS3, but all my SATA drives are 6Gbit/s SATA only.

Does having a SAS3 backplane in place of SAS2 one provides any advantages?

That is, will the 12GBit/s uplink from the backplane to the HBA gives any speed increase for the pack of 6GBit/s disks?
Or does only having 6GBit/s SATA disks means that the entire chain between the disk and the HBA gets negotiated at 6GBit/s ?

SAS2 backplane are pretty cheap over the SAS3 one (though we're talking US$50 difference)

 

[Davvo]

If you are using hard drives it matters little, you won't be even near to saturate either on the drives to backplane side.

 

[jyavenard]

If you are using hard drives it matters little, you won't be even near to saturate either on the drives to backplane side.

even with 12 drives?
I guess it's 4 lanes x 6Gbit/s = 2.4GB/s

The supermicro SAS2-826EL1 has a single SAS uplink port, while the SAS3-826EL1 has two (not referring to failover ports). Which from my reading is only used for additional bandwidth between the backplane and the HBA.
So the SAS3 backplane does seem to have a bit more flexibility.

 

[sretalla]

But if the difference is $50, maybe the option to later install SSDs and get better speed is an attractive thing.

 

[jyavenard]

But if the difference is $50, maybe the option to later install SSDs and get better speed is an attractive thing.

Was US$119 for SAS3 and US$78.64 for SAS2; that's comparing the 2 cheapest on ebay :)
And the SAS3 comes with two sff-8643 cables.

I ordered the SAS3 for the exact reasons you mentioned.

 

[jyavenard]

I finally got the SAS3 backplane. A BPN-SAS3-826EL1. Swap in my chassis was super-simple.
Connected with two miniSAS cables to the 9300-8i HDA.

Code:

ses0 at mpr0 bus 0 scbus2 target 20 lun 0
ses0: <SMC SC826-P 100b> Fixed Enclosure Services SPC-3 SCSI device
ses0: 1200.000MB/s transfers
ses0: Command Queueing enabled
ses0: SES Device
ses0: da0,pass0 in 'Slot00', SAS Slot: 1 phys at slot 0
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048020a3ba7f addr 5003048020a3ba40
ses0: da1,pass1 in 'Slot01', SAS Slot: 1 phys at slot 1
da1 at mpr0 bus 0 scbus2 target 9 lun 0
da1: <ATA WDC WD40EFZX-68A 0A81> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number WD-WX32DC0HYPSH
da1: 1200.000MB/s transfers
da1: Command Queueing enabled
da1: 3815447MB (7814037168 512 byte sectors)
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048020a3ba7f addr 5003048020a3ba41
da0 at mpr0 bus 0 scbus2 target 8 lun 0
da0: <ATA WDC WD40EFRX-68W 0A80> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number WD-WCC4E0281971
da0: 1200.000MB/s transfers
da0: Command Queueing enabled
da0: 3815447MB (7814037168 512 byte sectors)
da0: quirks=0x8<4K>

 

[jyavenard]

It seems that the backplace is using 25W.. 175W draw now, was 150W before.
That's a bummer and surprisingly high.

 

[sretalla]

double the frequency for much less than double the "price" though...

 

[Davvo]

I mean, you are running 12 drives.

 

[jyavenard]

I mean, you are running 12 drives.

that only accounts for 70W.
This is before/after replacing the backplane.

 

[jyavenard]

double the frequency for much less than double the "price" though...

As I understand how this SAS3 backplane works, it uses both uplink for the 12 drives. So there's 24Gbit/s available bandwidth for the 12 disks. Unlike the SAS2 version which can only use a single SAS link.

 

[sretalla]

Either shaking the same number of electrons around twice as fast or double the number at the same speed... either way, it's the price of the faster tech that you're doing more with the power to get faster speed.

We didn't discuss that in the benefits (and maybe should have been downsides) discussion before you pulled the trigger. If low(est possible) power was a requirement, I guess we would have landed with SAS2.

 

[jyavenard]

If low(est possible) power was a requirement, I guess we would have landed with SAS2.

That's a good point.
More disappointing to me is that since merging 3x 2013 machines into 1, (upgrading the TrueNAS machine with an AMD Epyc 7302 (16 cores) and 64GB of RAM), and virtualising the two others with bhyve, the power went up by 50W :)
My old TrueNAS was using a Celeron G830 with 32GB of ECC RAM
One PC was using a Xeon E3-1220v3 with 16GB of ECC RAM
One was a celeron g530 with 16GB of ECC RAM.

either way, it's the price of the faster tech that you're doing more with the power to get faster speed.

but is it? power isn't a linear coefficient especially 10 years apart. Processors are an order of magnitude faster yet use up to 1/10th of the power.