Will it TrueNAS (Scale)? Upgraded Huawei RH2288Hv.3 - pondering how to get things working

[SecCon]

I was about to post in the Will it FreeNAS sub forum but noticed there are no recent posts there so I figure best to post here.

Recently I got myself a Huawei server and have since upgraded it. The current hardware specs are based on https://support.huawei.com/enterprise/en/intelligent-servers/RH2288H V3-pid-9901881 but with 12 SATA LFF drives in the front. 2 SFF SAS SSD in the back for OS.

• Motherboard is the BC11HGSA0 based on Intel C610 /X99
• CPU are 2 Intel Xeon E5-2683 v4
• RAM is 64GB ECC Kingston Server Memory: KSM26RD8/16HDI x 4. Ample slots to upgrade.
• HDD RAID BC61ESMQ LSI SAS 3108 on a Mezzanine card.
• NIC is Huawei SM231 based upon Intel 82599
• HDD's are currently split up in a 4 x 6TB 16TB RAID and additional drives in another 2 RAIDS for Misc. SMB share in Windows environment.
  • RAID's will be disbanded and replaced with ZFS Storage Pools
  • SMB share with some kind of backup (rsync?) to a NAS is a must but should not be any issues, aside from me not having done that before. Expect questions.

Will it TrueNAS, Core ?

 

[sretalla]

Will it TrueNAS, Core ?

Generally looks like a reasonable fit.

Intel NIC, LSI HBA, ECC RAM, Xeon CPU... all good.

 

[danb35]

All the specs sound fine with the possible exception of the disk controller--I don't think that one can act as a true HBA.

 

[sretalla]

the disk controller--I don't think that one can act as a true HBA.

Good point, I was reading 3008, not 3108. Looks like the MegaRAID version... no bueno.

Should be possible to add a proper one via PCIe though

 

[SecCon]

Thanks guys...

Indeed it is the Megaraid version. I always seem to stumble in to those.

HBA... I forgot what that was, something about the direct IO access or... ?

Suggestions? Due to the cabling it might be easier with another Mezzanine card.

 

[sretalla]

HBA... I forgot what that was, something about the direct IO access or... ?

This one covers it: https://www.truenas.com/community/t...s-and-why-cant-i-use-a-raid-controller.81931/

Suggestions? Due to the cabling it might be easier with another Mezzanine card.

It seems from the video on replacing the RAID card that the backplane allows for connection via Mini-SAS connectors... it should just be a case of connecting the 2 Mini-SAS connectors from any HBA with those same connectors instead of the RAID card's.

Although I note that the position of the onboard RAID card may be quite different from the location of the PCIe slots, so the included cable may be far too short to reach it, hence requiring some annoying replacement/re-routing of a new pair of cables.

 

[c77dk]

The 3108 has a JBOD mode, which _seems_ to do things right (but haven't been tested as much as proper HBAs, so there might be gremlins hiding). When the possibility for a proper HBA exists, then that's the right way to go (to get the more "battle proven" stability).

 

[SecCon]

Although I note that the position of the onboard RAID card may be quite different from the location of the PCIe slots, so the included cable may be far too short to reach it, hence requiring some annoying replacement/re-routing of a new pair of cables.

Yeah, I think that may be the case. The wiring from the backplane in to that is complex, at least to me. Having to replace short cables for long cables might become an adventure in itself.

@c77dk I am ok with getting another proper card for a reasonable amount of cash instead of experimenting.

 

[SecCon]

Ok, I checked. Cable length seems to be ok for a PCIE card in the rear, as opposed to a Mezzanine card in the midst.

So I would be looking for a dual port HBA controller card than can handle up to 14 drives. I am assuming the descriptions saying 8 channels per port would allow for a total of 16 drives with two ports?

Like this:

That card is a Broadcom SAS 3008, this link https://www.amazon.com/Internal-Controller-Broadcoms-compatible-9300-8I/dp/B07VV91L61

Can you suggest any alternatives that may be a bit cheaper...
Maybe some like this: https://www.ebay.com/itm/1857440902...KliDglnwcY8kl/yhOZ9wkcQXY=|tkp:Bk9SR9islMC8YQ

​

 

[sretalla]

For sure you can find many IBM or other third party cards carrying that LSI chip at cheaper prices, particularly 2nd hand online. (usually around €/$60)

The way those can sometimes work is 4 drives per connector (particularly when using breakout to SATA cables), but that isn't a limit of the connector or the chip (which can handle hundreds of drives).

It's up to the backplane and how it's set up.

It could be that it will allow only the 4 drives per cable (I guess in that case, getting to the 12 disks using 8 from the RAID card and 4 more from the onboard SATA (also using Mini SAS from the picture).

Without some experience with that backplane, it's hard to say how it's hooked up... maybe those 2 connections will cover all the disks.

 

[SecCon]

@sretalla as it is hooked up now with the 3108, all 12 drives are connected via those two cables so that is all in place.

The internal cabling instructions are extensive, but perhaps normal for this type of machine

I made a PDF from the latest official docs that cover this, so have a peek if you like :)

IBM HBA Card... would it be too much to ask for an example? A link?
Something like this? https://www.ebay.com/itm/1323332208...SqAa+LZXbHTmYCf85s7D8InA==|tkp:Bk9SR4yi-P68YQ

 

[SecCon]

The 3108 has a JBOD mode, which _seems_ to do things right (but haven't been tested as much as proper HBAs, so there might be gremlins hiding). When the possibility for a proper HBA exists, then that's the right way to go (to get the more "battle proven" stability).

What if i just did not configure any RAID? Just leave it unconfigured? How would I see if that works or not in TrueNAS?

 

[sretalla]

What if i just did not configure any RAID? Just leave it unconfigured? How would I see if that works or not in TrueNAS?

Back to the point in this resource: https://www.truenas.com/community/t...s-and-why-cant-i-use-a-raid-controller.81931/

Not the same and puts your data at risk.

I can't tell you what risks you're prepared to take with it, so it's up to you, but I wouldn't do that with data I was interested in keeping.

IBM HBA Card... would it be too much to ask for an example? A link?

Something like this:

Supermicro 9300-8I 12Gb/s LSI 9300-8i ZFS PC TRUENAS HBA Card w/ 2 Cables | eBay

ETA 3 to 8 busines. 2x Cables SFF-8643 to 4 SATA 1M.

www.ebay.com

or

LSI SAS3008 9302-8I IT-Mode HBA JBOD PCI-E 3.0 SATA / SAS 8-Port SAS3 12Gb/s | eBay

ETA 3 to 8 busines. We keep our promises that we will provide you with the best solution within 24 hours.

www.ebay.com

will probably be easier for you as it already has the Mini SAS ports... going for a cheaper PCIe v2 card can save you around $50, but will mean you need different cables to go from the SFF ports to Mini SAS.

I don't/can't endorse either of the vendors specifically, the above examples are provided for information regarding the type of thing you're looking for only.

 

[SecCon]

@sretalla thanks. That helps a lot and now I know more of what I am looking for. Learning by example... :)

 

[Davvo]

I'm not really sure about the NIC being Intel though.

 

[SecCon]

Got this via an STH ad:

IT flashed and ready to replace the 3108...

 

[SecCon]

The 3108 has a JBOD mode, which _seems_ to do things right (but haven't been tested as much as proper HBAs, so there might be gremlins hiding). When the possibility for a proper HBA exists, then that's the right way to go (to get the more "battle proven" stability).

Reading up a bit after finding the manual for this, and other, cards.

It says:

3.13 Hard Drive Pass-Through
Also known as transparent data transmission, the hard drive pass-through (JBOD)
data transmission method is one in which transmission devices transmit data
without processing the data.

7.2.4 Transparent Transmission
Also known as hard drive direct connection, this data transmission method is one
in which data is not processed by transmission devices and only transmission
quality is guaranteed.
The controller card enables user commands to be directly transmitted to
connected hard drives, facilitating hard drive access and control by upper-layer
services or management software. For example, when installing an OS on a server,
configure a hard drive mounted to the controller card as the installation drive, as
opposed to just a configured virtual drive.

Now, I have the card on its way so this is probably just moot for me right now, but others may find it useful.

 

[Davvo]

Quoting the HBA resource:

5) A RAID controller that supports "JBOD" or "HBA mode" isn't the same.

 

[sretalla]

To elaborate on that a bit:

What's important is that you're able to trust your disk controller to not do anything at all which would intervene between the system (ZFS) and the disk with anything like a caching structure (memory/buffer of some sort).

You may find a mode on a "RAID" card which does really route all the communication around the buffers it's normally put in place to provide, but many have no such pathway, so any claim of HBA mode is some form of trickery in terminology and not in silicon. Worse still, you still push it through a driver made for RAID, so there may even be stuff in there which adds to the problem.

To know if the card you have will really give you true HBA behavior, you would need to run it for millions of hours to prove it's as stable as the HBAs and driver combinations recommended by the forums here.

If you don't have the time for that and care about your data, it's just not worth taking the risk.

Of course it's your hardware and your data, so you can do as you please.

 

[SecCon]

What's important is that you're able to trust your disk controller to not do anything at all which would intervene between the system (ZFS) and the disk with anything like a caching structure (memory/buffer of some sort).

You may find a mode on a "RAID" card which does really route all the communication around the buffers it's normally put in place to provide, but many have no such pathway, so any claim of HBA mode is some form of trickery in terminology and not in silicon. Worse still, you still push it through a driver made for RAID, so there may even be stuff in there which adds to the problem.

To know if the card you have will really give you true HBA behaviour, you would need to run it for millions of hours to prove it's as stable as the HBAs and driver combinations recommended by the forums here.

Understood and recommendations taken seriously. Or I would not be here.