More than 8 drives with Intel expander RES2SV240 and/or dual AOC-S3008L-L8e

[zbyte]

Hi everyone:

About a year and some ago I installed my 1st Truenas Core server, I experimented virtualizing it over ESXi (with ECC memory and the such) and so far it's been working steady with no issues. I passthrough a Supermicro AOC-S3008L-L8e with the latest firmware (from Supermicro) and created a pool with 2x RAIDZ2 vdevs, 4x4TB and 4x6TB drives. Scrubs and SMART checks performed regularly with no issues. I have been doing the software updates with no issues, performance is good and all that.

Now, I need to add storage to my pool so I decided to replace the 4x4TB vdev with 12TB drives, as you all know there are two methods for this. The preferred method were you replace the disk without degrading the pool and the other method were you actually take the disk offline to do so all this is detailed here:

9. Storage — FreeNAS®11.2-U5 User Guide Table of Contents

www.ixsystems.com

Since I don't want to degrade the pool I wanted to proceed with the replace method. I do have a RES2SV240 Intel expander, and I added the expander connecting it to the AOC-S3008L-L8e HBA controller, leaving 4 drives in the HBA and 4 drives in the expander. When Truenas comes back online it detects all the drives connected to the expander and only ONE (1) drive connected to the HBA, which is odd.

I decided then to remove the expander and add a second HBA AOC-S3008L-L8e (same firmware) I purchased as spare connecting 4 drives (1 vdev) to each card, passthrough the card to ESXi and Truenas boot in degraded state as the last drive da8 on the pool is detected but with errors.

All I'm trying to do is to be able to see 9 drives so I can start replacing the 4TB's by 12TBs

I wonder if aside of ESXi and virtualized Core not being supported and these issues could perfectly be due to the virt environment if anyone has any previous experience like this.

At some point in the future I will have a baremetal system but for now this will have to do.

I'm currently backing up the entire contents so I can destroy the pool and re create it with the new drives but replacing and resilvering would be great to have

Thank you in advance for any answers.

 

[Nick2253]

Just to make sure, are the HBA cards flashed in IT mode?

In theory, the expander should work, but I'm not experienced enough to be confident. However, passing through both HBAs should definitely work, so it's weird that you're unable to get the pool working.

I would suggest the following:

1. Get into a known good state (one HBA, all vdevs working).
2. Then, without changing anything else, add the second HBA.
3. Add the 12TB drives to this second HBA, and attempt the disk replacement.

 

[jgreco]

I wonder if aside of ESXi and virtualized Core not being supported and these issues could perfectly be due to the virt environment if anyone has any previous experience like this.

There's nothing that should be deliberately limiting here. The L8e is an IT-mode HBA with support for up to 122 drives (behind an expander obviously). It's possible some cabling is bad.

 

[zbyte]

Just to make sure, are the HBA cards flashed in IT mode?

Yes

I would suggest the following:

1. Get into a known good state (one HBA, all vdevs working)

Thank you for the hints, I just replaced with the good pool the HBA by the other one, it seems to be a good HBA, same firmware, same model, same manufacturing date, same everything, however in that pool the da4 drive fails to boot with this specific HBA, like if the channel taking care of that drive is bad. My mistake with the HBA was to "assume it worked properly"

There's nothing that should be deliberately limiting here. The L8e is an IT-mode HBA with support for up to 122 drives (behind an expander obviously). It's possible some cabling is bad.

Yeah and I agree, I swapped cables still same results, either the expander is bad or this is related to the way ESXi handles the HBA, and thinking over again it should be entirely transparent yes but for some reasons it fails.

No big of a deal, thank you for the answers