HBA Passthrough Dell R230

hardlivinlow

Dabbler
Joined
May 31, 2017
Messages
30
Ive been running a Dell R230 with, I think its a H210 flashed to LSI firmware in IT mode. It has been running great for a few years. I updated to TrueNAS 13 and now the system wont boot with the controller passed through to TrueNAS. Im running ESXI 7 U3. I reinstalled ESXI and it worked ok until I tried transferring data to the pool and the VM will just crash and cut off. I can remove the controller from the VM and the system will boot fine. Below is the error I get.

Do I need to upgrade the HBA to make this work with TrueNAS 13? Appreciate any info. Thanks

Capture.JPG
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
Do I need to upgrade the HBA to make this work with TrueNAS 13?
I don't recall seeing anything about needing to upgrade the HBA firmware.

Did you upgrade the pool? Hopefully not, then you can roll back to the previous version of TrueNAS. I never upgrade my pool unless there is a new feature I will use.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Do I need to upgrade the HBA to make this work with TrueNAS 13? Appreciate any info. Thanks

Based on historical reports here, yes. The only way to make the "doorbell stop ringing" seems to be jumping to the SAS2308 or beyond controllers such as the 9207-8i. It seems to be something that crops up when you get the combination of all three of the below:

  • SAS2008 based HBA
  • FreeBSD 13.x
  • TrueNAS as ESXi VM

Change any one of those (SAS2008 + TrueNAS 12.x in a VM) and it works - but add all three, and it's like the fire-triangle of "fuel, oxygen, and ignition source"

I believe an upstream bug has been filed with FreeBSD but this might be a VMware issue given that it seems to hang up until the bare-metal host is rebooted. The bug seems to still exist in ESXi 7.0 as well - I don't know if anyone's been brave enough to try ESXi 8.0 yet.
 

hardlivinlow

Dabbler
Joined
May 31, 2017
Messages
30
Thanks for you info!, I have a LSI 9207-8i (SAS2308) on order already for another project. Ill order another for this box. I figured it was a combatibility problem with FreeBSD 13/ESXi. I was running 7 U1 and I was like maybe I can try U3. It worked and I thought I won.. But I was quicky shot back down lol
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
It isn't this simple. I've seen it also on VM's with large memory, where dropping the amount of memory results in it working successfully.
Right, I remember reading that as well, where some users experienced issues after reaching 128GiB of RAM assigned to a VM - but I also saw users reporting the same doorbell issue at 64GiB assigned.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
but I also saw users reporting the same doorbell issue at 64GiB assigned.

That was probably me. I had gobs of RAM (256GB at the time) on one system and made the NAS big (128) just for giggles, then paid the price 'cuz I had to restart the hypervisor. I believe I discovered that 64GB also failed if the system only had 128GB.
 

Suril

Cadet
Joined
Dec 4, 2022
Messages
4
If you're running a multi-CPU machine, most of the time you'll experience MUCH greater performance if your virtual workload is hosted on a single NUMA node.

e.g. I have a dual socket machine with 256GB RAM (128GB per physical CPU). I shouldn't allocate more than around ~120GB RAM (half system RAM + small overhead) to a particular VM to ensure that only the RAM directly connected to the hosting CPU is used. The same goes for virtual CPU cores - don't allocate more than what you have on one physical CPU to a VM. Different hypervisor platforms will have settings that let you control whether or not your workload should span across multiple nodes. ESXi and Hyper-V do for example.
 

hardlivinlow

Dabbler
Joined
May 31, 2017
Messages
30
Just wanted to swing back in and let everyone know that upgrading to the LSI 9207-8i (SAS2308) did fix the problem. Just transferred 50G to the pool with no issues. :cool:
 
Top