Scale doesn't have drivers for LSI 9201-16e

[oumpa31]

I was trying to test Scale on my main server that is currently Core. I use a LSI 9201-16e to connect to my disk shelfs but none of the disks would show up. I think Scale is missing the drivers for it. Is there anyway to add the driver for it to scale.

 

[brando56894]

It should work without issue, I'm using a LSI 9201-16i and the only problem that I have is that Linux boots up too quickly to initialize the HBA, which halts the boot process, but once you manually import the boot pool it will detect the HBA and load the drives. see here

Once you're booted into SCALE, drop to a shell and execute lspci -k|grep mpt3sas if that returns output, then the modules are indeed loaded and you have another issue.

The full output for lspci (cut down to just show the HBA output) should look like this

Code:

Serial Attached SCSI controller: Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02)
        Subsystem: Broadcom / LSI SAS 9201-16i
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas

 

[oumpa31]

@brando56894 Thanks for that help. I will have to try that later. I'll have to put Scale on a flash drive just to test it to see the outputs. I saw nightly go to 21.04 so i'm getting a bit excited for the release. lol

 

[brando56894]

You're welcome. I tried to switch my personal server to using the SCALE nightlies but it still has too many issues for me to use it day to day.

 

[oumpa31]

I can understand that. This really is the last test i need to run so i know once the release is out its ready to go. Hardware transcoding plex with a p4000 will take a huge load off my threadripper cpu.

 

[Kris Moore]

@oumpa31 - If its still not working after poking at it, please let us know via a jira.ixsystems.com ticket. Those are the kinds of things we want to fix during these pre-release cycles :)

 

[oumpa31]

@Kris Moore i reported the bug here https://jira.ixsystems.com/browse/NAS-109947 but was told they do not have this card hanging around.

testing was done on TrueNAS-SCALE-21.04-MASTER-20210402-015242

lspci -k|grep mpt3sas gives mme this

truenas# lspci -k|grep mpt3sas
Kernel modules: mpt3sas
Kernel driver in use: mpt3sas
Kernel modules: mpt3sas

lspci gives me a ton of stuff but these are the only sas cards neither are mine.

09:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02)
42:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02)

So I SSH'ed into the box and added this
add rootdelay=10 to the end of GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub.d/truenas.cfg and regenerate grub (via the update-grub command)
Nothing happened so I restarted the server and that disappeared.
I then added
add ZFS_INITRD_POST_MODPROBE_SLEEP='10' or ZFS_AUTOIMPORT_TIMEOUT='10' to /etc/default/zfs and then regenerate the initrd with update-initrd -k all -u
tried both and when i run the update i get this
truenas# update-initrd -k all -u
zsh: command not found: update-initrd

still no changes after reboot

 

[oumpa31]

I'm at the point of looking into a LSI SAS 9305-16E card but I don't know if that would work either. Plus that means new cables too and don't really want to spend the money unless I know it would work.

 

[Kris Moore]

I did some googling on issues with this card on stock debian, and found this:

1878332 – mpt3sas modules fails to setup LSI 9201-16e on 5.8 series kernels

bugzilla.redhat.com

Seems to indicate its a driver / kernel issue, but there was a work-around by doing this to add some args to grub:

Code:

grubby --update-kernel=ALL --args="mpt3sas.max_queue_depth=10000"

Can you experiment with that setting by adding it to grub and seeing if it'll fix the issue?

 

[oumpa31]

I did some googling on issues with this card on stock debian, and found this:

1878332 – mpt3sas modules fails to setup LSI 9201-16e on 5.8 series kernels

bugzilla.redhat.com

Seems to indicate its a driver / kernel issue, but there was a work-around by doing this to add some args to grub:

Code:

grubby --update-kernel=ALL --args="mpt3sas.max_queue_depth=10000"

Can you experiment with that setting by adding it to grub and seeing if it'll fix the issue?

I definitely can do that. Do I just type that in the Shell? I just want to make sure I know where to put it I can test it soon.

 

[Kris Moore]

In our case you'd edit by hand /boot/grub/grub.cfg and look for the first boot-entry which looks like this:

Modify it to look like this:

Save and reboot, then see if device shows up.

 

[oumpa31]

In our case you'd edit by hand /boot/grub/grub.cfg and look for the first boot-entry which looks like this:

View attachment 46353

Modify it to look like this:

View attachment 46354

Save and reboot, then see if device shows up.

I use WinSCp to edit that file and save but that code disappeared after restart. I was able to import the test pool i had made but when the system restarted the pool was no longer connected.

How can I make that a permanent change?

 

[waqarahmed]

@oumpa31 can you please open "/usr/local/bin/truenas-grub.py" file via SSH and then update

Code:

    config = [
        'GRUB_DISTRIBUTOR="TrueNAS Scale"',
        'GRUB_CMDLINE_LINUX_DEFAULT="libata.allow_tpm=1 systemd.unified_cgroup_hierarchy=0 '
        'amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on"',
    ]

to

Code:

    config = [
        'GRUB_DISTRIBUTOR="TrueNAS Scale"',
        'GRUB_CMDLINE_LINUX_DEFAULT="libata.allow_tpm=1 systemd.unified_cgroup_hierarchy=0 '
        'amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on mpt3sas.max_queue_depth=10000"',
    ]

After doing this, please execute "midclt call etc.generate grub" and then this change should persist across reboots but not upgrades of the system.

Please let me know if there are issues. Thank you

 

[oumpa31]

Code:

    config = [
        'GRUB_DISTRIBUTOR="TrueNAS Scale"',
        'GRUB_CMDLINE_LINUX_DEFAULT="libata.allow_tpm=1 systemd.unified_cgroup_hierarchy=0 '
        'amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on mpt3sas.max_queue_depth=10000"',
    ]

After doing this, please execute "midclt call etc.generate grub" and then this change should persist across reboots but not upgrades of the system.

Please let me know if there are issues. Thank you

I can use WinSCP to get to that file and edit it and save it then I went into PuTTy tried to even map to that directory so I could run
midclt call etc.generate grub
and i get this

truenas# /usr/local/bin
zsh: permission denied: /usr/local/bin

I tried to run it at the toplevel and got this
truenas# midclt call etc.generate grub
null

 

[waqarahmed]

@oumpa31 i am able to modify the file without any issues as root, can you please confirm your permissions ? And you can perhaps use "nano" ( i have not used WinSCP so not sure if there are any catches to be aware of ) ?

 

[oumpa31]

@oumpa31 i am able to modify the file without any issues as root, can you please confirm your permissions ? And you can perhaps use "nano" ( i have not used WinSCP so not sure if there are any catches to be aware of ) ?

I'm logging in as root. i just got Git and nano so loading them now

 

[oumpa31]

@waqarahmed ok maybe im doing something very wrong trying to ssh in. PuTTY will let me in but all i get is this
login as: root
root@192.168.1.10's password:
Last login: Tue Apr 6 09:30:49 2021 from 192.168.1.30

TrueNAS (c) 2009-2021, iXsystems, Inc.
All rights reserved.
TrueNAS code is released under the modified BSD license with some
files copyrighted by (c) iXsystems, Inc.

For more information, documentation, help or support, go here:
http://truenas.com
Welcome to TrueNAS
truenas# ls
samba syslog
truenas#

 

[waqarahmed]

@oumpa31 after logging in with Putty, please type "nano /usr/local/bin/truenas-grub.py" and then modify the lines. Please let me know if there are issues, thank you

 

[oumpa31]

@oumpa31 after logging in with Putty, please type "nano /usr/local/bin/truenas-grub.py" and then modify the lines. Please let me know if there are issues, thank you

Ok that worked thank you. The code I added in WinSCP was still there so I saved it. Then it brings me back to the main level and I ran
midclt call etc.generate grub and it gives me an answer of null for shitts and giggles I decided to try a restart and it seams to have worked. my test pool I created for importing from core to scale on core was still there.

There was another nighty update that was pushed out so should i see if this stays even after an update?

 

[oumpa31]

@waqarahmed I did an update to the latest nightly which is just as good as reloading the OS just to confirm the fix, and that is definitely the fix.
When I reloaded the OS my test pool wasn't connected. So I went and edited the /usr/local/bin/truenas-grub.py file added that line of code to the end where you told me to. Saved it and ran the "midclt call etc.generate grub" code. I then restarted the server and everything came up like it should.

Thank you so much @Kris Moore and @waqarahmed for your help in hunting this down. I added the fix to my bug report https://jira.ixsystems.com/browse/NAS-109947 I hope this can be added to the OS code soon. I am very excited for 21.04 release.