NVMe problems - are there nightlies based on 12-STABLE, already?

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Hi all,

we have some problems with our shiny new NVMe only FreeNAS systems. Looks like FreeBSD 11 is missing completion interrupts and subsequently running into timeouts. This leaves the system "hanging" whenever the problem occurs.

Warner Losh (imp) has helped to track down the root cause - we are still not quite sure if it is really a general problem in the FreeBSD driver or rather erratically behaving firmware, problems with the mainboard ... could be anything to be honest.

But one workaround that seems to get the system into a stable working condition is to run FreeBSD 12-STABLE instead of 11. I have ben extensively testing a spare box with stock FreeBSD without FreeNAS and a simple swap of the kernel (thanks to boot environments) reliably switches the problem off or back on.

So: is there a version of FreeNAS based on 12-STABLE? I would not mind running a nightly build in production - we could test if the few features we need work, and then not upgrade until at least a beta is out. Internal systems, so no big security risk, either.

If not I would try to build a 12-STABLE kernel with the FreeNAS modifications, make sure COMPAT_11 is active, and manually patch that into a fresh boot environment with 11.2-U3.

Comments? If you are interested in the entire conversation, start here please:
https://lists.freebsd.org/pipermail/freebsd-stable/2019-April/090818.html

Thanks and kind regards,
Patrick
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
So: is there a version of FreeNAS based on 12-STABLE? I would not mind running a nightly build in production - we could test if the few features we need work, and then not upgrade until at lease a beta is out. Internal systems, so no big security risk, either.
I hope there is another use running nightly builds that can help with your answer here, but I would suggest submitting a bug ticket on this so the development team is aware because they don't always follow discussions on the forum. I am sure they would be interested to know what you have found. They are not able to directly test all hardware so the more input they can get from the community, the better they can make FreeNAS.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
OK, so that went easier than expected ...

1. Build a FreeBSD 12-STABLE kernel:
Code:
include        GENERIC
ident        FREENAS

## FreeNAS specific modifications
# misc
device        padlock
device        snp
device        ucom

# Watchdog support
device        ichwd
device        amdsbwd
device        viawd
device        wbwd
## End FreeNAS specific section

Install it somewhere convenient and tar it.

2. On the FreeNAS
Code:
beadm create 11.2-U3-K12
beadm activate 11.2-U3-K12

Reboot
Add loader.conf tunable: zfs_load="YES"
Code:
cd /boot
mv kernel kernel.old
mv modules modules.old
tar xvfz /path/to/kernel12.tar.gz

Reboot

3. Result
I'm running a 12-STABLE kernel with a FreeNAS 11.2-U3 userland and the system looks stable so far.

4. Open question:
What is freenas_sysctl.ko, do I need it and if yes, can I build it for FreeBSD 12-STABLE?

Kind regards,
Patrick
 

sovking

Cadet
Joined
Apr 25, 2014
Messages
9
Have you tried to use nda instead nvd ?
In one of my freenas installation I added hw.nvme.use_nvd="0" into /boot/loader.conf.local
 
Last edited:

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Yes, I tried - did not change anything.
 

Geek Baba

Explorer
Joined
Sep 2, 2011
Messages
73
Top