NVMe M.2 Issues

[Andy_UK]

I wasn't sure whether this should go in the hardware or storage thread. Please move it if it is incorrect.

I am currently testing my new build and am attempting to use an NVMe M.2 drive for L2ARC.

The M.2 is mounted via a Supermicro AOC-SLG3-2M2 card into a HP DL380 G7 server. FreeNAS is 11.0-U1

The drive is detected by the BIOS and FreeNAS, but when I try to use it for L2ARC I get the following via the console:

Code:

nvme0: READ squid:8 cid:127 nsid:1 lba:4194431 len:1
nvme0: ABORTED - BY REQUEST (00/07) squid:8 cid:127 cdw0:0
nvme0: resetting controller
nvme0: aborting outstanding i/o

I then tried to use the drive as a standard volume, but got the following output via GUI:

Code:

Environment:

Software Version: FreeNAS-11.0-U1 (aa82cc58d)
Request Method: POST
Request URL: http://192.168.1.142/storage/volumemanager/


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  39.			 response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.			 response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.			 response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.		 return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.				 return view_func(request, *args, **kwargs)
File "./freenasUI/storage/views.py" in volumemanager
  157.		 if form.is_valid() and form.save():
File "./freenasUI/storage/forms.py" in save
  290.					 init_rand=init_rand,
File "./freenasUI/middleware/notifier.py" in init
  238.			 f(objectid, *args, **kwargs)
File "./freenasUI/middleware/notifier.py" in _init_volume
  1057.		 self.__create_zfs_volume(volume, swapsize, kwargs.pop('groups', False), kwargs.pop('path', None), init_rand=kwargs.pop('init_rand', False))
File "./freenasUI/middleware/notifier.py" in __create_zfs_volume
  802.			 vdevs = self.__prepare_zfs_vdev(vgrp['disks'], vdev_swapsize, encrypt, volume)
File "./freenasUI/middleware/notifier.py" in __prepare_zfs_vdev
  738.								  swapsize=swapsize)
File "./freenasUI/middleware/notifier.py" in __gpt_labeldisk
  409.				 raise MiddlewareError(f'Unable to GPT format the disk "{devname}": {error}')

Exception Type: MiddlewareError at /storage/volumemanager/
Exception Value: [MiddlewareError: b'Unable to GPT format the disk "nvd0": gpart: Input/output error\n']

Using the drive is by no means critical and should it be incompatible I'll just run without it. I'm just interested in knowing whether this is config related.

Thanks.

 

[jgreco]

I'd suspect that this is highly dependent on whether or not the system supports PCIe bifurcation in a way that's compatible with Supermicro. You're essentially cramming two devices into a single slot, and if the host system isn't correctly configured at the hardware level by the system BIOS, you'd run into all sorts of problems.

Supermicro has had a lot of incentive to make this sort of thing work on its boards, but not all the vendors do. You might have a better chance with something like

http://www.addonics.com/products/ad3m2spx4.php

There's no guarantee that will work, either, but it would help clarify if this was a bifurcation thing or a general hardware compatibility issue. There are ways to dig into this deeper with "lspci -vvxxx" but learning to understand how to debug problems with the output there is probably not a beginner's level exercise.

 

[Ericloewe]

I'd suspect that this is highly dependent on whether or not the system supports PCIe bifurcation in a way that's compatible with Supermicro. You're essentially cramming two devices into a single slot, and if the host system isn't correctly configured at the hardware level by the system BIOS, you'd run into all sorts of problems.

But would it even show up in the OS? Would it get far enough to be recognized as an NVMe device?

ou might have a better chance with something like

http://www.addonics.com/products/ad3m2spx4.php

Good idea, that'll at least verify if the drive itself is good. Should it be, the guaranteed compatibility solution is to use one of those M.2 adapters that have a PCIe switch to allow for several devices connected to one PCIe slot.

 

[Andy_UK]

Thanks for the replies. I'll take a look at the Addonics.

 

[jgreco]

But would it even show up in the OS? Would it get far enough to be recognized as an NVMe device?

I imagine that has a lot to do with how well the things that do work actually work. I believe that FreeBSD is mostly looking at the PCI configuration data passed in from the BIOS to determine device presence, so if that "works" but doesn't actually WORK then you could have problems, yes.

Not to put too fine a point on it, but my first thought was along the lines of "HP has little incentive to make that work with another vendor's passive card."

I'd hope that either the Addonics or the (probably much pricier) PLX switch based cards, I think there may be some with M.2 support, would work.

 

[Ericloewe]

PLX switch based cards, I think there may be some with M.2 support

There definitely are. I think Supermicro has one such model.

 

[Andy_UK]

I've managed to quickly get my hands on a Lycom DT-120. It appears to work without issue, both as L2ARC and a standard volume.

Lesson learnt - don't assume the more expensive card will be better!

 

[jgreco]

Well, in this case, I think that the more expensive card *would* have been better - a PLX switch based card. The problem here is likely to be the lack of PLX switch, so choosing a cheap card that is probably reliant on motherboard bifurcation support becomes a real pain point.

You'd think that computers would be designed so that something relatively trite like PCIe lane mappings would just work, wouldn't you. At least, I think any sane person would assume this. :-/

 

[Ericloewe]

Well, in this case, I think that the more expensive card *would* have been better - a PLX switch based card. The problem here is likely to be the lack of PLX switch, so choosing a cheap card that is probably reliant on motherboard bifurcation support becomes a real pain point.

You'd think that computers would be designed so that something relatively trite like PCIe lane mappings would just work, wouldn't you. At least, I think any sane person would assume this. :-/

With NVMe becoming more popular, I expect this to be more of a priority on new systems.

 

[jgreco]

With NVMe becoming more popular, I expect this to be more of a priority on new systems.

It is possible that you have misunderstood how the server hardware business works.

There is little incentive for the big boys to make a cheap third-party manufacturer's card bifurcation work when they have things like the HP Z Turbo Drive to sell to you at full price.

It is easy to forget that the money is in all the value-adds, not the core hardware, so there's a lot of disincentive to make "cheaper" alternatives work.

 

[Stux]

FWIW, I was checking out SuperMicro's Xeon D boards last night. Apparently they now have full PCIe bifurcation support on the x16 slot. All that is needed is a clock buffer on the riser cards

(4/4/4/4 I think)

Doesn't help OP

 

[Stux]

FWIW, I was checking out SuperMicro's Xeon D boards last night. Apparently they now have full PCIe bifurcation support on the x16 slot. All that is needed is a clock buffer on the riser cards

(4/4/4/4 I think)

Doesn't help OP

Actually playing with my X10SDV build at the moment :)

Super impressed with the bifurcation options on SLOT7

 

[Colin-sg]

I've managed to quickly get my hands on a Lycom DT-120. It appears to work without issue, both as L2ARC and a standard volume.

Lesson learnt - don't assume the more expensive card will be better!

Hi Andy, can you advise what NVMe M.2 drive you are using? i have the DL380 G7, and is considering to improve the storage

 

[Henius]

Hi, stuck in the same "reseting controller" thing.
Can even install FreeNAS on the SSD. Isn't tied with
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211713

?

 

[Ericloewe]

Hi, stuck in the same "reseting controller" thing.
Can even install FreeNAS on the SSD. Isn't tied with
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211713

?

You're giving us very little to go on, so please open a new thread, include your hardware, and tell us exactly what you're trying to do.

 

[JKK]

Hi, I managed to get my Intel nvme m.2 ssd in the same "resetting controller" issue, while I reinstalled FreeNas after adding couple of drives etc. The Intel ssd showed up in "View Disks", but couldn't add it in Volume Manager, or wipe the disk. Turns out the issue was me turning on Power saving/ERP in motherboard Bios while checking settings. Turned power savings off again, and I could add the Intel ssd again in Volumes.