diskinfo submit error

[Yves_]

Hi there,

Just tried to make a diskinfo on my Intel Optane 900p but it always crashes. Any idea why?

Code:

diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        280065171456    # mediasize in bytes (261G)
        547002288       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPED1D280GA     # Disk descr.
        PHMB742400E4280CGN      # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     20.7 usec/IO =     23.6 Mbytes/s
           1 kbytes:     21.1 usec/IO =     46.3 Mbytes/s
           2 kbytes:     21.8 usec/IO =     89.8 Mbytes/s
           4 kbytes:     13.7 usec/IO =    284.8 Mbytes/s
           8 kbytes:     18.6 usec/IO =    419.3 Mbytes/s
          16 kbytes:     25.4 usec/IO =    616.3 Mbytes/s
          32 kbytes:     35.8 usec/IO =    874.0 Mbytes/s
          64 kbytes:     52.6 usec/IO =   1189.0 Mbytes/s
         128 kbytes:     85.3 usec/IO =   1465.8 Mbytes/s
         256 kbytes: diskinfo: AIO write submit error: Operation not supported

Thanks for your help

Cheers,
Yves

 

[gonzoWL]

Same here with P4800 (or P4801? cannot see in the chassis) - U.2 in a simple PCIE-Adapter

(hello to all - I'm new here and love your work! since one year...),

I'm working now on a Dell R730 - first on ESXi, now on bare metal with FreeNAS 11.3 U3.2.

The strenge thing is:
1) I had same (identical) behaviour like @Yves_ before, on the HP Z440 (bare metal) and now on Dell;
2) I thought, Z440 is not a good machine, only workstation etc. etc.
3) first try on R730 was a short test - I've putted a SATA SSD with previous FreeNAS install (sorry, forgot which version it was, about 11.2 I guess) and the box started nice. Test with the Optane was good - up to 1070 MB/s, all block sizes (is this the first value in the line of diskinfo output?)
4) after that I was experimenting with ESXi - with problems for nvme at all; so I've installed FreeNAS directly;

And now back on FreeNAS on bare metal R730: 256 kbytes: diskinfo: AIO write submit error: Operation not supported...

But it was OK on the identical config - I don't understand that...

And have 2 questions:
1) is this a problem? Is there a solution? i've already tried some TRIM etc. tricks - without success...
2) maybe I'm doing something wrong?
3) is this important for production use as SLOG? If a testing tool doesn't work is not bad - but if it means, there is a problem...

On the Optane is no data, freshly wiped...

Thanks for any ideas - in advance!


Wojtek

 

[wl714]

Same for me with P3700 (2.5" U.2 version)

Code:

root@freenas[~]# smartctl -a /dev/nvme2
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPE2MD020T4
Serial Number:                      CVFT7385000P2P0SGN
Firmware Version:                   8DV10171
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          2,000,398,934,016 [2.00 TB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Thu Jul  9 15:34:59 2020 CST
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
0 +    25.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
0 +     512       0         2
1 -     512       8         2
2 -     512      16         2
3 -    4096       0         0
4 -    4096       8         0
5 -    4096      64         0
6 -    4096     128         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        41 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    4 [2.04 MB]
Data Units Written:                 12,216 [6.25 GB]
Host Read Commands:                 47
Host Write Commands:                410,250
Controller Busy Time:               0
Power Cycles:                       1
Power On Hours:                     0
Unsafe Shutdowns:     

Code:

root@freenas[~]# diskinfo -wS /dev/nvd2
/dev/nvd2
        512             # sectorsize
        2000398934016   # mediasize in bytes (1.8T)
        3907029168      # mediasize in sectors
        131072          # stripesize
        0               # stripeoffset
        INTEL SSDPE2MD020T4     # Disk descr.
        CVFT7385000P2P0SGN      # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     18.1 usec/IO =     27.0 Mbytes/s
           1 kbytes:     17.8 usec/IO =     54.9 Mbytes/s
           2 kbytes:     17.3 usec/IO =    113.2 Mbytes/s
           4 kbytes:     15.4 usec/IO =    253.3 Mbytes/s
           8 kbytes:     17.4 usec/IO =    447.9 Mbytes/s
          16 kbytes:     22.2 usec/IO =    704.2 Mbytes/s
          32 kbytes:     29.2 usec/IO =   1072.0 Mbytes/s
          64 kbytes:     40.8 usec/IO =   1531.2 Mbytes/s
         128 kbytes:     72.2 usec/IO =   1731.0 Mbytes/s
         256 kbytes: diskinfo: AIO write submit error: Operation not supported

My test system info:
FreeNAS-11.3-U3.2
CPU: 2X CPU E5-2650L v3
Board Chipset: PCH C610
U.2 HBA : LinkReal LRNV9349-8I (PCIe3.0x16 with 8ports SFF8643) connected via SFF8643 to SFF8639 cable to P3700 (Note: 3.3V is not supplied on this cable as my mainboard has 5V and 12V connectors only, and Intel support told me 3.3V is optional)

Where could the problem be? The SSD itself (engineering sample or some water down version), missing 3.3V power, disk controller, motherboard, some configuration, ...? I will have a LSI9400-16i to test in a few days and post after more testing. At the mean time, I would really appriciate if anyone can point me in the right direction. Thanks!
PS. I create a pool from it and add a SMB share and seems to work ok for read/write in my brief testing. What test can I do to find out more about the SSD or controller?

 

[gonzoWL]

Reeding some additional sources leads me to the conclusion, that there is a software problem at the time with diskinfo, but I'm not sure... I'd like a hint, too...

 

[wl714]

How to do testing similar to diskinfo -wS in Win10 and/or Linux?

 

[gonzoWL]

Unfortunately, I don't know what has the exact same functionality... Under Linux I normally use smartctl and hdparm to see, what disk says. But I never used a configuration with SLOG under Linux, so I never had the need to test this kind of configuration...

Greets!

 

[wl714]

I found the man page of diskinfo, under -S switch says : Blocks of more then 128KB are written with multiple parallel operations. So, diskinfo didn't run into problem until it try to spawn parallel write operations. And it imply all basic write operation for 128KB or less are working fine. From what I know that's all the job required of the disk controller and a block device. Schedule and run multiple parallel read/write operations on them is the job of the OS. So it makes no sense to me why diskinfo works for others with similar or same SSD but not ours. FreeNAS handles multiple I/O requests on the same device all the time. Why can't diskinfo?

 

[gonzoWL]

I found the man page of diskinfo, under -S switch says : Blocks of more then 128KB are written with multiple parallel operations.

Is this new?

As I wrote above, my attempt with an older version of FreeNAS was 100% successful. The same system, the same SSD but 11.3 => and the error was there...

No idea why diskinfo can't that... My skills are only some php, scripting on shell and configuring some stuff.

 

[wl714]

This is from my current build 11.3-U3.2 Bottom shows July 4, 2017, so this man page has been around for years. Do you know the version of your older/working FreeNAS?

Code:

root@freenas[~]# man diskinfo
DISKINFO(8)             FreeBSD System Manager's Manual            DISKINFO(8)

NAME
     diskinfo – get information about disk device

SYNOPSIS
     diskinfo [-citSvw] disk ...
     diskinfo [-p] disk ...
     diskinfo [-s] disk ...

DESCRIPTION
     The diskinfo utility prints out information about a disk device, and
     optionally runs a naive performance test on the device.

     The following options are available:

     -v     Print fields one per line with a descriptive comment.

     -c     Perform a simple measurement of the I/O read command overhead.

     -i     Perform a simple IOPS benchmark.

     -p     Return the physical path of the disk.  This is a string that
            identifies the physical path to the disk in the storage enclosure.

     -s     Return the disk ident, usually the serial number.

     -S     Perform synchronous random write test (ZFS SLOG test), measuring
            time required to write data blocks of different size and flush
            disk cache.  Blocks of more then 128KB are written with multiple
            parallel operations.

     -t     Perform a simple and rather naive benchmark of the disks seek and
            transfer performance.

     -w     Allow disruptive write tests.

     If given no arguments, the output will be a single line per specified
     device with the following fields: device name, sectorsize, media size in
     bytes, media size in sectors, stripe size, stripe offset, firmware
     cylinders, firmware heads, and firmware sectors.  The last three fields
     are only present if the information is available.

HISTORY
     The diskinfo command appeared in FreeBSD 5.1.

AUTHORS
     The diskinfo utility was written by Poul-Henning Kamp <phk@FreeBSD.org>.

BUGS
     There are in order of increasing severity: lies, damn lies, statistics,
     and computer benchmarks.

FreeBSD 11.3-RELEASE-p9          July 4, 2017          FreeBSD 11.3-RELEASE-p9

 

[gonzoWL]

Right, doesn't look like new. Maybe is the OS-Handling new.

My random test was with - i guess, not really sure - 11.2... Just the last one on a stick pulled from previous machine... I bought a Dell R730 and wanted to see if the small Optane would be good for SLOG. Just started and everything was fine, exactly like in tests on servethehome...

 

[wl714]

I am downloading src.txz of FreeBSD 11.2 and 11.3, I hope I can find diskinfo source code in them to diff. Even if identical, maybe I can get some hint about when/why diskinfo puts out the AIO error message (when I have the time to spare)

 

[gonzoWL]

Good luck! But don't forget: maybe it is not diskinfo itself - maybe it is the OS, how does it handle the write operations...

 

[wl714]

diskinfo on FreeBSD 11.2 and 11.3 are identical except one line of comment. It yields that error message when the system call to aio_write() returns -1 (=>failed to schedule the async write operation). The file descriptor used to setup aio_write is the same one used to make sync pwrite call. The function that does these sync and async writes (for slogbench) here:

Code:

parwrite(int fd, size_t size, off_t off)
{
    struct aiocb aios[MAXIOS];
    off_t o;
    int n, error;
    struct aiocb *aiop;

    // if size > MAXIO, use AIO to write n - 1 pieces in parallel
    for (n = 0, o = 0; size > MAXIO; n++, size -= MAXIO, o += MAXIO) {
        aiop = &aios[n];
        bzero(aiop, sizeof(*aiop));
        aiop->aio_buf = &buf[o];
        aiop->aio_fildes = fd;
        aiop->aio_offset = off + o;
        aiop->aio_nbytes = MAXIO;
        error = aio_write(aiop);
        if (error != 0)
            err(EX_IOERR, "AIO write submit error");
    }
    // Use synchronous writes for the runt of size <= MAXIO
    error = pwrite(fd, &buf[o], size, off + o);
    if (error < 0)
        err(EX_IOERR, "Sync write error");
    for (; n > 0; n--) {
        error = aio_waitcomplete(&aiop, NULL);
        if (error < 0)
            err(EX_IOERR, "AIO write wait error");
    }
}

Anyone knows why aio_write() failed to schedule while pwrite runs without problem?