Horrible NVMe pool performance

Status
Not open for further replies.

jblakeney

Dabbler
Joined
May 6, 2015
Messages
10
I am setting up a new system and I am seeing horrendous performance from the pool I create.

Specs:
  • FreeNAS-9.10.2-U1 (86c7ef5)
  • 2xIntel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
  • 128GB RAM
  • SuperMicro X10DRU-i+
  • 6 X Intel P3700 NVMe
No matter how I create the pool I can only get 170MB/s
Code:
[root@freenas] /mnt/data# dd if=/dev/zero of=/mnt/data/bifgile bs=1024 count=1024000
1024000+0 records in
1024000+0 records out
1048576000 bytes transferred in 6.095852 secs (172014671 bytes/sec)

I have tried RaidZ2, Mirror of 2 drives, Mirror / Stripe 3x2, and single drive.
All produce within 1MB/s of that speed.

When I ran the badblocks burn-in it only took 3 hours to read / write the drive 4 times or ~ 1.2 GB / sec per drive.

nvmecontrol gives:
Code:
[root@freenas] /mnt/data# nvmecontrol perftest -n 32 -o write -s 512 -t 30 nvme3ns1
Threads: 32 Size:  512 WRITE Time:  30 IO/s:  13636 MB/s:  6

I know I have something configured / set wrong but I have never seen that level of degradation in performance.

Now that I think about it, that is such a small file (1GB) for the dd test it should be cache / RAM speed and badblocks / nvmecontrol are likely direct IO.

I ran a memory test for 24 hours with no errors.

Other odd notice. When I run nvmecontrol it never change the cpu utilization in top, but dd goes to 100% as did rsync (90%) copying the file locally.
Code:
[root@freenas] /mnt/data# rsync -vvaiP bifgile bifgile1
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
>f+++++++++ bifgile
 10,485,760,000 100%  183.71MB/s  0:00:54 (xfr#1, to-chk=0/1)
total: matches=0  hash_hits=0  false_alarms=0 data=10485760000

sent 10,488,320,087 bytes  received 102 bytes  188,978,742.14 bytes/sec
total size is 10,485,760,000  speedup is 1.00

I have much less powerfully systems running much faster than this and see NVMe systems in the forums topping GB/s speeds.

Thanks for all the help and work on a great product.
Jimmy
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
Write with block as small as 1024 bytes probably more tests OS overhead then backing device speed. Try to increase it to something like 64K or 1M.
 

jblakeney

Dabbler
Joined
May 6, 2015
Messages
10
Wow, I did not realize the bs made that much difference:
Code:
[root@freenas] /mnt/data# dd if=/dev/zero of=/mnt/data/bifgile bs=1M count=409600
409600+0 records in
409600+0 records out
429496729600 bytes transferred in 147.999201 secs (2902020600 bytes/sec)

400GB in 147 secs

I guess rsync is just limited in what it can do as it was also (and still is) showing ~180MB/s.

I will do further testing but this make me feel much better.
Thanks for the help.
Jimmy
 

brando56894

Wizard
Joined
Feb 15, 2014
Messages
1,537
I'm using FreeNAS 10 with a single M.2 Samsung 960 Evo and I see around 2 GB/sec read and about 900 MB/sec write when testing with dd. First off you want to make sure that the dataset that you're using with dd is uncompressed, secondly you want to create a file that is much bigger than your ARC so that it will go directly to the disks and not be cached.
 
Joined
Jul 29, 2015
Messages
6
Hi Jimmy

How did you manage to install FreeNAS onto a SuperMicro 1028U with Intel NVMe P3700 ?
I have tried installing on 4x P3700's and get the same error every time.

The installer stops at a point and shows this error :

KBD: enter : panic
[ thread pid 0 tid 100000 ]
Stopped at kdb_enter+0x3e: movq $0,kdb_why
db>


Any ideas what it is ?
Do we need special settings in the BIOS ?

I have the latest BIOS loaded ver 2.0b

Thanks in advance
Vince






I am setting up a new system and I am seeing horrendous performance from the pool I create.

Specs:
  • FreeNAS-9.10.2-U1 (86c7ef5)
  • 2xIntel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
  • 128GB RAM
  • SuperMicro X10DRU-i+
  • 6 X Intel P3700 NVMe
No matter how I create the pool I can only get 170MB/s
Code:
[root@freenas] /mnt/data# dd if=/dev/zero of=/mnt/data/bifgile bs=1024 count=1024000
1024000+0 records in
1024000+0 records out
1048576000 bytes transferred in 6.095852 secs (172014671 bytes/sec)

I have tried RaidZ2, Mirror of 2 drives, Mirror / Stripe 3x2, and single drive.
All produce within 1MB/s of that speed.

When I ran the badblocks burn-in it only took 3 hours to read / write the drive 4 times or ~ 1.2 GB / sec per drive.

nvmecontrol gives:
Code:
[root@freenas] /mnt/data# nvmecontrol perftest -n 32 -o write -s 512 -t 30 nvme3ns1
Threads: 32 Size:  512 WRITE Time:  30 IO/s:  13636 MB/s:  6

I know I have something configured / set wrong but I have never seen that level of degradation in performance.

Now that I think about it, that is such a small file (1GB) for the dd test it should be cache / RAM speed and badblocks / nvmecontrol are likely direct IO.

I ran a memory test for 24 hours with no errors.

Other odd notice. When I run nvmecontrol it never change the cpu utilization in top, but dd goes to 100% as did rsync (90%) copying the file locally.
Code:
[root@freenas] /mnt/data# rsync -vvaiP bifgile bifgile1
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
>f+++++++++ bifgile
10,485,760,000 100%  183.71MB/s  0:00:54 (xfr#1, to-chk=0/1)
total: matches=0  hash_hits=0  false_alarms=0 data=10485760000

sent 10,488,320,087 bytes  received 102 bytes  188,978,742.14 bytes/sec
total size is 10,485,760,000  speedup is 1.00

I have much less powerfully systems running much faster than this and see NVMe systems in the forums topping GB/s speeds.

Thanks for all the help and work on a great product.
Jimmy
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
Vince, it could help if you shown more of the panic context. And possibly typed `bt` command after that.
 
Joined
Jul 29, 2015
Messages
6
Vince, it could help if you shown more of the panic context. And possibly typed `bt` command after that.


Hi MA,
See attached screen shot of the full panic.
I tried to type 'bt' but the keyboard has frozen ??
FreeNAS on NVMe Panic screen-1.jpg


Can you make any sense of the screen shot ?

Regards
Vince
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
`bt` is a short version of `backtrace` -- actually the same as shown on the first part of your panic screenshot. Unfortunately the most interesting part of the panic didn't fit the screen. From provided I can only tell that it happened during device probe/attach process.
 
Joined
Jul 29, 2015
Messages
6
`bt` is a short version of `backtrace` -- actually the same as shown on the first part of your panic screenshot. Unfortunately the most interesting part of the panic didn't fit the screen. From provided I can only tell that it happened during device probe/attach process.


Ok , so is there anything I could do ? is there a new NVMe driver for FreeNAS 9.10.2 ? I am no Unix expert :-( , would there be a way to inject the new driver before the installer kicks off ?

Should I try install the last version that was available of Corral ?

Vince
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
You should not try Corral, since it is discontinued, but you may try FreeNAS 11.0-RC to see whether it improve the situation for you.
 
Joined
Jul 29, 2015
Messages
6
You should not try Corral, since it is discontinued, but you may try FreeNAS 11.0-RC to see whether it improve the situation for you.


Ok , will do so , I have looked for the 11.0-RC and cannot find it , please could you send me an URL to download it , Thanks again for the support , appreciate you patience with a Noob !!!
 
Joined
Mar 22, 2016
Messages
217
System:
Supermicro 2028U-TN24R4T+
32GB Samsung RAM
2x E5-2623 V3
FreeNAS 11.0-RC

Sorry to dig up an old thread, but I'm curious to see if anyone can help with this. I am getting the same panic message that Vince got. I'm running a Supermicro 2028U-TN24R4T+ with 4 Intel P3520 1.2TB drives and now 4 Intel p900 280GB drives. I originally had 1 p900 drive and everything functioned fine. I added the 3 other drives and I got the panic. I removed the three drives and tried booting with only 1 attached of each of the new drives and it booted just fine. Add another drive and I get the panic. If I add the drives in after the system boots it restarts on it's own.

I'm in the process of updating to the newest version of FreeNAS to see if that will sort out of the issue, but any advice in the meantime would be excellent!
 

jblakeney

Dabbler
Joined
May 6, 2015
Messages
10
I had the same issue originally. I had to add these loader tunables:
hw.pci.enable_msi = 1
hw.pci.enable_msix = 0

After that, the system boot and ran without issue.
Hope that helps.
 
Status
Not open for further replies.
Top