Poor write speed on striped mirrored pool

[TheStevenator]

Hello,


This is my first TrueNAS adventure. I created a striped mirror "raid 10" setup with 4x WD 5400rpm drives and am getting very low write speeds (would expect 4x read 2x write if no network bottleneck):

Code:

root@truenas[~]# zpool status
  pool: boot-pool
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          nvd0p2    ONLINE       0     0     0

errors: No known data errors

  pool: POOL_NAME
 state: ONLINE
 config:

        NAME                                            STATE     READ WRITE CKSUM
        POOL_NAME                               ONLINE       0     0 0
          mirror-0                                      ONLINE       0     0 0
            gptid/44d1d5bb-5ffb-11ec-a854-3ceceff155fe  ONLINE       0     0 0
            gptid/44df9ca7-5ffb-11ec-a854-3ceceff155fe  ONLINE   
          mirror-1                                     ONLINE       0     0 0
            gptid/44d1d5bb-5ffb-11ec-a854-3ceceff155fe  ONLINE       0     0 0
            gptid/44df9ca7-5ffb-11ec-a854-3ceceff155fe  ONLINE       0     0 0

Using crystaldiskmark I see consistent 87 MB/s read (I only have 1Gbps LAN, so that's not far from the maximum), but the max write speed is 25 MB/s (those numbers hold well for copying 20GB files back and forth as well). Even local copy (copying files on the store itself) goes no faster than this, so I am suspecting that it's a compression/sync/record size issue. I think my hardware is pretty strong? (ready to get told otherwise):

• Intel E-2124
• Supermicro X11SCL-IF
• 16 GB Supermicro DDR4 2666

I have the drive mounted on windows using NFS:

Code:

Local    Remote                                 Properties
-------------------------------------------------------------------------------
Z:       \\192.168.1.191\mnt\POOL\STORE  UID=1000, GID=1000
                                               rsize=131072, wsize=131072
                                               mount=soft, timeout=3.2
                                               retry=1, locking=no
                                               fileaccess=755, lang=ANSI
                                               casesensitive=no
                                               sec=sys

I would appreciate any pointers, thank you.

 

[Samuel Tai]

You need a SLOG, because your ZIL is on your pool, and causing double writes to your pool disks, first to the ZIL, and then to the data VDEVs.

 

[TheStevenator]

You need a SLOG, because your ZIL is on your pool, and causing double writes to your pool disks, first to the ZIL, and then to the data VDEVs.

That makes sense. Do you suggest that it can be a partition of the boot ssd or should I get a separate device?

 

[Samuel Tai]

A separate device. Optanes work very well as SLOGs, because they have tremendous write endurance, are very fast, and also have power protection, so writes can be saved for later replay back to your VDEVs if your system loses power.

 

[TheStevenator]

Ok, I guess it will take time to test out. I'll report after I get one. Cheers.

 

[Arwen]

Please list the exact model of each of your WD 5400 RPM drives.

Western Digital did something nasty, the WD Red line, (not the Red Plus or the Red Pro), are SMR type drives. SMR is Shingled Magnetic Recording and is SIGNIFICANTLY slower in writing more than the drive's cache can hold. Their is then a pause for the drive to flush it's cache to shingled sectors.

While I own a Seagate 8TB SMR drive, (aka Archive drive in Seagate's terminology), I was aware of the write slowness. Even though I am using it for backing up my regular pool, it does not mater to me if the backup takes all night or 2 days.

See this resource for detection of SMR drives:

Scripted Detection of SMR Drives

SMR technology is a timebomb waiting to go off in a ZFS file system. This quote from Slashdot... At absolute best, SMR disks underperform significantly in comparison to CMR disks; at their worst, they can fall flat on their face so badly that...

www.truenas.com

 

[TheStevenator]

Please list the exact model of each of your WD 5400 RPM drives.

I was aware of this, they are WD60EFZX (CMR).

 

[TheStevenator]

Removing sync doubles writes to 60 MB/s. Not necessarily leaving it like that, but the primary diagnosis seems to be correct. Seeing where else I could pick up bandwidth.

 

[Arwen]

I was aware of this, they are WD60EFZX (CMR).

Excellent!

Just as a note, when you mention WD drives or even WD Red drives, you should probably mention that the are either Red Pluses, Pros or CMR. Lots of us in the forums are still very annoyed about this issue. Not just because of the way it was original pushed out by WD. But, TrueNAS users are STILL getting bit by it. So no harm in checking.

 

[TheStevenator]

I copied over a blu-ray of Apollo 13 locally using dd:

Code:

root@truenas[~]# dd if=/mnt/POOL/DATASTORE/apollo13.mp4 of=/root/copy/a13.mp4
3771032+0 records in
3771032+0 records out
1930768384 bytes transferred in 12.77532 secs (151106525 bytes/sec)
root@truenas[~]# dd if=/mnt/POOL/DATASTORE/apollo13.mp4 of=/root/copy/a13.mp4
3771032+0 records in
3771032+0 records out
1930768384 bytes transferred in 12.117289 secs (159339970 bytes/sec)

Both of those #s are still faster than the network NFS performance with no sync (~85 MB/s read, ~65 MB/s write). I did test my network to/from with iperf and it's fine (950 Mbps avg).

I guess I have two questions:

1) What's could the missing network performance be from?

2) Why is the "raid 10" equal in local read/write, shouldn't it be ~4x read, ~2x write compared to single disk (~120 MB/s).

 

[Samuel Tai]

1) What's could the missing network performance be from?

2) Why is the "raid 10" equal in read/write, shouldn't it be ~4x read, ~2x write compared to single disk (~120 MB/s).

To answer 1):
By default, TrueNAS isn't well-tuned for best performance. For a 1G network, I get good results enabling autotune under System->Advanced and rebooting, and also setting my TCP congestion control algorithm to CUBIC.

TCP congestion control algorithm selection in TrueNAS Core/FreeNAS

Networking-minded TrueNAS Core/FreeNAS admins may need to change the TCP congestion control algorithm for their specific installations from the default TCP new Reno to achieve the best performance. (For reference, see http://intronetworks.cs.luc.edu/current/html/newtcps.html.) There are several...

www.truenas.com

To answer 2):
dd isn't a good test of read/write, because you're mostly exercising your ARC. fio will provide a truer picture of your pool's raw performance.

 

[TheStevenator]

Code:

1) Thanks, do you mind specifying the exact syntax for loading the cubic module? As in, where does one use the "cc_cubic_load="YES""?

I found the System -> tunables, sorry I have no FreeBSD experience.

2) Is that still the case with a 2+ GB file? It's well above any cache size, it did literally take 12.7s to copy the movie over.

 

[Samuel Tai]

1) Thanks, do you mind specifying the exact syntax for loading the cubic module? As in, where does one use the "cc_cubic_load="YES""?

Create a LOADER type tunable with variable cc_cubic_load and value YES. Reboot to have it take effect.
Create a SYSCTL type tunable with variable net.inet.tcp.cc.algorithm and value cubic. This takes effect immediately, but only if the previous LOADER tunable is in place.

 

[TheStevenator]

No significant improvement using cubic over default (newreno), still ~85/65 R/W. Working on fio testing.

 

[TheStevenator]

I think I did the correct thing with fio, checking what the uncached direct-to-pool read/write is?

Code:

$ zfs create POOL/DATASTORE/fio
$ zfs set primarycache=none POOL/DATASTORE/fio
$fio --directory=/mnt/POOL/DATASTORE/fio --name=async_uncached_write --rw=write --bs=1M --size=10G


async_uncached_seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1

..

Run status group 0 (all jobs):
  WRITE: bw=424MiB/s (445MB/s), 424MiB/s-424MiB/s (445MB/s-445MB/s)
 
$fio --directory=/mnt/POOL/DATASTORE/fio --name=async_uncached_read --rw=read --bs=1M --size=10G

async_uncached_seqread: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1

...

Run status group 0 (all jobs):
   READ: bw=291MiB/s (306MB/s), 291MiB/s-291MiB/s (306MB/s-306MB/s), io=10.0GiB (10.7GB), run=35143-35143msec

Those values are much higher, but they also seem kind of goofy. The read speed for these drives in "raid 10" seems too low and should be closer to 600 MB/s (4x single), and the write speed seems much too high and should be closer to 250 MB/s (2x single). I'm not sure how to interpret these results.

 

[TheStevenator]

SMB mount with same pool configuration gives full network read/write performance 112 MB/s. So something is funny with how NFS is configured/implemented on my windows system. I will test NFS/SMB on an arch linux machine shortly.