[TrueNAS 13 Performance] Just upgraded from 12-8.1 to 13.0 and have found significant performance gains - multi-CPU System - 10GbE - MacOS

[SpaceRex]

Hi All,

Figured I would share my experience upgrading from 12-8.1 to 13.0 (release) in terms of performance. I have done minimal testing to see if anything weird has broken but I have been able to my normal workflow (macOS, SMB, Final Cut Pro, with 10 gig) and have had no errors or issues. Doing some basic performance testing using macOS connecting to the trueNAS server over 10 gig I have been able to get a pretty solid bump in both read and write speeds. The following tests were not scientific, but repeated a few times and I can say after upgrading to 13 I have gotten significantly better sequential throughput then I had ever had on 12.

Black Magic Speed test: 12-8.1

Blackmagic speed test 13.0

System: (r630)
Dual Intel(R) Xeon(R) CPU E5-2628 v3 @ 2.50GHz
128 gig ram
7 SATA (cheap) SSD's in Z1
10GbE (W/ Jumbo frames)

 

[Samuel Tai]

This is likely due to Samba 4.15 enabling SMB multi-channel by default.

Samba 4.15.0 - Release Notes

 

[SpaceRex]

This is likely due to Samba 4.15 enabling SMB multi-channel by default.

Samba 4.15.0 - Release Notes

This is with auxiliary SMB parameters "server multi channel support = no" (I had tested SMB multichannel in the past and was not good for video editing) so SMB multichannel should not have been active.

I have attributed this to the freeBSD upgrade that was supposed to perform tasks with NUMA significantly faster but I am no expert.

 

[anodos]

This is likely due to Samba 4.15 enabling SMB multi-channel by default.

Samba 4.15.0 - Release Notes

We have multichannel disabled by default in Core / SCALE.

 

[HoneyBadger]

Intel P-states work correctly in FreeBSD 13, and with the nature of SMB to be single-threaded this could be resulting in the active cores turboing up and giving you the performance boost you're seeing.

That's a pretty solid jump especially in the sequential writes.

 

[anodos]

Intel P-states work correctly in FreeBSD 13, and with the nature of SMB to be single-threaded this could be resulting in the active cores turboing up and giving you the performance boost you're seeing.

That's a pretty solid jump especially in the sequential writes.

We use kernel aio(4) for samba in 12/13. IIRC there were changes related to this as well.

 

[SpaceRex]

We have multichannel disabled by default in Core / SCALE.

I thought that this was changed in 13 with the new SMB 4.15 now taking it out of alpha

 

[anodos]

I thought that this was changed in 13 with the new SMB 4.15 now taking it out of alpha

Our default settings are not always the same as upstream samba settings.

 

[SpaceRex]

Our default settings are not always the same as upstream samba settings.

Oh interesting, did not even thing about that. Do you know the reason for it? I have found that its slow on random reads sometimes and was wondering if its that, or a stability issue

 

[SpaceRex]

Intel P-states work correctly in FreeBSD 13, and with the nature of SMB to be single-threaded this could be resulting in the active cores turboing up and giving you the performance boost you're seeing.

That's a pretty solid jump especially in the sequential writes.

Yeah I thought so as well. I knew it was going to be faster (had heard at least) but was really impressed by how much faster. I was assuming the reason I was not getting "full" 10 gig was MacOS.

By the way I had read these forums so many times and this is my first time posting and its funny because I have seen all three of y'alls avatars hundreds of times over the years, always with great info!

 

[mav@]

In addition to FreeBSD, the performance improvements may also be caused by new OpenZFS version, including the work of your obedient servant: https://wiki.freebsd.org/DevSummit/202111?action=AttachFile&do=get&target=FreeBSD,+ZFS+and+iSCSI,+or+one+year+of+TrueNAS+development.pdf . ;)

 

[anodos]

In addition to FreeBSD, the performance improvements may also be caused by new OpenZFS version, including the work of your obedient servant: https://wiki.freebsd.org/DevSummit/202111?action=AttachFile&do=get&target=FreeBSD,+ZFS+and+iSCSI,+or+one+year+of+TrueNAS+development.pdf . ;)

Oh yeah, forgot to mention that. Mav is also constantly working to make everything faster and better :)

 

[Kevo]

I can't verify any total bandwidth improvement as my server is only gigabit ethernet and could already saturate the network. I can however, say that initial connection to the server is now much faster. It only takes a second or so to get a connection whereas in 12 it could take a rather long time to get a connection. I used to bring up the connection window and just move it to my other screen to work on something else while I waited for it to list the shares. Now I get the share list and click a share and it mounts immediately. It's really nice.

 

[Christian K]

In addition to FreeBSD, the performance improvements may also be caused by new OpenZFS version, including the work of your obedient servant: https://wiki.freebsd.org/DevSummit/202111?action=AttachFile&do=get&target=FreeBSD,+ZFS+and+iSCSI,+or+one+year+of+TrueNAS+development.pdf . ;)

That is very impressive.
I wish there was also ongoing work on improving sequential (large file) read performance from a pool with multiple vdevs with mirrored disks per vdev.
This is my use case and I find read performance not so impressive.

 

[mav@]

@Christian K There is. :) Wide HDD pools require large speculative prefetch distance to reach full speeds. You may see TrueNAS autotuner setting vfs.zfs.zfetch.max_distance to 32MB instead of default 8MB. There is also ongoing work on that direction to make it adaptively reach even further to 64MB: https://github.com/openzfs/zfs/pull/13452 . Here are my other ZFS PRs if curios about more ongoing work: https://github.com/openzfs/zfs/pulls?q=is:pr+author:amotin+ .

 

[Christian K]

Yes I am aware of vfs.zfs.zfetch.max_distance and I had already set it to 32MB. To be honest I measured neglectable sequential read performance gains compared to the stock default 8MB.

 

[mav@]

@Christian K What is your pool and what exactly is "not so impressive"?

 

[Christian K]

@Christian K What is your pool and what exactly is "not so impressive"?

I do not have access to my system at this time, but let me provide a brief summary.

The setup is x86_64 consumer hardware (AMD Ryzen system) with 16GB ECC DDR4 memory and 4 on-board SATA ports, with a Seagate Ironwolf 4TB hard disk connected to each.10 GBe ethernet NIC.
Running "dd" I can read ~170 MB/sec sequentially from each of the 4 drives in parallel.
Server and client are connected as 10GB Ethernet. netperf TCP tests showed thorougput well above 900 MB/sec

I am sharing to a Linux client the only configured pool via NFS4 and temporarily unsecured FTP (for testing purposes). NFS sequential large file read performance is pretty consitent at ~240 MB/sec , while plain FTP actually manages ~270 MB/sec. This is what I consider "not so impressive" in comparison to the theoretical read limit of 4 x 170 MB/sec.
Copy target is /dev/null (or a RAM disk) to that the client storage does not pose a bottleneck.

I have all 4 drives assigned to one ZFS pool. The pool is comprised of two vdevs, with two mirrored disks in each vdev, for a total of 4 disks.
The basic pool and vdev settings are pretty much default. Compression (lz4 IIRC) is enabled, no dedup. kernel autotuning is enabled.

 

[mav@]

@Christian K I can only say that it is not trivial to reach the 4 x 170 MB/sec speeds in 2xmirror configuration for HDDs. Since disks in a mirror store the same data, if read interleave is not large enough, it may end up that while one disk is reading, another is flying head over the same data. There is a logic in mirror implementation to improve that, but I don't think it is perfect. For SSD it obviously should work better.

Also, I guess 170MB/s may be the maximum speed on outer tracks of the HDDs. On inner tracks it may be much lower.