22.02 RELEASE Performance Regressions

[BitByteBit]

Further to the above, if the Atom C3758 does suffer from performance hit when using native ZFS encryption in Scale on Linux, is the Intel QAT chip onboard able to be used to offload the ZFS encrpytion / compression to mitigate?

 

[diskdiddler]

Looks like this thread linked is on the money @BitByteBit

ICP: Implement Larger Encrypt/Decrypt Chunks for Non-AVX Processors · Issue #10846 · openzfs/zfs

@AttilaFueloep has indicated that the original question's premise, a belief that slow performance on non-AVX processors was due to a non-accelerated GHASH, was in error. @AttilaFueloep indicates th...

github.com

They're working on it, the lack of AVX is holding the processor back when it comes to encrypted data sets.
Now I really want to see Lawrence run his benchmarks on SCALE but without encryption, be interesting to see if the performance issues are only due to that.

(I'm biased, I don't encrypt)

 

[markds]

@LawrenceSystems.... first off love the videos, please keep up the good work....

For the record I experienced the exact opposite... I found the local disk speed of Scale to be faster than Core. Significantly... so to others, if you have held off installing because of reports of slow downs, test your use case may actually be the opposite and these were on an encrypted dataset.

Command:
sync;fio --randrepeat=1 --ioengine=posixaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=readwrite --ramp_time=4

Core:
READ: bw=89.5MiB/s (93.9MB/s), 89.5MiB/s-89.5MiB/s (93.9MB/s-93.9MB/s), io=1686MiB (1768MB), run=18832-18832msec
WRITE: bw=89.4MiB/s (93.8MB/s), 89.4MiB/s-89.4MiB/s (93.8MB/s-93.8MB/s), io=1684MiB (1766MB), run=18832-18832msec

Scale:
READ: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=1504MiB (1577MB), run=11397-11397msec
WRITE: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=1501MiB (1574MB), run=11397-11397msec

Hardware:
HP Microserver Gen 8
Xeon E31260L
16GB Ram
4x 16TB EXOS Drives

 

[diskdiddler]

@LawrenceSystems.... first off love the videos, please keep up the good work....

For the record I experienced the exact opposite... I found the local disk speed of Scale to be faster than Core. Significantly... so to others, if you have held off installing because of reports of slow downs, test your use case may actually be the opposite.

Command:
sync;fio --randrepeat=1 --ioengine=posixaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=readwrite --ramp_time=4

Core:
READ: bw=89.5MiB/s (93.9MB/s), 89.5MiB/s-89.5MiB/s (93.9MB/s-93.9MB/s), io=1686MiB (1768MB), run=18832-18832msec
WRITE: bw=89.4MiB/s (93.8MB/s), 89.4MiB/s-89.4MiB/s (93.8MB/s-93.8MB/s), io=1684MiB (1766MB), run=18832-18832msec

Scale:
READ: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=1504MiB (1577MB), run=11397-11397msec
WRITE: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=1501MiB (1574MB), run=11397-11397msec

Hardware:
HP Microserver Gen 8
Xeon E31260L
16GB Ram
4x 16TB EXOS Drives

Do you know if these results are replicated over SMB and SCALE represents higher performance there?

 

[markds]

Do you know if these results are replicated over SMB and SCALE represents higher performance there?

most of my usage relates to nfs so didn't test Samba performance. The tests above were local to the server, as three were concerns over the speed of encrypted datasets.

 

[morganL]

Do you know if these results are replicated over SMB and SCALE represents higher performance there?

The Linux and FreeBSD OSes handle some scheduling issues differently.
At this stage, SMB single client is not faster, but SMB with many clients appears to be faster on SCALE.
Caching on TrueNAS CORE is better.... there are some issues to be worked on TrueNAS SCALE.
So, our advice is not to move to SCALE for performance, yet. Wait for more testing and bug fixes.

However, we do appreciate any testing that is done. Finding issues is how we'll get SCALE to a solid performance level.

 

[diskdiddler]

I will wait patiently.

Due to the processor, in my NAS lacking quicksync, and having a spare micro PC, I have started the migration of my little VM which runs docker containers for me, over to a new rig. Which will sadly reduce my needs for SCALE, that I was so eager for.

Still I may check SCALE out at some point if it becomes clearly superior.

 

[morganL]

I will wait patiently.

Due to the processor, in my NAS lacking quicksync, and having a spare micro PC, I have started the migration of my little VM which runs docker containers for me, over to a new rig. Which will sadly reduce my needs for SCALE, that I was so eager for.

Still I may check SCALE out at some point if it becomes clearly superior.

I'd expect by SCALE 22.02 U1 will have many performance issues resolved and U2 will have most....
I'm just warning against users migrating to SCALE purely with a goal of performance improvements. It will be hit or miss.
TrueNAS CORE has inherited 10 years of performance tweaking with FreeNAS/FreeBSD and ZFS.
TrueNAS SCALE is catching up on this dimension....

 

[morganL]

@LawrenceSystems Sice the performance issue seems to be coupled with a known OpenZFS encryption performance issue, I submitted a bug report. https://jira.ixsystems.com/browse/NAS-115648

 

[LawrenceSystems]

There are still some issues when using Intel Atom C3758 / 64G memory / Micron_5210 SSD with TrueNAS-SCALE-22.12.1 and I don't really have the time to test if this issue exists in TrueNAS Core. If I create a random large file in an encrypted data set using DD then copy over to another system via either SMB or rsync the transfer rate is about 73 MB/sec. If type to copy the file again it transfers at about 630 MB/Sec. If I do the same test in an unencrypted dataset I get the fast 620 MB/Sec when doing transfers first time. One other thing about this that boggles me is the write speed is consistently about 400 MB/Sec when copying files to the encrypted datasets. Not sure if this is something that will be fixed in the future or if I am doing something wrong. I am open to suggestions.

 

[morganL]

There are still some issues when using Intel Atom C3758 / 64G memory / Micron_5210 SSD with TrueNAS-SCALE-22.12.1 and I don't really have the time to test if this issue exists in TrueNAS Core. If I create a random large file in an encrypted data set using DD then copy over to another system via either SMB or rsync the transfer rate is about 73 MB/sec. If type to copy the file again it transfers at about 630 MB/Sec. If I do the same test in an unencrypted dataset I get the fast 620 MB/Sec when doing transfers first time. One other thing about this that boggles me is the write speed is consistently about 400 MB/Sec when copying files to the encrypted datasets. Not sure if this is something that will be fixed in the future or if I am doing something wrong. I am open to suggestions.

As you can imagine, our standard QA process tests for functional operation. We've now started performance testing for different use-cases. Linux encryption is very different from FreeBSD and so there are several issues to be resolved. More improvements are coming in SCALE 22.12.2 later this month.

With Micron 5210, there is also variability in QLC write speeds... the timing and size of different writes can have a significant impact. We try to use higher performance SSDs for baselines... performance is more reliable.