Very poor performance Scale vs Core on TrueNAS Mini XL+ also what is expected performance penalty for encryption?

[scoots]

Hi,

I recently purchased a TrueNAS Mini XL+ and wanted to compare the performance of CORE vs SCALE before getting everything setup. I removed the pre-installed Boot NVME storage, installed TrueNAS on a Kingston A400 SATA SSD, and put a Samsung 970 EVO Plus 2TB in the NVME slot on the motherboard and used that as the storage pool. No other drives installed at the moment.

After running benchmarks on CORE, I wiped everything and just installed SCALE (latest version) - and the performance of SCALE seems very poor - and I am not sure what is going on.

Also, in the benchmark numbers below I also compared encrypted vs non-encrypted performance, and saw about 100-240MB/s penalty on reads and writes.

Here are the results all in MB/s (using a 10GB connection) (if there is a slash, it is because I ran the benchmark again a day later, to get another data point).

TrueNAS CORE

2TB Samsung
Encrypted
128KB Record Size

Blackmagic 5GB
Write: 665 / 680
Read: 767 / 780

AJA 64GB
Write: 932 / 940
Read: 1035 / 1082

File Copy from Mac to NAS
Network: 800-950 (usually around 850)


2TB Samsung
NOT Encrypted
128K Record Size

Blackmagic 5GB
Write: 750 / 747
Read: 1000 / 998

AJA 64GB
Write: 920 / 950
Read: 1080 / 1076

File Copy from Mac to NAS
Network: 950-1000+ (usually around 1000)

TrueNAS SCALE

2TB Samsung
Encrypted
128KB Record Size

Blackmagic 5GB
Write: 329
Read: 66

AJA 64GB
Write: 434
Read: 460

File Copy from Mac to NAS
Network: 300-450 (usually around 350)

Note: I did not run benchmarks with a non-encrypted dataset because the results were startling and I wanted to post here to get help!

Also, I ran the ATTO benchmark and here is the comparison between the CORE encrypted dataset (first image) and the SCALE encrypted dataset:

Any help would be much appreciated!

 

[Samuel Tai]

This is expected Scale vs Core. Scale has had very little optimization, as the developers have focused primarily on correct functionality. Furthermore, ARC is limited to 50% of RAM in Scale, unlike Core, which can have ARC expand to ~80% of RAM, so Scale also suffers from less caching.

 

[scoots]

Thank you Samuel for responding, and explaining the differences in ARC between the two versions.

However, I wouldn't have expected that the differences between the two versions would be so dramatic.

That the write speed would be 50% and the read speeds would be less than 10%.

CORE
Blackmagic 5GB
Write: 665 ~ 680
Read: 767 ~ 780

SCALE
Blackmagic 5GB
Write: 329
Read: 66

The reason I was investigating Scale was because I saw there was a community guide for supporting Spotlight search (https://www.truenas.com/docs/scale/scaletutorials/communityrecommends/spotlightsupportsamba/), and I couldn't find anything on that for Core.

 

[Samuel Tai]

@Patrick M. Hausen has a good tutorial on Spotlight search on Core. Just search the forum for Spotlight and his name.

 

[Patrick M. Hausen]

@Patrick M. Hausen has a good tutorial on Spotlight search on Core. Just search the forum for Spotlight and his name.

Not really. As far as I know the result was that there is a problem with smbd communicating with Elastic that is still not fixed.

 

[anodos]

Not really. As far as I know the result was that there is a problem with smbd communicating with Elastic that is still not fixed.

Linux-isms in some string parsing functions. I can see about fixing for U5.

 

[Patrick M. Hausen]

@anodos

1st reaction: love you dude, great work!
2nd reaction: oh now, more work - got to write a resource ...

 

[scoots]

Thank you everyone for chiming in.

I was trying to make sense of the "Faster Searching from Mac Spotlight or Finder of Freenas Files" thread, however it is a bit technical for me. https://www.truenas.com/community/t...light-or-finder-of-freenas-files.81532/page-7

If I understand correctly, at the moment there isn't a solution for a Mac to search file contents on a SMB volume, for either Core or Scale. My interest would certainly be for a solution with Core (as the Scale performance (as shown above) is difficult to work with).

Or perhaps I have misunderstood, or maybe didn't phrase my question correctly. What I'm looking for Core is something like: "Search for Anything on your NAS Directly through Finder - new in DSM7 (tutorial)" by SpaceRex on YouTube. https://www.youtube.com/watch?v=TCLRp3KHV_M

 

[anodos]

@anodos

1st reaction: love you dude, great work!
2nd reaction: oh now, more work - got to write a resource ...

BTW, I'm working on our Samba 4.18 port now. This will be a good change I think.

 

[anodos]

Also somewhat relevant to this convo is that we added a kqueue / kevent based event backend to Samba in Core a while back (IIRC it gave about a 20% boost in ops / sec in torture tests) and reads / writes get submitted directly onto this (remove some levels of indirection) and handed off to kernel threads for I/O. In core we also eliminated xattr reads for basic ops (DOSATRIB reads), by using DOS flags returned in stat(2) output. This all has an impact on perceived and actual performance. SCALE is a different beast altogether (Linux / epoll, more xattr reads, and using io_uring module).

Core and SCALE also use different samba versions. ZFS also behaves differently in Linux than FreeBSD, and drivers are also of course different. So very apples-to-oranges comparison.

 

[scoots]

@anodos Thank you for the explanations. It all sounds very fascinating

I wanted to share more about my use-case, so that it doesn't just sound like I am benchmarking to try to poke holes in TrueNAS.

How I imagined to set up my Mini XL+ was to have a "fast as economically possible for 10G" 2TB NVME - in this case a Samsung 970 EVO PLUS, as a single volume to act as the first point of storage / editing for video files.

The goal is that after a day of shooting videos, I would plug in a fast external SSD drive to my Mac via Thunderbolt and then dump the video files to the NAS over a 10G connection. That's why I was keen to see the impact of non-encrypted file copy on Core (~1,000MB/s) compared to encrypted on Core (~850MB/s) vs Scale encrypted (~350MB/s). I was testing with copying some sample 10GB video files, and using the AJA 64GB test. As the video files get larger, the speed difference starts to be meaningful.

I would have a daily script to backup the contents of the 2TB NVME to a 6-drive RAID10 (and only after that delete the footage from the external SSD). The 6-drive RAID would also hold all of the finished video files, as well as act as a repository for all my photography, work, and other personal documents. That's where being able to search many TBs of files and their contents via Finder is needed.

Is the performance drop that I'm seeing from using Encryption normal? What is typically the bottleneck for that?

 

[anodos]

@anodos Thank you for the explanations. It all sounds very fascinating

I wanted to share more about my use-case, so that it doesn't just sound like I am benchmarking to try to poke holes in TrueNAS.

How I imagined to set up my Mini XL+ was to have a "fast as economically possible for 10G" 2TB NVME - in this case a Samsung 970 EVO PLUS, as a single volume to act as the first point of storage / editing for video files.

The goal is that after a day of shooting videos, I would plug in a fast external SSD drive to my Mac via Thunderbolt and then dump the video files to the NAS over a 10G connection. That's why I was keen to see the impact of non-encrypted file copy on Core (~1,000MB/s) compared to encrypted on Core (~850MB/s) vs Scale encrypted (~350MB/s). I was testing with copying some sample 10GB video files, and using the AJA 64GB test. As the video files get larger, the speed difference starts to be meaningful.

I would have a daily script to backup the contents of the 2TB NVME to a 6-drive RAID10 (and only after that delete the footage from the external SSD). The 6-drive RAID would also hold all of the finished video files, as well as act as a repository for all my photography, work, and other personal documents. That's where being able to search many TBs of files and their contents via Finder is needed.

Is the performance drop that I'm seeing from using Encryption normal? What is typically the bottleneck for that?

Which version of SCALE is this?

 

[scoots]

Which version of SCALE is this?

Latest version I believe TrueNAS-SCALE-22.12.1

 

[Antoshka]

I faced even worse performance degradation after the Core->Scale switch (up to 80% speed loss).
The same use case: speed from the encrypted pool is 160Mb/s instead of 850Mb/s unencrypted.
I'm using the custom system, not Mini XL+.

 

[el5]

Lawrence from Lawrence Systems on youtube mentioned his 10GbE speed degrades with encrypted datasets and might be linked to the decryption of the data which, according to him, runs only as a single-threaded process and kills performance, but encryption is fast because it's running as multi-threaded. He makes this observation in a few of his recent videos, but does not have a dedicated video on this. Unfortunately, he does not have a solution about this. He did observe the network performance degradation in truenas scale 22.12 and 23.10 with encrypted datasets.

 

[Dysonco]

Hi All,

Just jumping in here to give my 2 pennies worth.

I've recently converted an existing Core install to Scale and have no encryption enabled on any of the volumes, yet I'm still seeing a massive degradation in performance down from being able to fully saturate my 10Gbe in both read and write operations (usually around the 900+MBs) to maxing out at around 230MBs second read and maybe 300MBs write.

So in my case, its definitely not the encryption....

 

[alexmarkley]

Quick question. Is it possible your performance issues are related to mine? https://www.truenas.com/community/t...ance-characteristics-on-truenas-scale.112957/

I've been doing a deep-dive investigation into Samba performance issues on SCALE and I think I'm getting close to narrowing the issue down.