Truenas Scale vs Core

[vineesh]

Hello all,
I recently configured a NAS server for media sharing purposes, then tried both scale and core on it.

Hardware configuration.​

Mobo : Asrock X570 Taichi
CPU : AMD 5600x
HDD : 8 X 12 TB Exos
RAM : Corsair Vengeance 128 GB RAM
NIC : Intel X540 - T2
SATA : Using Mobo's 8 inbuilt Sata Ports
L2ARC : 512TB WD SN850 Gen4

• Both Server and client use 9000 Jumbo frames.
• 8 X 12 TB HDD is in raidz2 config.

TrueNAS SCALE 22.02.1​

Iperf result

[ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec receiver

FIO result

Run status group 0 (all jobs): READ: bw=325MiB/s (341MB/s), 325MiB/s-325MiB/s (341MB/s-341MB/s), io=8908MiB (9340MB), run=27424-27424msec WRITE: bw=325MiB/s (340MB/s), 325MiB/s-325MiB/s (340MB/s-340MB/s), io=8902MiB (9335MB), run=27424-27424msec

Disk Speed Test

TrueNAS Core 13.0 RC1​

Iperf result

[ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 11.4 GBytes 9.79 Gbits/sec sender [ 5] 0.00-10.00 sec 11.4 GBytes 9.79 Gbits/sec receiver

FIO result

Run status group 0 (all jobs): READ: bw=340MiB/s (356MB/s), 340MiB/s-340MiB/s (356MB/s-356MB/s), io=23.7GiB(25.4GB), run=71407-71407msec WRITE: bw=340MiB/s (356MB/s), 340MiB/s-340MiB/s (356MB/s-356MB/s), io=23.7GiB(25.4GB), run=71407-71407msec

Disk Speed Test

TrueNAS Core 12.0 U8.1​

Iperf result

(icwnd/mss/irtt=625521/8948/1000) [ ID] Interval Transfer Bandwidth [ 1] 0.00-10.01 sec 11.1 GBytes 9.55 Gbits/sec

FIO result

Run status group 0 (all jobs): READ: bw=376MiB/s (395MB/s), 376MiB/s-376MiB/s (395MB/s-395MB/s), io=23.4GiB(25.2GB), run=63788-63788msec WRITE: bw=376MiB/s (395MB/s), 376MiB/s-376MiB/s (395MB/s-395MB/s), io=23.4GiB(25.2GB), run=63788-63788msec


Disk Speed Test

I expected better performance from Scale. Even though it shows better results in the Diskspeed test, caching performance was very poor. It also fell behind both versions of core in the FIO test. Truenas core 12 U8 was a clear winner in real-world performance.

However, I am disappointed in the lower FIO performance in all the cases, because hardware RAID would have given a better throughput IMHO.

Please let me know if I am doing something wrong here. Is there a way that I can improve my FIO read and write?

Thank you

NB : fio command used
sync;fio --randrepeat=1 --ioengine=posixaio --direct=1 --name=test --filename=test --bs=4k --size=50G --readwrite=readwrite --ramp_time=4

 

[Kris Moore]

We're already looking at some improvements that will land in a SCALE update shortly. CORE has had almost a decade of performance tweaks and tunings, SCALE is still pretty much out of box Linux + OpenZFS. We've already identified about half a dozen things we'll be changing to bring some of that performance back to par, or even beyond CORE soon.

 

[vineesh]

We're already looking at some improvements that will land in a SCALE update shortly. CORE has had almost a decade of performance tweaks and tunings, SCALE is still pretty much out of box Linux + OpenZFS. We've already identified about half a dozen things we'll be changing to bring some of that performance back to par, or even beyond CORE soon.

Thanks for the reply. Is there a roadmap for this feature set? Just curious.

 

[Kris Moore]

Tickets are internal still, but I'd expect them to start getting pushed into the public trackers for 22.02.2 and 22.02.3.

 

[MrGuvernment]

Tickets are internal still, but I'd expect them to start getting pushed into the public trackers for 22.02.2 and 22.02.3.

Just to bring this back curious if any of those improvements are in the current SCALE release?

Just doing all the digging in myself on performance to make sure I go the right route, but also do not want to reinstall TrueNAS in 6 months time or a year cause I made the wrong choice now :)

 

[Kris Moore]

Lots of perf stuff is landing in Bluefin right now. RC1 drops Nov 15th which is remarkably stable and will only get moreso as releases continue the polish and tuning process.

 

[MrGuvernment]

Lots of perf stuff is landing in Bluefin right now. RC1 drops Nov 15th which is remarkably stable and will only get moreso as releases continue the polish and tuning process.

Appreciate the info. Prob by the time I decided on my drive configs and continue reading the forums here until my head explodes (about half way there) then perhaps I give it a try!

 

[William Bravin]

Hello all

I read this thread and others and i understand the differences between core and scale.

I use my solution (currently core) to store and manage my media and documents. All works fine unless i screw it up

In this age of rediculous energy prices and my latest discovery of proxmox, i plan to migrate both of my truenas core solutions to vms on proxmox I'm currently using 4 servers (home assistant and opnsense) and i would like to save 2 servers. I also have a plex and emby jail at the moment. *still trying to decide which one to chose.

I personally do not need migrate to scale. ( i would not use the additional functionality or benefits)

my only concern is, if truenas core will eventually dissapear in favor of scale (you know one less app to maintain and all .

If so should i bite the bullet now and move scale and be donne with it?

Thank you for your help

 

[MrGuvernment]

Unless you need absolutely cutting edge performance that has gone into a decade of CORE, you could be fine with scale. I had the same question when doing my system, but I am going with Core for the simple fact is is tried, true and tested... for over a decade, so for me it is purely about reliability and support and knowledge already out there for CORE vs SCALE.

 

[William Bravin]

Unless you need absolutely cutting edge performance that has gone into a decade of CORE, you could be fine with scale. I had the same question when doing my system, but I am going with Core for the simple fact is is tried, true and tested... for over a decade, so for me it is purely about reliability and support and knowledge already out there for CORE vs SCALE.

Thank you so much for taking the ime to respond and sharing your experience. This helped me to confirm my decision to remain with core and to add truenas core as a VM to proxmox

Once again thank you

 

[greenot]

I was encouraged by the Bluefin info because we too went with Scale and are regretting it from a performance standpoint on NAS, but I have an odd experience.

For testing we deployed a Core server so we could do some side by side comparisons with NFS shares. We are using Proxmox to manage the VMs and have both a truenas Core file server and a Scale file server.

We confirmed that VMs that run stored on Core perform very well, getting around 6000 IOPS on fio tests. We moved the VM disk over to Angelfish, and for very poor performance. On the order of 900 IOPs.

Since we have a number of Scale deployments, we wanted to test the performance of Bluefin, so we then upgraded the Core server to Scale Bluefin release. The VM still got 6000 IOPS after upgrade to Blue fin. Very exciting!

However, we then upgraded the Anglefish storage server to Bluefin. This server never had Core on it, it was origianlly deployed with Scale. We used the GUI upgrade process, not ISO. Sadly, with this storage server, we still only got 900 IOPS even on Bluefin.

Any idea why a Core server upgraded to Scale/Bluefin would keep the performance benefits whereas a Scale server upgraded to Bluefin would not?