SAMBA speeds dropped by half going from TruenasCore to TrueNAS-SCALE-22.12.0

[Gagik]

Hello,

I have a problem that I cannot figure out, and I wanted to ask you guys. I built TruenasCore on a system with 4 x NVME and 8 x SSD drives and created a pool with each drive. I shared it out over Samba, and I have a 40GB network card in both the Truenas system and the Windows PC. My speeds while doing network tests to the pool over Samba share were 2,000 MB/s for both read and write, using the AJA system test and Blackmagic. Even copying files to and from the system was getting 2k MB/s non-stop. However, I decided to give Scale a test with the exact same setup and pool, and my speeds are only 900 MB/s for both read and write. Same system, just Scale instead of Core. The only thing I can see that's different is that Core uses Samba version 4.15.11, and Scale uses Samba version 4.17.2. Any help would be appreciated.

Thank you.

 

[anodos]

> The only thing I can see that's different is that Core uses Samba version 4.15.11, and Scale uses Samba version 4.17.2. Any help would be appreciated.

Core is FreeBSD-based. We have done some fairly heavy modifications to it (using a kevent-based event backend in Samba and FreeBSD kernel AIO). 22.12 is Linux, epool, and io_uring. Very different technologies. Do note that SCALE will also use less of the RAM for ZFS ARC.

 

[morganL]

Hello,

I have a problem that I cannot figure out, and I wanted to ask you guys. I built TruenasCore on a system with 4 x NVME and 8 x SSD drives and created a pool with each drive. I shared it out over Samba, and I have a 40GB network card in both the Truenas system and the Windows PC. My speeds while doing network tests to the pool over Samba share were 2,000 MB/s for both read and write, using the AJA system test and Blackmagic. Even copying files to and from the system was getting 2k MB/s non-stop. However, I decided to give Scale a test with the exact same setup and pool, and my speeds are only 900 MB/s for both read and write. Same system, just Scale instead of Core. The only thing I can see that's different is that Core uses Samba version 4.15.11, and Scale uses Samba version 4.17.2. Any help would be appreciated.

Thank you.

I'd like to see full system specs.
Any encryption being used?

A pool with each drive... so 12 pools? (that's unusual.. unlikely we have tested)

When running on CORE.. do you know whta cache hit rate you were seeing for these benchmarks?

 

[Gagik]

I'd like to see full system specs.
Any encryption being used?

A pool with each drive... so 12 pools? (that's unusual.. unlikely we have tested)

When running on CORE.. do you know whta cache hit rate you were seeing for these benchmarks?

Thanks for the feedback from you guys.

so what I have is very simple two individual pools of raidz and have tried raidz2 too.

8 x 2TB SSD drivers in a raidz

4 x Nvme in a raidz

As simple as that and just shared out over samba no encryption nothing at all. Booth pools shared over samba both mounted individually on core I get 2k MB's and scale 900MB's

System is a 16 core AMD 5950x with 128GB of Ram. very simple system just to test Truenas.

So is there no way to modify the settings for samba in scale?

So on a regular linux setup Rocky, CentOS, Redhat with samba export this is what I usually have and tweaked for max performance and I get very good speeds.


[global]

workgroup = WORKGROUP
netbios name = SAMBA
server string = Samba Server %v
map to guest = Bad User
log file = /var/log/samba/log.%m
max log size = 50
read raw = yes
write raw = yes
large readwrite = yes
security = user
# min receivefile size = 16384
use sendfile = true
use receivefile = true

#Socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
min receivefile size = 16384
aio read size = 16384
aio write size = 16384
#aio write behind = true





TCP Buffers
# add to /etc/sysctl.conf
# allow testing with 2GB buffers
net.core.rmem_max = 2147483647
net.core.wmem_max = 2147483647
# allow auto-tuning up to 2GB buffers
net.ipv4.tcp_rmem = 4096 87380 2147483647
net.ipv4.tcp_wmem = 4096 65536 2147483647

 

[morganL]

Can you describe the testing config. What is the active dataset size being used?

It would be useful to see the iperf network performance comparison..... since the performance is so evenly reduced. FreeBSD and Linux have different TCP algorithms. Generally, I thought Linux was superior, but perhaps your situation is not?

The major performance issue still in SCALE is that only half the RAM is used for ARC. This can have a big impact if the workload is mostly fitting in RAM. This is less of an issue if the RAM is being used for VMs and Apps.

If the workload is mostly accessing the storage drives, then SCALE generally performs a little better.

Our advice for most users, is that for a storage-only workload, CORE is more mature and has had more performance issues worked out. It should be expected to perform a little better. If that performance is critical, stay with CORE.

That being said, SCALE has better VMs and Apps... and performance is catching up.