Tomaae
Dabbler
- Joined
- Feb 24, 2021
- Messages
- 12
Hello,
I'm looking if it is possible to get viable performace for application use. Problem here is that it loads many small files and loading take up to 5 times slower, which is up to several minutes. I dont expect same performace as local, but cutting current slowdown to half would be more the enough.
Network itself is not a problem, as any other load will easely saturate its capacity.
I'm looking to find what is actually causing the slowdown. I dont believe ZFS is a problem here as this is mostly read and NAS is used only by single person.
SMB could be the issue, but I did not find any good way to test similiar load synthetically.
Does anybody has experience with tuning truenas for small random reads?
I have already looked up everything I could find on these forums and adjusted SMB accordingly, which resulted in about 15% performance increase.
Hardware:
Everything except backup disk is set up in mirrors.
SMB aux parameters:
Test bellow is performed on mirrored m.2 665p both locally from shell and from windows pc over network. Not sure how helpful these are, since they look way too good even on client.
On NAS:
Client PC over SMB:
I'm looking if it is possible to get viable performace for application use. Problem here is that it loads many small files and loading take up to 5 times slower, which is up to several minutes. I dont expect same performace as local, but cutting current slowdown to half would be more the enough.
Network itself is not a problem, as any other load will easely saturate its capacity.
I'm looking to find what is actually causing the slowdown. I dont believe ZFS is a problem here as this is mostly read and NAS is used only by single person.
SMB could be the issue, but I did not find any good way to test similiar load synthetically.
Does anybody has experience with tuning truenas for small random reads?
I have already looked up everything I could find on these forums and adjusted SMB accordingly, which resulted in about 15% performance increase.
Hardware:
This is single user NAS, designed for data accessibility, safety and backup. 99% load is read, so Q/TLC and SSD write cache on cheaper SSD is never a problem.MB: Supermicro X11SCH-LN4F
CPU: i3-8300 CPU
RAM: 128GB ECC
DISKS:
2x WD Red SA500 2TB
2x Samsung SSD 860 EVO 2TB
1x Samsung SSD 860 QVO 2TB
2x INTEL SSD 665p 1TB
2x Crucial BX500 120GB (Boot)
CTRL: Build-in
LAN: Build-in dual i210
Everything except backup disk is set up in mirrors.
SMB aux parameters:
socket options = IPTOS_LOWDELAY TCP_NODELAY SO_RCVBUF=65536 SO_SNDBUF=65536
mangled names = illegal
store dos attributes = no
map archive = no
map hidden = no
map system = no
map readonly = no
use sendfile = yes
read raw = yes
write raw = yes
aio max threads = 100
aio read size = 1
aio write size = 1
allocation roundup size = 1048576
max xmit = 65535
getwd cache = yes
Test bellow is performed on mirrored m.2 665p both locally from shell and from windows pc over network. Not sure how helpful these are, since they look way too good even on client.
fio --bs=16k --direct=1 --directory . --gtod_reduce=1 --iodepth=32 --group_reporting --name=randrw --numjobs=12 --ramp_time=10 --runtime=60 --rw=randread --size=256M --time_based
On NAS:
randrw: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=32
...
fio-3.28
Starting 12 processes
Jobs: 6 (f=6): [_(3),E(1),_(1),E(1),r(6)][100.0%][r=13.6GiB/s][r=890k IOPS][eta 00m:00s]
randrw: (groupid=0, jobs=12): err= 0: pid=70972: Mon Oct 10 15:03:56 2022
read: IOPS=878k, BW=13.4GiB/s (14.4GB/s)(805GiB/60046msec)
bw ( MiB/s): min= 5536, max=27652, per=99.94%, avg=13715.35, stdev=300.27, samples=1428
iops : min=354312, max=1769751, avg=877777.79, stdev=19217.48, samples=1428
cpu : usr=3.33%, sys=29.10%, ctx=46259, majf=0, minf=1
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=52736524,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=13.4GiB/s (14.4GB/s), 13.4GiB/s-13.4GiB/s (14.4GB/s-14.4GB/s), io=805GiB (864GB), run=60046-60046msec
Client PC over SMB:
randrw: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=windowsaio, iodepth=32
...
fio-3.32
Starting 12 threads
Jobs: 12 (f=12): [r(12)][100.0%][r=112MiB/s][r=7177 IOPS][eta 00m:00s]
randrw: (groupid=0, jobs=12): err= 0: pid=28312: Mon Oct 10 14:52:00 2022
read: IOPS=7125, BW=111MiB/s (117MB/s)(6699MiB/60120msec)
bw ( KiB/s): min=98360, max=121470, per=100.00%, avg=114171.18, stdev=440.91, samples=1434
iops : min= 6140, max= 7588, avg=7131.71, stdev=27.58, samples=1434
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.4%, 32=99.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=428357,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=111MiB/s (117MB/s), 111MiB/s-111MiB/s (117MB/s-117MB/s), io=6699MiB (7024MB), run=60120-60120msec