Hello,
this is my first post as i am very lost in how to proceed with my DIY NAS, running TrueNAS-SCALE-23.10.1
Im seeing very slow upload speeds, getting absolutely stuck at like 50% when copying a big 24GB file from my PC to my NAS
This is the network overview for copying this single file
(You see it nearly stops inbetween but CPU and RAM are nearly not utilized)
(File copy process needing the whole range where i put the green rectangle --> My File Explorer is like stuck during this phase)
I am absolutely not a guy with big knowledge about all of this zfs-magic, just trying to configure a OK-budget nas with a fitting HBA to have full bandwith
Maybe you tell me that my drives are completely wrong for this application or something but by now i have no clue where a bottleneck could be
Sadly i do not have any spare drives laying around but im willing to rip out drives one by one (hoping my pool stays safe) and trying to individually check disks if this is necessary
System
HBA running in IT-Mode, all 6 Drives are attached to this HBA
fio test showing slow iops --> Write IOPS=16
fio test, with variation of pool sync setting
this is my first post as i am very lost in how to proceed with my DIY NAS, running TrueNAS-SCALE-23.10.1
Im seeing very slow upload speeds, getting absolutely stuck at like 50% when copying a big 24GB file from my PC to my NAS
This is the network overview for copying this single file
(You see it nearly stops inbetween but CPU and RAM are nearly not utilized)
(File copy process needing the whole range where i put the green rectangle --> My File Explorer is like stuck during this phase)
- Very detailed System Information and ran tests below.
- Also with help over at another resource i already tested my Network speed with iperf3 which seems to be having full 1Gbit-Performance From and to my PC vs NAS
- I am only using 6 identical SSDs in a 3xMirror + 3xMirror configuration (6x Crucial BX500 2TB SSDs)
- The last thing i was not able to check for 100% is each single drive individually for slow performance
- Is it even somehting related to zfs and my pool or do i have to check some bios settings etc?
- I am uploading my files via NFS but the same thing already tested with SMB and using another client (Both of my PCs running Fedora Workstation)
- No RAID settings activated in my BIOS settings
I am absolutely not a guy with big knowledge about all of this zfs-magic, just trying to configure a OK-budget nas with a fitting HBA to have full bandwith
Maybe you tell me that my drives are completely wrong for this application or something but by now i have no clue where a bottleneck could be
Sadly i do not have any spare drives laying around but im willing to rip out drives one by one (hoping my pool stays safe) and trying to individually check disks if this is necessary
System
Code:
+------------------------+------------------------------------------+ | version | TrueNAS-SCALE-23.10.1 | | buildtime | 2023-12-18T19:47:08+00:00 | | hostname | truenas | | physmem | 33521528832 | | model | Intel(R) Core(TM) i3-9100F CPU @ 3.60GHz | | cores | 4 | | physical_cores | 4 | | loadavg | 0.72119140625 | | | 6.57666015625 | | | 5.35693359375 | | uptime | 1 day, 2:51:12.641773 | | uptime_seconds | 96672.641772554 | | system_serial | System Serial Number | | system_product | P11C-I Series | | system_product_version | 4201 | | license | <null> | | boottime | 2024-01-14T13:22:39+00:00 | | datetime | 2024-01-15T16:13:51+00:00 | | birthday | <null> | | timezone | Europe/Berlin | | system_manufacturer | ASUSTeK COMPUTER INC. | | ecc_memory | false | +------------------------+------------------------------------------+ root@truenas[~]# lspci 00:00.0 Host bridge: Intel Corporation 8th Gen Core 4-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 08) 00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 08) 00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model 00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10) 00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) 00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10) 00:15.0 Serial bus controller: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10) 00:15.1 Serial bus controller: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 (rev 10) 00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10) 00:16.4 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller #2 (rev 10) 00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10) 00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0) 00:1c.1 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #2 (rev f0) 00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0) 00:1e.0 Communication controller: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller (rev 10) 00:1f.0 ISA bridge: Intel Corporation Device a30a (rev 10) 00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10) 00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10) 01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) 02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 04:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04) 05:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
HBA running in IT-Mode, all 6 Drives are attached to this HBA
Code:
LSI 9207-8i SAS2308 6G SAS HBA IT Mode
fio test showing slow iops --> Write IOPS=16
Code:
root@truenas[/mnt/tank/tank]# fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=60s --readwrite=write --name=testfile
testfile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.33
Starting 1 process
testfile: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [W(1)][1.1%][w=7168KiB/s][w=7 IOPS][eta 01h:39m:56s]
testfile: (groupid=0, jobs=1): err= 0: pid=1106473: Thu Jan 18 07:03:10 2024
write: IOPS=16, BW=17.0MiB/s (17.8MB/s)(1052MiB/62031msec); 0 zone resets
bw ( KiB/s): min= 2048, max=264721, per=100.00%, avg=17652.42, stdev=46503.11, samples=122
iops : min= 2, max= 258, avg=17.23, stdev=45.39, samples=122
cpu : usr=0.08%, sys=0.81%, ctx=1228, majf=0, minf=37
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1052,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=17.0MiB/s (17.8MB/s), 17.0MiB/s-17.0MiB/s (17.8MB/s-17.8MB/s), io=1052MiB (1103MB), run=62031-62031msec
root@truenas[/mnt/tank/tank]# fio test, with variation of pool sync setting
Code:
root@truenas[~]# zfs set sync=disabled tank
root@truenas[~]# zfs get sync tank
NAME PROPERTY VALUE SOURCE
tank sync disabled local
root@truenas[~]# fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=60s --readwrite=write --name=testfile
root@truenas[~]# cd /mnt/tank/tank
root@truenas[/mnt/tank/tank]# fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=60s --readwrite=write --name=testfile
testfile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][0.4%][w=5125KiB/s][w=5 IOPS][eta 05h:28m:05s]
testfile: (groupid=0, jobs=1): err= 0: pid=33007: Sat Jan 20 14:57:29 2024
write: IOPS=5, BW=5373KiB/s (5502kB/s)(370MiB/70513msec); 0 zone resets
bw ( KiB/s): min= 2048, max=10240, per=99.79%, avg=5362.89, stdev=1581.56, samples=141
iops : min= 2, max= 10, avg= 5.23, stdev= 1.55, samples=141
cpu : usr=0.04%, sys=0.26%, ctx=413, majf=0, minf=38
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,370,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=5373KiB/s (5502kB/s), 5373KiB/s-5373KiB/s (5502kB/s-5502kB/s), io=370MiB (388MB), run=70513-70513msec
root@truenas[/mnt/tank/tank]#
