Slow write performance. 1 disk in RAIDZ1 100% busy during writes

[esalts]

Hi all,

I'm having some performance issues with my machine. I have 3 4TB WD Red drives in my build in a RAIDZ1 configuration. I've noticed writes are quite slow. When writing a file, one of the disks always seem to be pinned at 100% busy according to the reporting tab of TrueNAS.

I did a long S.M.A.R.T test on the disk in question and it passed, but I'm wondering if this drive is failing and causing the poor performance?

To rule out network issues I ran a iperf3 test, and am able to get gigabit speeds (I only have a gigabit NIC, so this checks out):

Code:

%
./iperf3 -c nas.local
Connecting to host nas.local, port 5201
[  6] local 172.16.1.225 port 50429 connected to 172.16.1.154 port 5201
[ ID] Interval           Transfer     Bandwidth
[  6]   0.00-1.00   sec  18.0 MBytes   151 Mbits/sec                 
[  6]   1.00-2.00   sec  16.6 MBytes   139 Mbits/sec                 
[  6]   2.00-3.00   sec  18.3 MBytes   153 Mbits/sec                 
[  6]   3.00-4.00   sec  17.2 MBytes   144 Mbits/sec                 
[  6]   4.00-5.00   sec  17.1 MBytes   143 Mbits/sec                 
[  6]   5.00-6.00   sec  17.9 MBytes   150 Mbits/sec                 
[  6]   6.00-7.00   sec  17.9 MBytes   151 Mbits/sec                 
[  6]   7.00-8.00   sec  17.5 MBytes   146 Mbits/sec                 
[  6]   8.00-9.00   sec  16.7 MBytes   140 Mbits/sec                 
[  6]   9.00-10.00  sec  17.8 MBytes   150 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  6]   0.00-10.00  sec   175 MBytes   147 Mbits/sec                  sender
[  6]   0.00-10.00  sec   175 MBytes   146 Mbits/sec                  receiver

I also ran a local transfer speed test using dd:

Code:

root@nas[/mnt/Data/test]# dd if=/dev/zero of=tmp.dat bs=2048k count=50k
^C4636+0 records in
4635+0 records out
9720299520 bytes transferred in 873.465399 secs (11128431 bytes/sec)

The test was taking a very long time so I aborted, and as you can see the transfer speed is quite low.

My gut is telling me that the disk in question is the root of my issues, and just needs to be replaced. I wanted to get some other opinions though before purchasing a new one. Any help would be much appreciated!

 

[esalts]

Also, here's an example of the metrics when doing a transfer. As you can see one of the disks is pinned at 100% busy, while the others are fine.

 

[somethingweird]

Are the drives SMR?

got hardware specs to share for everyone?

 

[esalts]

Are the drives SMR?

got hardware specs to share for everyone?

I ran this SMR check script and it turns out ada0 (the one with the issue) is SMR. The other two drives didn't come up, and I believe they are CMR. I suppose that is the likely culprit then, hey?

As for other specs:

CPU: Intel Core i3-9100 3.6GHZ
Motherboard: Asus Prime H310I-PLUS R2.0/CSM Mini ITX LGA1151
Memory: 16GB DDR4-3200 (non-ECC)

 

[esalts]

Here's the link to the SMR check script: https://github.com/basilhendroff/truenas-smr-check

 

[somethingweird]

I suppose that is the likely culprit then, hey?

Yup... SMR and ZFS aren't compatible.

 

[somethingweird]

side note - Realtek NIC on motherboard *might* give you problems also.

 

[esalts]

Yup... SMR and ZFS aren't compatible.

Makes sense...definitely remember hearing that but I guess I didn't realize it would have this much of an impact on performance. I'll swap it with another CMR drive.

Please note - Realtek NIC on motherboard *might* give you problems also.

Yeah I heard there are issues with these. However I seem to be able to saturate the gigabit connection when doing bandwidth tests so I think I'm fine there. If I'm still having issues with performance after swapping the drive I'll do some more network testing and perhaps try an Intel NIC.

Thanks a ton for the help!