SOLVED Sawtooth performance while transfering files

[deksor]

Hello,
I've built my first NAS few weeks ago and I noticed a very bizarre and annoying issue while copying large amounts of data (tens of gigs) to my NAS.
The initial burst is at full speed (I have 2.5Gbe so ~220MB/s), then it drops ... all the way down to kilobytes per second. Sometimes I am even prompted with a message telling me there was a write error. The transfer freezes for some time, then resumes at full speed, and then the same thing happens, over and over.

Specs are :

• Asus ROG STRIX B550-I
• AMD Ryzen 5600G
• 32GB RAM
• Network card is the integrated Intel I225-V 2.5Gb
• 2xSeagate IronWolf 8TB
• 1xToshiba N300 8TB
• 1xWestern Digital Red 8TB
• 1xCorsair Force MP510 (256GB pcie gen 3 ssd)
• Boot drive is a Kingston A400 (cheap sata 120GB SSD)

Those 4 drives are set in a RaidZ2 pool
Since the board only has 4 sata ports, one of those drives is connected to a sata controller (J-Micron JMB582) on the m.2 port

The nvme SSD is used as a log drive.

When the problem occurs, i checked the RAM, it was far from being full (~13GB of RAM used for the zfs cache), CPU is barely used, drives don't seem to be fully used, nothing seems to be overheating, I have no alerts and nothing relevant in /var/log/messages (no specific message when this occurs).

I have checked the bandwidth between my PC and the NAS using iperf and I do indeed have the full speed, even when the problem occurs so I think network is out of the equation here.
I use SMB for file transfer.

Write speed : 4.4GB/sec

Code:

root@truenas[~]# dd if=/dev/zero of=/mnt/Pool/Shared/test bs=1M count=100000
100000+0 records in
100000+0 records out
104857600000 bytes transferred in 22.082899 secs (4748362139 bytes/sec)

Read speed : 19GB/sec

Code:

root@truenas[~]# dd of=/dev/zero if=/mnt/Pool/Shared/test bs=1M count=100000
100000+0 records in
100000+0 records out
104857600000 bytes transferred in 4.937646 secs (21236353837 bytes/sec)

I think I get reasonable performance for my needs. The "Shared" Dataset is compressed, but even there, the CPU is barely used (9% and 60°C).

Now there's something I noticed here while running this test. I'm not 100% sure, but I think the slowdowns only occur while writing using samba. While I ran DD, I had no issues at all. If I tried to write 100GB worth of data from my main PC it'd have take way more than 22 seconds, it'd have taken me 30 minutes.

Do you think this issue is samba related ? I also noticed while doing my tests that when the transfert is freezing, anything related to HDDs on the NAS' interface is freezing too. Checking pools ? Frozen. Trying to watch the HDDs' reports ? You'll have to wait until it unfreezes.
Trying to do the above DD command while the transfer is freezing ? You'll have to wait as well.

Can anyone help me ?
Any help is appreciated

 

[mav@]

Neither 19GB/s nor even 4.4GB/s are realistic for the pool of 4 HDDs, especially in RAIDZ2, where half of bandwidth goes to parity. Please disable compression on the dataset when using `dd if=/dev/zero ...`. For pool of 4 HDDs I'd use either RAIDZ1 (for better capacity and possibly throughput) or mirror (for better IOPS).

 

[c77dk]

Which version of TrueNAS?

The raidz2 on 4 drives is quite ok to run, when the downsides are accepted (one of my prod. servers run this setup, to allow for more time to get on site in case of disk failure).

Regarding your specific issues - try to monitor the disk activity with

Code:

gstat

- just to see how busy your disks are.
What are the sync settings for the dataset? And just to be 100% sure - no dedup has been enabled?

 

[deksor]

OK, I'll redo the benchmarks in an uncompressed dataset soon.

My initial plan was to use 5 drives, but I only had 4 when I started. Now I'll have to destroy the current pool and recreate a new one with the 5th drive, but that's my mistake. I thought I could expand it afterwards, but apparently truenas doesn't do that yet. Anyways that's not the problem here.

I use TrueNAS-12.0-U8.1.

Your mention of dedup has caught my attention, because it is enabled in that dataset indeed. I tried to upload a 9GB file into another of my dataset without dedup and it went smoothly. I've tried again in the main dataset and the same problem occurs. So I went into the settings to disable it. We'll see soon if it is indeed the cause of trouble (and by the way the sync settings are just "standard").

 

[c77dk]

dedup sounds like a great feature, but in reality it demands some rather hefty hardware (lots of RAM and DDT on sVDEVs - Optane class) to make it work smoothly. @Stilez wrote a nice piece about the topic.

 

[rvassar]

Can you provide the exact model for the 8Tb WD Red? I don't believe there are any SMR drives in the 8Tb range yet, but...

 

[deksor]

So I can confirm it was dedup that was the cause of the issue.

The HDD is a WDC WD80EFBX-68AZZN0.

I will post the other benchmarks soon for completeness, but I think the problem is solved.
Thanks everyone !