TrueNAS Write Speed Slow but Read Speed Normal

[nsking]

Hello everyone, I have encountered some problems while using TrueNAS. The write speed of my TrueNAS is extremely slow, but the read speed is completely normal.

write speed:

root@truenas[/mnt/pool/volume1]# dd if=/dev/urandom of=/mnt/pool/volume1/testx.img bs=1M count=20480
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 511.211 s, 42.0 MB/s

read speed

root@truenas[/mnt/pool/volume1]# dd if=/mnt/pool/volume1/testx.img of=/dev/null bs=1M count=20480
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 25.4646 s, 843 MB/s

Even when reading a cold file that has not been cached, it can still reach a read speed of 520 MB/s.

root@truenas[/mnt/pool/volume1]# dd if=/mnt/pool/volume1/a_cold_file.mkv of=/dev/null bs=1M count=20480
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 41.3029 s, 520 MB/s

As the above test results show, my read speed can reach 843MB/S, but the write speed is only 42MB/s.
The dataset I used for testing has Atime and compression functions turned off, and is configured with sync=ALWAYS.

My physical hardware configuration is AMD 5650GE+64G ECC MEM+ASUS B550M motherboard; TrueNAS runs as a VM in Proxmox, and I have allocated 4 cores and 32G memory to the TrueNAS VM. I have passthrough the onboard SATA controller to TrueNAS so that TrueNAS can directly operate the hardware, so I believe that my virtualization should not be significantly related to the performance problems I encountered.

My storage pool configuration is as follows. I used 4*16T HDDs(2 WD,2 Toshiba) to form a 2x2 mirror pool:

pool: pool
state: ONLINE
scan: resilvered 9.75T in 23:38:35 with 0 errors on Sat Sep 30 07:36:35 2023
config:

NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
2343edf3-760d-4ed7-b7eb-caa44e715193 ONLINE 0 0 0
dbc71442-bc2b-4df4-94de-bb308c837dd4 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
3248d680-ab45-47d4-8d4e-0828b1b03687 ONLINE 0 0 0
4cb3c5f9-1113-41c1-be65-8b498cae56ac ONLINE 0 0 0
cache
476716f5-5577-46e3-92fe-be42efa95151 ONLINE 0 0 0

errors: No known data errors

I hope you can help me, thank you again and have a nice day.

 

[Patrick M. Hausen]

/dev/urandom is limited in read speed itself. Generation entropy is an expensive operation.

Disable compression on a test dataset and retry with /dev/zero.

 

[nsking]

/dev/urandom is limited in read speed itself. Generation entropy is an expensive operation.

Disable compression on a test dataset and retry with /dev/zero.

It's still slow.

root@truenas[~]# dd if=/dev/zero of=/mnt/pool/volume1/test2.img bs=1M count=2048 ^C1148+0 records in 1148+0 records out 1203765248 bytes (1.2 GB, 1.1 GiB) copied, 25.8331 s, 46.6 MB/s

 

[morganL]

Mirrors are fast for Reads, but not for Writes
No SLOG - are the datasets sync? That doubles bandwidth and increases latency
Record size?
SATA controller? (SAS is better)

 

[sfatula]

Here is mine using OP command with a mirror of Seagate Exos drives.

dd if=/dev/zero of=/mnt/tank/test bs=1M count=8192

8589934592 bytes (8.6 GB, 8.0 GiB) copied, 2.45831 s, 3.5 GB/s

Mirror pretty fast here! Mine is a single vdev on SATA so should be slower than his. Mine is (obviously) not sync writes.

Of course, he's running in Proxmox so who knows how that plays into it. Mine is bare metal.

He has sync=ALWAYS so there's a problem.

 

[nsking]

Mirrors are fast for Reads, but not for Writes
No SLOG - are the datasets sync? That doubles bandwidth and increases latency
Record size?
SATA controller? (SAS is better)

Could you please advise me on how to check if the dataset is sync? And if it is not synchronized, what steps should I take to sync it?

 

[morganL]

Screen is here: https://www.truenas.com/docs/scale/...etsscreensscale/#add-and-edit-dataset-screens
Sync = Disabled is faster

 

[nsking]

Screen is here: https://www.truenas.com/docs/scale/...etsscreensscale/#add-and-edit-dataset-screens
Sync = Disabled is faster

I tried disable the Sync, and indeed it brought faster speeds. However, this still doesn't answer my question: why is the write speed so low under sync=always —— even less than 50MB/S.
Even with mirrors, the speed shouldn't be this low, should it?

 

[winnielinnie]

However, this still doesn't answer my question: why is the write speed so low under sync=always —— even less than 50MB/S.

Considering ZFS + VM + spinning HDDs + the data must fully flush to the actual platters on all mirrored drives: 40-50 MB/s sounds about right

This doesn't take into account other factors, such as recordsize; checksum (which I'm assuming you left at the default); and how compression can paradoxically result in faster writes, especially with larger recordsizes and compressible data.

Why set sync to "always"? Why not leave it at its default?

 

[anodos]

Screen is here: https://www.truenas.com/docs/scale/...etsscreensscale/#add-and-edit-dataset-screens
Sync = Disabled is faster

Generally it's not a great idea to disable sync on datasets unless you understand the setting and are willing to take the associated risks.

 

[morganL]

I tried disable the Sync, and indeed it brought faster speeds. However, this still doesn't answer my question: why is the write speed so low under sync=always —— even less than 50MB/S.
Even with mirrors, the speed shouldn't be this low, should it?

Sync often requires the data to be written twice... once to put it in the ZIL... then once more to write it to the pool.

In this case, the data is mirrored so it is being written to 2 drives.

So, in total, the data is being written 3X... 50MB/s is actually 150MB/s of disk writes. This overhead makes the data writes very reliable.

Add an SSD SLOG if you want to make it faster.

 

[mav@]

sync=disabled means "violate applicaiton requests". Unless there is a real reason to not trust the application, sync=standard should be preferable. sync=always is indeed expensive as Morgan told. Depending on few things it may or may not need to write data twice, but it does require to flush disk caches after each write request, that for HDDs and not top-end enterprise SSDs it a very expensive operation, dramatically reducing performance. sync=always is onlyneeded in some HA scenarios, when system must reliably keep all the data as possible across crashes and power cycles. In most cases it is not needed.