Extremely poor latency accessing pool

[CelticWebs]

I'm running Proxmox backup server in a VM. Setup as follows:

Spec
Dual Xeon based server running Trueness scale
48GB memory
LSI 9211-4i raid controller in IT mode.
2x 500gb Enterprise SSD I as mirrored boto drive
24 Enterprise 4TB SAS disks in a Draid draid3:8d:22c:1s-0

I've realised the main pool has latency issues but have no idea how to diagnose. I did some basic tests from the Truenas Terminal, results below.

Speed
dd if=/dev/zero of=test1.img bs=1G count=1 oflag=dsync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.77884 s, 386 MB/s

Latency
dd if=/dev/zero of=test2.img bs=512 count=1000 oflag=dsync
512000 bytes (512 kB, 500 KiB) copied, 11.3417 s, 45.1 kB/s

As you can see the write speed is "OK" but the latency is awful, the equivalent latency on my Synology DS1019+ with 5 disk BTRFS raid is :
512000 bytes (512 kB, 500 KiB) copied, 0.328728 s, 1.6 MB/s


Does anyone have any suggestions what could be causing this poor latency? I am at a loss as to what it could be!

Thanks in advance for your help!

 

[NickF]

I’m not sure exactly what you are testing for here. You are intentionally writing 512 byte blocks and only writing 512K of data.

This isn’t really a real world test. Can you kick off a backup for Proxmox? While running kick off this command on the TrueNAS

Zpool iostat -vvyl 120 1

Which will sample pool performance, including latency, for two minutes

 

[CelticWebs]

I’m not sure exactly what you are testing for here. You are intentionally writing 512 byte blocks and only writing 512K of data.

This isn’t really a real world test. Can you kick off a backup for Proxmox? While running kick off this command on the TrueNAS

Zpool iostat -vvyl 120 1

Which will sample pool performance, including latency, for two minutes

Thanks for the response. So run a backup and then run the iostat command outside of the vm in the main truenas terminal?

 

[NickF]

Thanks for the response. So run a backup and then run the iostat command outside of the vm in the main truenas terminal?

Yep, while the backup is running that command will observe the performance and print out telemetry

 

[CelticWebs]

Yep, while the backup is running that command will observe the performance and print out telemetry

Here’s the results, not sure if they’re good or bad tbh.

 

[CelticWebs]

I’ll do it from an actual computer shortly so that the screen is better laid out to read.

 

[CelticWebs]

I've no idea if this is good or bad?

 

[NickF]

@CelticWebs you have disk latency averaging about 12ms. What this effectively means is that the bottleneck you are trying to illustrate is probably not there. I think your testing methodology is just poorly illustrative of the workload.

 

[CelticWebs]

@CelticWebs you have disk latency averaging about 12ms. What this effectively means is that the bottleneck you are trying to illustrate is probably not there. I think your testing methodology is just poorly illustrative of the workload.

Thanks Nick, I did some tests on my Synology system and figures wise, everything performed better on the Truenas Scale system, whcih you'd expect wit hath spec difference. So I'm at a total loss why It operates so slowly when doing backups and Garbage collection in Proxmox Backup Server.

It's been running for a day on a garbage collection and as you can see, it hasn't got to 2% yet and the system it's self is under no load at all!

I'm wondering if it's the way the VM is setup somehow.

 

[NickF]

I'm not sure entirely how Proxmox Backup does it's thing. Looking here:

Technical Overview — Proxmox Backup 3.1.4-1 documentation

pbs.proxmox.com

Are you using compression on the Proxmox Side and the ZFS side?

Code:

For block based backups (like VMs), fixed-sized chunks are used. The content (disk image), is split into chunks of the same length (typically 4 MiB).

You can try increasing the ZFS record size to 1M from 128K to see if that helps.

 

[CelticWebs]

I’ll have a look and give it a go and see what happens.