100MBs max read/write bottleneck on 10G connectivity

[senses]

Hello dear Truenas Community

Im new to Truenas and im reaching out regarding a problem that ive been having and so far havent been able to solve.

I have a write and read bottleneck when transfering data to and from a newly installed truenas core VM inside proxmox (doing so because i run other VMs and services and needed some consolidation due to lack of rack space).

On iperf3 i get 9.8Gbit/s speed max, but when i try to move files in and out of a share on the only pool i have, i get around 100MB/s write/read at maximum.

Here are the host proxmox specs and the truenas VM specs:

Proxmox Host:

Asrockrack EPYCD8-2T MOBO
AMD EPYC 7452 32-Core CPU
4x 32GB Kingston 3200 MHz KSM32RD4/32MEI
2X 120G Kingston A400 SSDs (Proxmox boot drives in raid-1)
LSI SAS 9207-8i PCIe card (passed trough to Truenas)
Seagate Ironwolf 510 960GB NVME SSD (VMs stored here)
8x Seagate EXOS 4TB HDDs ( all connected to LSI HBA)
3x Samsung 970 EVO 250GB SSDs (All passed trough to truenas)

Truenas VM:
VM image is 50GB and stored in the seagate Ironworld NVME SSD
32GB Ram
12 CPU Cores
Passed trough LSI Controller with the 8 Seagate drives attached
Passed trough the 3 Samsung 970 EVOs

My desktop:
16 core threadripper
32G ram
Adata xpg 1TB NVME SSD
Asus XG-C100F

Connecting everything together:
Mikrotik CRS312-4B+8XG switch

The setup:

The pool i created inside Truenas is a Raid-Z2 with a hot spare of one of the 8 seagate drives, 2 cache 970 EVOS and 1 970 EVO for log.

The VM is using 1 of the 2 10G ports of the mobo and is connected to the Mikrotik switch and so is my main desktop, traffic is flowing directly between them and on the same LAN.
Truenas share is a SMB share on the Z2 pool.
Truenas shows 10Gbase-T media subtype.
MTU is set to default on everything, proxmox, truenas and my desktop.

Didnt modify anything else besides installing, creating pool, share, and testing speeds.

On iperf3 i get a maximum of 9.8Gbps with it being steady above 9.3Gbps.
When i transfer a 15G movie to a share in the pool i get 100MBs write speed to it, when i transfer it to my desktop again i get the same speed, even if i do it many times to make sure im reading from the truenas cache.

This happen when writing/reading all kinds of files and not movies, media/rar files, its a global speed limit.

What am i missing and what can i try to solve this?

Thanks to everyone in advance!

 

[NugentS]

A Z2 Pool has the IOPS of a single disk (simplistically)
Could you be running into the max IO from the RAIDZ2 pool
Mind you the second time you transfer you should be getting better speeds - as it should be in ARC rather from the spinnning platters.

Dunno

 

[Spearfoot]

Hello dear Truenas Community

Im new to Truenas and im reaching out regarding a problem that ive been having and so far havent been able to solve.

I have a write and read bottleneck when transfering data to and from a newly installed truenas core VM inside proxmox (doing so because i run other VMs and services and needed some consolidation due to lack of rack space).

On iperf3 i get 9.8Gbit/s speed max, but when i try to move files in and out of a share on the only pool i have, i get around 100MB/s write/read at maximum.

Here are the host proxmox specs and the truenas VM specs:

Proxmox Host:

Asrockrack EPYCD8-2T MOBO
AMD EPYC 7452 32-Core CPU
4x 32GB Kingston 3200 MHz KSM32RD4/32MEI
2X 120G Kingston A400 SSDs (Proxmox boot drives in raid-1)
LSI SAS 9207-8i PCIe card (passed trough to Truenas)
Seagate Ironwolf 510 960GB NVME SSD (VMs stored here)
8x Seagate EXOS 4TB HDDs ( all connected to LSI HBA)
3x Samsung 970 EVO 250GB SSDs (All passed trough to truenas)

Truenas VM:
VM image is 50GB and stored in the seagate Ironworld NVME SSD
32GB Ram
12 CPU Cores
Passed trough LSI Controller with the 8 Seagate drives attached
Passed trough the 3 Samsung 970 EVOs

My desktop:
16 core threadripper
32G ram
Adata xpg 1TB NVME SSD
Asus XG-C100F

Connecting everything together:
Mikrotik CRS312-4B+8XG switch

The setup:

The pool i created inside Truenas is a Raid-Z2 with a hot spare of one of the 8 seagate drives, 2 cache 970 EVOS and 1 970 EVO for log.

The VM is using 1 of the 2 10G ports of the mobo and is connected to the Mikrotik switch and so is my main desktop, traffic is flowing directly between them and on the same LAN.
Truenas share is a SMB share on the Z2 pool.
Truenas shows 10Gbase-T media subtype.
MTU is set to default on everything, proxmox, truenas and my desktop.

Didnt modify anything else besides installing, creating pool, share, and testing speeds.

On iperf3 i get a maximum of 9.8Gbps with it being steady above 9.3Gbps.
When i transfer a 15G movie to a share in the pool i get 100MBs write speed to it, when i transfer it to my desktop again i get the same speed, even if i do it many times to make sure im reading from the truenas cache.

This happen when writing/reading all kinds of files and not movies, media/rar files, its a global speed limit.

What am i missing and what can i try to solve this?

Thanks to everyone in advance!

You've discovered the sad truth that once you move to 10G, your storage subsystem becomes the new bottleneck. The best way I can think of to overcome it would be to use an all-flash storage system, but that would be hideously expensive!

There's just no way to flog more speed out of 6Gb/s SATA3/SAS2 hard drives, but there are some things you can do to improve performance: use mirrors instead of RAID-Z2, use jumbo frames (some will scoff at this, but it made a big difference for me), and set some tunables:

I'd go ahead and use all 8 of your HDD's in 4 x mirror vdevs. Hot spares make sense for critical/remote production servers, but not in (most) home labs. Just keep a tested cold spare on-hand. I see you're using Seagates... maybe you ought keep 2 or 3 cold spares?

Samba is single-threaded, or at least it was the last time I checked, so you're probably wasting all those cores on your TrueNAS VM. I only use 4 for my FreeNAS VM (see 'my systems' below). TrueNAS always benefits from more memory, so you might try increasing that.

I've done all of the above and the best I can get for network transfer rates is ~3Gb/s. Here are CrystalDiskMark v3.0.4 results from my PC accessing a Samba share on my main FreeNAS server, along with the server's interface traffic graph:

Hope this gets you started... good luck!