Extremely slow read speeds.

[Amplificator]

I'm having an issue with extremely slow read speeds on my FreeNAS server. Write speeds are great and saturates my 1Gb LAN connection without a problem, but anything read-related is slow. Here's a short clip: https://imgur.com/a/0vGFFFw - yes, that is real-time :-/

My server is a Dell PowerEdge T610 with 2x L5640 cpu's, 96GB of memory, 2x 160GB SSD's for boot drive, 1x 300GB SSD which caches for the remaining 5x 8TB drives - all WD Red's, Ironwolf and the like.

I have tried with each drive set up individually and currently with all drives in Raid6 on the PERC H700 controller - same result in speeds.
I have also tried Z2 in FreeNAS but currently doing just a stripe, as the disks are set up using the hardware raid controller.

When using ssh and copying files as root around in various datasets I see no issues and things are fast. The problem occours when I want other machines to read from the server. I have tried Samba, NFS and AFP shares, all with the same result.
I have tried with deduplication off, atime off and anything else I could think of, to no availability.

Windows can achieve slightly faster speeds than macOS can, both on a cabled connection, but still fluctuates more than it should.

Currently I have no idea what is going on, but could really appreciate some help here.

Ask me anything, be it logs or whatever, and I will provide it :)

 

[Samuel Tai]

Your narrative raises a couple of red flags:

I have tried with each drive set up individually and currently with all drives in Raid6 on the PERC H700 controller - same result in speeds.

If the PERC H700 isn't running in IT mode, I'm surprised it's working at all. This is the first thing to check.

1x 300GB SSD which caches

If your L2ARC is thrashing, then it would manifest as slow reads. Try removing the L2ARC, and provide the output of /usr/local/bin/arc_summary.py.

 

[Amplificator]

H700 doesn't support IT mode :-/
However, several weeks ago it was running fine. Nothing changed in between. I even did a clean install of 11.3U4 just a couple of days ago to make sure I didn't do something silly.

Here's the output from arc_summary.py : https://pastr.io/view/gCyOlFMFNo1

 

[Samuel Tai]

The most likely scenario is the H700 is failing. You should replace it with an H200 flashed to IT mode.

 

[Amplificator]

I'm actually looking at some H200's now.
Any particular model to get? I see there are several part numbers.

I'm looking at these:

Dell PERC H200 6Gb/s SAS Controller Card 047MCV 0U039M | eBay

HALESFIELD 9. TECH POINT. XPO IT SERVICES LTD. General Information.

www.ebay.co.uk

Dell PERC H200 6GBps PCI-E SAS SATA RAID Controller 03J8FW 3J8FW | eBay

Find many great new & used options and get the best deals for Dell PERC H200 6GBps PCI-E SAS SATA RAID Controller 03J8FW 3J8FW at the best online prices at eBay! Free delivery for many products!

www.ebay.co.uk

Dell PERC H200 8Port 6Gb/s Adapter SAS SATA Controller Card= 9211-8I 47MCV M1015 | eBay

Interface SAS, SATA, Two internal miniSAS connectors. Dell C2100 and other DELL servers. Form Factor PLUG IN. support RAID0,1,10,1E and JBOD. Part Number 047MCV /00RR9J/0U039M.

www.ebay.co.uk

Carte Raid Controller PCIe SATA SAS DELL PERC H200 0H216J 047MCV E2K-UCS-71 B 3664870196842 | eBay

Find many great new & used options and get the best deals for Carte Raid Controller PCIe SATA SAS DELL PERC H200 0H216J 047MCV E2K-UCS-71 B at the best online prices at eBay! Free delivery for many products.

www.ebay.co.uk

Any part number that is better than the other?

 

[Samuel Tai]

They should all flash to the same firmware, so there's no preference. To save yourself some work, you could get one already pre-flashed to IT mode.

 

[Amplificator]

Can't seem to find one in Europe that is already prefalshed with the latest firmware :/

 

[Samuel Tai]

Here's one in the UK:

Dell H200 Integrated 6Gbps SAS HBA LSI 9210-8i P20 IT Mode ZFS FreeNAS unRAID | eBay

Find many great new & used options and get the best deals for Dell H200 Integrated 6Gbps SAS HBA LSI 9210-8i P20 IT Mode ZFS FreeNAS unRAID at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[Amplificator]

Will I need additional cables? (if yes, which ones) or can I use whatever is already in the machine?

 

[Amplificator]

The same seller is also selling a H310 for the same price - which would be the best card? As far as I can tell they both have the same chip on it.

 

[Samuel Tai]

Your current SAS cables will work. As for the H310 vs H200, the only difference I can see is the placement of the ports. The H200 has the ports in the same place as the H700, so there'll be less fiddling with cables.

 

[Amplificator]

Alright, I'm ordering the H200 card you found then, thanks :)

 

[Amplificator]

I'm still having the same issue. I also tried Proxmox and an Arch Linux samba server.

My pool is created using this:

• zpool create -f -o ashift=12 -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase -m /mnt/Vault Vault raidz2 <DISKS> log <DISK>

• zfs set atime=off Vault
• zfs set compression=lz4 Vault
• zfs set xattr=sa Vault
• zfs set dnodesize=auto Vault
• zfs set acltype=posixacl Vault
• zfs set recordsize=128k Vault
• zfs set sync=disabled Vault

This is weird, any suggestions?

 

[Amplificator]

I think I fixed it. It appears to have been a samba issue. I changed "aio max threads" from 2 to 256 and now it's flying..

Now I have samba, nfs and webdav all working with no issues it seems.