Slow read/write speeds over smb/nfs...

[jagdtigger]

Hi all.

So i have an issue i cant quite figure out. When i copy over/from the NAS speeds start out pretty decent but are dropping rather quickly, sometimes all the way down to 15 MB/s (even if i use the client machine's ssd as destination/source), thats weird behavior only happened on linux. On windows its simply stuck at 5 MB/s. Started a scrub to see if one of the disks acting up but i didnt see any hdd sticking out in terms of latency and io, smart values are ok on all 5. The pc runs xcp on bare metal then truenas as a VM with 8 cores and 32GB RAM. Raid card HBA(hp h220 flashed to IT mode) passed over to vm, it has 5 disks in raidz2. Host also has an extra Intel 4 port NIC configured as LAG with LACP.


Did anyone here run into a similar issue or has some idea where to look for the problem?

/EDIT
Tested a different VM on the same xcp host, no probs maxing out the gig link of a asustor NAS. So it seems like its not an xcp-ng issue.

 

[joeschmuck]

This is a TrueNAS bug. Do a search for "TrueNAS slow smb aio" and that should find you the thread to fix it. Or just look at this thread and see if that fixes things for you. https://www.truenas.com/community/t...ers-over-10gbe-but-ftp-fast.87850/post-608878

 

[jagdtigger]

Added aio read size =0 to aux params but no effect (yes i disabled then re-enabled the smb service), plus the same is happening when i use nfs....

 

[joeschmuck]

plus the same is happening when i use nfs....

Try to roll back to your previous FreeNAS version, hopefully you did not upgrade the pools. See if you still have the same problem. You may need to submit a bug report for the NFS, or at least check to see if a bug report has been posted for this problem. But first verify it is TrueNAS causing the issue.

Good Luck!

 

[jagdtigger]

Unless the OS upgrade doesnt do it automatically probably not. I can only roll back to 12 beta 2.1 though.....

 

[joeschmuck]

Unless the OS upgrade doesnt do it automatically probably not. I can only roll back to 12 beta 2.1 though.....

No, upgrading the OS does not update the pool, that is a separate and very intentional act. We do not recommend anyone update the pool unless there are specific features offered by that upgrade because once done, you cannot use a version of FreeNAS that does not support that pool structure.

With that said, it sounds like you built your pool with TrueNAS 12, not a FreeNAS version, so that means you are unlikely to be able to roll into FreeNAS 11.3-U5. If you built your pool on a FreeNAS version then you can install FreeNAS 11.3-U5 and restore an earlier configuration file and be up and running. With you having an account going back to 2017 I would have thought you are using an older pool.

I'm sure you will figure it out or wait until the next version of FreeNAS to come out. You could also try the nightly builds if you are a risk taker.

 

[jagdtigger]

Yes it was built on beta 2.1, rolled back to rc1 but it exhibited the same behavior, ill try the beta. As for the nightlies are there any chance of getting a corrupted pool by running it?

/EDIT
Rolled back to the beta, same thing going on. Could this be caused by the VM changing? (Assigned more ram and 4 more cares recently, vm was shut down ofc.)

/EDIT2
Ok, now im totally lost:

Code:

heathcliff@zbox-magnus:~$ iperf -c 10.125.210.43

------------------------------------------------------------

Client connecting to 10.125.210.43, TCP port 5001

TCP window size:  544 KByte (default)

------------------------------------------------------------

[  3] local 10.125.210.150 port 40598 connected with 10.125.210.43 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec  1.10 GBytes   940 Mbits/sec

heathcliff@zbox-magnus:~$ iperf -c 10.125.210.22

------------------------------------------------------------

Client connecting to 10.125.210.22, TCP port 5001

TCP window size:  476 KByte (default)

------------------------------------------------------------

[  3] local 10.125.210.150 port 47492 connected with 10.125.210.22 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec  1.04 GBytes   889 Mbits/sec

Both IP belongs to a VM on the same xcp host, 43 is a zabbix vm and 22 is the truenas vm. (This was done from my desk PC.) So its not the network, scrubbing was showing speeds i expect from 5 hdd so its reasonable to say its not the storage. What could affect smb and nfs at the same time?

 

[jagdtigger]

Updated to nightly using this iso:
TrueNAS-12.1-MASTER-202011180538-b0e8c3a.iso

Nothing changed. On linux it caps out at ~10 MB/s, on windows it hovers between 20-30 MB/s......

 

[jagdtigger]

Bare metal test run achieves 91 MB/s..... Guess ill have to ditch xcp :( .

 

[joeschmuck]

ESXi works very well, that is all I can personally speak to, and I do use the free version (ESXi alone). Proxmox I've read works too.

 

[jagdtigger]

Works well, on server grade HW that is....

 

[joeschmuck]

I have use ESXi 5.5 and 6.0 also on much less that server grade hardware, but I agree and if you read the Recommended Hardware Guide, it's recommended to use server grade hardware. As I'm hearing, ESXi 7.0 has been tailored towards server grade only and more current hardware but I could be wrong. I have ESXi 7.0 running on my system which is not "new" but I hope it lasts for another decade.

Good luck whichever path you go down.

 

[jagdtigger]

Back when i tried it i couldnt get it to recognize the onboard sata so i dropped it pretty quickly. As i dont have that much free time i opted for the lazy way and gone bare metal with truenas. I really hope bhyve is stable enough for my game servers.

 

[joeschmuck]

You might give Proxmox a try if you haven't already. I have little faith in bhyve, it isn't very mature. You have likely already done your homework with respect to bhyve and whatever game server you want to run. If someone else has made it work, just ask them for details if they haven't already posted them.

 

[jagdtigger]

Dont have any spare machines and time to try. For now the mineos plugin covers my needs. Maybe 1 or 2 years later i might consider running a different server but right now i hardly get the free time to play mc, even if i do most of the time im too tired and fire up youtube and usually fall asleep while watching something....