Slow write speeds are confusing me

[iamrufus]

Evening all,

I have setup my first Truenas server and im liking the look of everything so far but i am getting really slow writes and i cannot work out why. I have a microserver gen10+ and TrueNas is installed on a usb3 external ssd. I have 2x 4TB Iron Wolf drives and 2x 1TB Samsung Evo SSDs installed for my data. My home is only a 1gb network so i was hoping to be able to transfer around the 100MB/s mark across the network

I set up my first pool named Slow_Storage and added my Iron Wolf drives in a mirror, i then created a second pool named Fast_Storage and added both SSDs to it. I then created a dataset named 'Media' on the slow storage pool and then created an SMB share named movies and put it in the directory /mnt/Slow_Storage/Media

I then wanted to test sending over a 25gb mkv file from my iMac so i mounted the share and then tried copying it via rsync so i could see the actual throughput speed of the transfer. I used rsync -avz /source /dest and all i was getting was around 15MB/s. I created the same share on the fast storage and got the exact same result even though i was copying to the SSD pool this time round.

Im really confused and not sure if i have missed something here. Is my hardware my bottleneck somehow? Ive even tried turning off sync and forcing it on but it doesnt seem to make a difference. I even tried creating a pool with a single stripe drive to see if it was the mirror causing the slow write but i was getting the same write speed that way as well.

Id be really grateful for some advice, i actually tested the same setup with unraid and it seemed to work perfectly but i want to stick with truenas as its a lot more feature rich.

Thank you

 

[HoneyBadger]

Assuming that the cables are good because you have been able to get UnRaid to perform at "full speed" - though an iperf test may also help this.

The SMB defaults now have strict sync enabled, and MacOS will leverage this.

How was the dataset exported (SMB or NFS) and do you see a difference in speeds with the dataset set to sync=disabled?

 

[iamrufus]

Hi, yes all the cables appear to be fine. I had the exact same speed result when hooking up an external usb hard drive and mounting it then rsyncing files to the slow array from it.

I tried all the sync settings but they all produced the same result. The max throughout I have seen is 25MB/s

Maybe I will try NFS but I just assumed SMB would work well out of the box

 

[sretalla]

tried all the sync settings but they all produced the same result. The max throughout I have seen is 25MB/s

Something's not right there... if you're getting the same speed with sync=always and sync=disabled that points to it not being the pool that's the problem, so look at network or the client..

 

[iamrufus]

The strange thing is, I get the same slow speed with a usb3 drive attached to the TrueNas server itself and mounted. When I run the rsync command it still only writes at 15-25MB/s with sync disabled or enabled.

Does TrueNas need to be rebooted for the sync setting to take effect when it’s changed? It’s not network related for sure, it can’t be as I have bypassed it with the usb3 drive

 

[HoneyBadger]

What is the CPU in your server? If you're using encryption and/or compression in the rsync command you might simply be bottlenecking at that point.

 

[iamrufus]

The cpu is a Xeon E-2224 4 core 4 thread. The only commands I am using with rsync is -av. I tried copying a file on the same array to another folder on the same array with the same command as well and that give me a transfer speed of around 65MB/s. It’s was a 25gb mkv file.

I’m on the verge of giving up with TrueNas I won’t lie, I had similar issues with proxmox performance as well with ZFS on the same hardware and disks. I have tried removing sync on the array and it did nothing at all. I still feel like I am missing something I just don’t know what.

 

[HarambeLives]

What HBA/SATA Controller?

Also, run a memtest. I've dealt with issues like this in the past and found a bad stick of RAM

 

[Forza]

The cpu is a Xeon E-2224 4 core 4 thread. The only commands I am using with rsync is -av. I tried copying a file on the same array to another folder on the same array with the same command as well and that give me a transfer speed of around 65MB/s. It’s was a 25gb mkv file.

I’m on the verge of giving up with TrueNas I won’t lie, I had similar issues with proxmox performance as well with ZFS on the same hardware and disks. I have tried removing sync on the array and it did nothing at all. I still feel like I am missing something I just don’t know what.

If you have a USB drive in your truenas. Try copy with mc. Shouldn't be a problem to reach 100MiB/s over 1gbit.

Disable compression and set sync=never on your pool

 

[iamrufus]

Thanks for your help, interestingly I have just spun up a windows 10 VM on my iMac and when I transfer a file say 25gb mkv from the VM it transfers at around 95MB/s (this is over wireless as well just as a test) to the pool.

So im guessing it must be something to do with rsync, I copied the same file directly from my iMac over rsync and it was hitting only 35MB/s.

 

[Elliot Dierksen]

So im guessing it must be something to do with rsync

rsync spends a non-trivial amount of time trying to determine IF it should copy something. It is not a full blast maximum use of the network kind of operation, so it isn't a good benchmark of network speed. If have seen this as well when I do an rsync between FreeNAS units over a dedicated 40G connection.