SOLVED RAIDZ1 single vdev performance issue

[NeverEnd]

Hello, I'm new to Freenas, I've read a lot (at leasst a lot to me) about Freenas and NFS online, it's the first time I'm trying to setup a Freenas storage server at home. This is the hardware I'm using:

Mother board: ASUS Z87
CPU: Intel i5 GEN4
MEM: 8GB DDR3
NIC: Intel I210-T1
DISK: 4x WD RED 4TB

I know - these things mearly meat the minimum hareware requrements of Freenas but it's what I have now, and I just want to setup a Freenas storage server at home so I can put my photos on it and edit from my MBP. The hard drives are connected to those SATA 6GB ports on the mother board (directly connected, without RAID function of the mother board), and I made a pool with a RAIDZ1 vdev.

Both the Freenas server and my MBP are connected to a Fortigate 60E (I'm usisng Apple's USB-C Thunderbolt Gigabit Ethernet adapter with MBP), and I'm expecting the file transfer speed to reach at least 100MBps or so, but it can only reach about 40-50 MBps when transfering a single large file, and about 20-30 MBps with small files, both "upload" (from MBP to Freenas) and "download" speed are about the same. I've also tried NFS and SMB, the same. SFTP is even slower.

Is there anything that I'm missing, or any suggestions?

 

[garm]

What is your write speed running something like dd on the NAS itself?

 

[NeverEnd]

What is your write speed running something like dd on the NAS itself?

I'll test it this evening and post the results here.

 

[NeverEnd]

I've done the write/read test running "dd" on Freenas and this is the result:

Code:

dd if=/dev/zero of=/mnt/Pool1/main01/ddtest bs=1024k count=100000
100000+0 records in
100000+0 records out
104857600000 bytes transferred in 52.244679 secs (2007048416 bytes/sec)

Code:

dd if=/mnt/Pool1/main01/ddtest of=/dev/zero bs=1024k count=100000
100000+0 records in
100000+0 records out
104857600000 bytes transferred in 21.132713 secs (4961861849 bytes/sec)

To me, the results seem too good to be true... I do have a SSD as SLOG device in the pool, not sure if it plays some role in the dd test.

 

[rvassar]

You're using /dev/zero for you testing, so the results are skewed by the slog and the compressibility of a string of all zero's.

Are you sure you have a 1GbE link all the way across? You may have a damaged patch cord, or have exceeded the bend radius of a cable somewhere, and the switch kicked you down to 100mbit.

Also... Have you turned WiFi off? It may be just be favoring the wrong interface.

 

[NeverEnd]

You're using /dev/zero for you testing, so the results are skewed by the slog and the compressibility of a string of all zero's.

Are you sure you have a 1GbE link all the way across? You may have a damaged patch cord, or have exceeded the bend radius of a cable somewhere, and the switch kicked you down to 100mbit.

Also... Have you turned WiFi off? It may be just be favoring the wrong interface.

Good point about the damaged patch cord or banding cable... I'll connect my MBP directly to NAS with a good cable and do the file transfer test to see how it works. Fortigate shows both links are 1000 Mbps however I know it can't always be trusted.

I do turn wifi off when doing test.

 

[garm]

Ya no use /dev/random for a meningfull result (won’t be real world performance still, but good enough for this). You are not writing 2GB/s. Also turn off compression (in enabled).

You are low on RAM, but given you don’t have any heavy services running I would expect you to saturate 1GbE.

 

[NeverEnd]

Disabled compression and did a random write and read dd test:

Code:

dd if=/dev/random of=/mnt/Pool1/main01/ddtest bs=1024k count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 65.603677 secs (159834944 bytes/sec)

Code:

dd if=/mnt/Pool1/main01/ddtest of=/dev/zero bs=1024k count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 38.865277 secs (269797642 bytes/sec)

It seems the disks' speed should be enough to saturate 1GbE, however with my MBP connected directly to the NAS box the write speed is still about 50 MBps and read speed is about 70 MBps. Nothing could be wrong to me, the cable is absolutely good.

 

[garm]

Looks better, that is the spec speed from WD.
So FreeNAS performs well under ideal conditions. 50 MBps sounds like a duplex issue, it’s very close to half the bandwidth. Can you do another dd run from MacOS to the share? (Maybe share the same dataset you used for the FreeNAS dd test?)

 

[NeverEnd]

Did a dd test from Mac, write speed is much slower than a single file copy... only 20 MBps:

Code:

dd if=/dev/random of=/Volumes/main01/ddtesst bs=1024k count=500
500+0 records in
500+0 records out
524288000 bytes transferred in 24.472271 secs (21423758 bytes/sec)

 

[HoneyBadger]

MacOS

Likely this is part of, if not entirely the issue. MacOS SMB will request sync SMB and since strict sync = yes flag is in the SMB extensions by default as of 11.1 it will slow things down. (Related thread.)

Your SLOG might be in the data path here as well but it's likely you don't truly need the sync writes here. Remove it, set strict sync = no on the share, and see if things rocket back up to normal speed.

 

[NeverEnd]

Took a look at /usr/local/etc/smb4.conf and it does say "strict sync = Yes", changed it to "no" and restarted SMB service, will do the test when I'm back home.
I did tests with SMB and NFS sharing, the speed is about the same, not sure if there's some similar configurations with NFS?
The pool/dataset has "Sync" set to "Disabled", SMB signing is turned off in " /etc/nsmb.conf" file on Mac, by the way.

 

[NeverEnd]

I have a configuration problem here. Checked with "testparm -v" on Freenas and the output says "strict sync = Yes" under "Global parameters", I thought it's controlled by "/usr/local/etc/smb4.conf" so I added "strict sync = no" in smb4.conf, restarted SMB service, checked again with "testparm -v", it still shows "strict sync = Yes". I Googled "freenas smb global configuration" but did not found anything particularly useful, could anyone tell me where's the right place to put "strict sync = no" in?

 

[garm]

You do everything in the UI.. in this case it’s “auxiliary parameters“ https://www.ixsystems.com/documentation/freenas/11.2-U4.1/services.html#smb

 

[NeverEnd]

You do everything in the UI..

alright... I'll do it sometime later today.
BTW, I think everything I do in the UI should be synced with those configuration files and vise versa, is it not the case?

 

[garm]

I it’s not. The UI stores the settings in a database and writes directly to config files, ignoring what ever is manually entered

 

[NeverEnd]

I removed the slog drive from the pool, put "strict sync = no" in "auxiliary parameters" on SMB config page, restarted NAS, then did the test, still the same... about 40-50MBps when copying single large file to/from NAS, can be 70MBps occasionally, but that's it.

dd test from Mac to the mounted SMB share still shows about 20MBps.

NFS share has the same speed.

I'm pretty sure the Thunderbolt USB-C Gib Ethernet adapter and the cables are fine.

Could there be anything else that matter?

 

[NeverEnd]

Finally... single large file transfer speed to/from NAS is about 115-120 MBps, turns out that the problem is Appel's thunderbolt Gigbit Ethernet adapter, replaced it with a cheap China made adapter and the speed is back to normal.

Many thanks to everyone, I've learned a lot and will continue to learn and have fun with Freenas.