Slow direct Read/Write performance

[TomP]

Hello,

I noticed recently that my write speed on my FreeNAS is pretty poor, I am only getting 30-40MBps (MegaBytes) write speed.
I am measuring this directly on the box, logged in as root, using dd:

[root@nas] /mnt/WDvol/Backup2# dd if=/dev/zero of=10gbfile bs=1024k count=10000
10485760000 bytes transferred in 271.820490 secs (38576047 bytes/sec)

vs. 77MBps read:

[root@nas] /mnt/WDvol/Backup2# dd if=10gbfile of=/dev/null bs=1024k count=10000
10485760000 bytes transferred in 135.784284 secs (77223664 bytes/sec)

On a different Linux system with a SATA 2 disk I get

$ dd if=/dev/zero of=10gbfile bs=1024k count=10000
10485760000 bytes (10 GB, 9.8 GiB) copied, 118.19 s, 88.7 MB/s

(8GB RAM, ext4 (no RAID), 4 Intel(R) Core(TM) i3-3225 CPU @ 3.30GHz)


But on FreeNAS then when I am reading directly from the raw disks, I am getting about 150MBps (MegaBytes)

[root@nas] ~# dd if=/dev/ada0 of=/dev/null bs=1024k count=10000
10485760000 bytes transferred in 69.976778 secs (149846282 bytes/sec)


I understand I am comparing apples and bananas here, but I would expect to have at least 80MBps write speed locally on FreeNAS.

Things I have tried to find the culprit:

• No SMART errors, no disk errors whatsoever. Running a scrub regularily and no errors reported ever.
• diskinfo -v ada(0,1,2) reports a stripesize of 4096, so I guess I have used 4k sectors when formatting
  
• zdb -U /data/zfs/zpool.cache gives me ashift: 12
• Tried with zfs sync=disabled WDvol/Backup2 and then test again, no change
• I noticed this slowness with with CIFS, but also with SSH as well. Tested networkspeed with iperf and ifconfig and everything is 1Gbps, what I expect.

The only odd thing what I couldn't explain is that "gstat" seems to show activity one second, the next all figures
are "0", then activity again - it keeps fluctuating. Whereas the throughput via network (rsync --stats --progress) is steady at about 30MBps).

What am I missing here? Is this normal?


Regards

Tom


Software and hardware specs
I have one RAIDZ1 pool spanning all 3 disks, no encryption, compression was enabled, but I disabled this now.
Compression rates were insignificant, like 1.06 was the best. Disabling compression didn't show any improvement.

FreeNAS-9.10.1 (d989edd)
AMD Turion(tm) II Neo N40L Dual-Core Processor, SATA 2 only
8130MB ECC RAM
3x2TB WD RED disks, SATA 3 (they support 4k or "Advanced Format")

 

[SweetAndLow]

How full is your pool?

Sent from my Nexus 5X using Tapatalk

 

[TomP]

How full is your pool?

WDvol 5.44T 685G 4.77T - 3% 12% 1.00x ONLINE /mnt

685G of 4.77T usable.

 

[SweetAndLow]

WDvol 5.44T 685G 4.77T - 3% 12% 1.00x ONLINE /mnt

685G of 4.77T usable.

This is your problem, you are way over the 90% usage rule. After 90% writes become super slow.

Sent from my Nexus 5X using Tapatalk

 

[TomP]

This is your problem, you are way over the 90% usage rule. After 90% writes become super slow.

Correction:

Pool size: 5.44T
Alloc: 665GB
Free: 4.79T

 

[gpsguy]

Remember that the CPU on your N40L is only 1.5GHz.

CIFS on FreeNAS works best with faster processors.


Sent from my iPhone using Tapatalk

 

[SweetAndLow]

What is your CPU and load usage when running the dd command?

Sent from my Nexus 5X using Tapatalk

 

[TomP]

What is your CPU and load usage when running the dd command?

I am running "dd if=/dev/zero of=10gbfile bs=1024k count=10000" onto a pool.

Load: 0.3, but does down then to 0.2
CPU: 0.4% user, 0.0% nice, 10.7% system, 0.4% interrupt, 88.5% idle

CPU pretty constant around 88% idle. No other load on the system right now.

Copying via CIFS would give me slightly lower speeds than 30-40MBps. But my problem is not with CIFS (or SSH), because the disk speed local on the system is slow.