[SOLVED]LAN SMB/FTP copy unstable result, will drop to around 20MB/s, related to ZFS cache?

[skygunner]

Hi,

I'm testing
FreeNAS-9.10.1-U2 (f045a8b) stable. My setup:

Code:

Intel S3420GPC Server board
Intel(R) Xeon(R) CPU X3470 @ 2.93GHz
8GB DDR3 ECC
Intel(R) PRO/1000 Network Connection
Intel 5 Series/3400 Series AHCI SATA controller with 6 2TB harddrives

cat: /etc/version.freenas: No such file or directory

System Load: 0.07, 0.10, 0.08

CPU Temperature:
dev.cpu.7.temperature: 40.0C
dev.cpu.6.temperature: 40.0C
dev.cpu.5.temperature: 43.0C
dev.cpu.4.temperature: 43.0C
dev.cpu.3.temperature: 46.0C
dev.cpu.2.temperature: 46.0C
dev.cpu.1.temperature: 41.0C
dev.cpu.0.temperature: 41.0C

Drive Activity Status
ada0:FF  ada1:FF  ada2:FF  ada3:FF  ada4:FF  ada5:FF

HDD Temperature:
ada0 35 9WM65P6Y ST32000644NS
ada1 36 9WM2E5BJ ST32000644NS
ada2 30 Z8E00331 ST2000DM001-1ER164
ada3 36 9WM1PNPB ST32000644NS
ada4 36 9WM1B0BH ST32000644NS
ada5 31 WD-WCAZA9734804 WDC

harddisks have been put into mirrored vdevs(each vdev has 2 harddisks)

zpool status
  pool: tank
 state: ONLINE
  scan: none requested
config:

		NAME											STATE	 READ WRITE CKSUM
		tank											ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/a3a48b01-8ca6-11e6-8402-001517f91241  ONLINE	   0	 0	 0
			gptid/a46240a3-8ca6-11e6-8402-001517f91241  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/a5a7d998-8ca6-11e6-8402-001517f91241  ONLINE	   0	 0	 0
			gptid/a6652796-8ca6-11e6-8402-001517f91241  ONLINE	   0	 0	 0
		  mirror-2									  ONLINE	   0	 0	 0
			gptid/a720e2ad-8ca6-11e6-8402-001517f91241  ONLINE	   0	 0	 0
			gptid/a850f609-8ca6-11e6-8402-001517f91241  ONLINE	   0	 0	 0

Each harddisk is 2TB, usable space 6TB.
enabled system services:
SSH
FTP
SMB
SMART

No ZIL or L2ARC setup yet.

Full gigabit network setup.

So here's the problem, the very first day, while the pool is empty, I tried to copy and paste a 2GB and 4GB file to the pool through LAN SMB, many times, both side gives me a result of solid 110MB/s.

However since the 2nd day, reading from the pool through SMB, initially will give around 110MB/s but will drop to as slow as 20MB/s after readed about 800MB~1000MB's data.
And writing to the pool has mixing result, sometime gaves me 110MB/s, sometimes it's around 50MB/s.

I wondering if this is related to I have no ZIL (accasionally slow to 50MB/s writing)
No L2ARC (reading drops to 20MB/s)?
Or maybe the 8GB RAM is not enough for 6TB usable space (6 or 2TB disks), maybe I should get at least 10GB?

I had a quick seach of FreeNAS ZFS 20MB and seems many people having the same problem, not getting the 110MB/s expecting speed for gigabit network.

Here're some snapshots of reading from the pool and the ZFS Status report:

 

[skygunner]

43 processes: 1 running, 42 sleeping
CPU: 0.0% user, 0.0% nice, 0.1% system, 0.0% interrupt, 99.9% idle
Mem: 94M Active, 438M Inact, 7257M Wired, 87M Free
ARC: 6810M Total, 5429M MFU, 1337M MRU, 32K Anon, 24M Header, 19M Other
Swap: 12G Total, 12G Free

 

[m0nkey_]

I wondering if this is related to I have no ZIL (accasionally slow to 50MB/s writing)

You have a ZIL, that's what RAM is for.

No L2ARC (reading drops to 20MB/s)?

Wouldn't do you any good on writes, L2ARC is a read cache.

Or maybe the 8GB RAM is not enough for 6TB usable space (6 or 2TB disks), maybe I should get at least 10GB?

FreeNAS loves RAM. Put as much in your box as possible. If you can go to 16GB, you'll see a vast improvement.

 

[skygunner]

Hi,

Thanks for the quick reply.
I might mean ZLOG? like using a SSD for writing cache.
L2ARC is read cache, that's why I thought I need one to improve the reading speed over smb.
I'm able to try out more ram later,
but I do expect to get 110MB/s over smb gigabit network starting from 8GB ram, without running much service or any activity on the box.

And why the first day I got full speed on both read and write, now later the speed drops quite a lot.
Also it seems to happen to many people.

 

[m0nkey_]

A SLOG will do nothing to improve writes speeds using Samba. A SLOG is only useful for synchronous writes. SMB only does asynchronous.

 

[SweetAndLow]

What is the client you are testing with?

Sent from my Nexus 5X using Tapatalk

 

[skygunner]

Hi,

It was copying file to a windows 8.1 instance.
Have tried more tests and it turned out the local intel RAID 10 setup is the problem.
Somehow the initial setup is good for 110MB/s while writing to this RAID 10 volume, but now it will just slow down to around
20MB/s after around 800MB's data been written.

I tried to copy file and write to local SSD drive and it was back to normal speed.

 

[logan893]

The first 800 MB at line speed (110 MB/s) is first filling your Windows client's write cache. The write cache will utilize unused RAM. As you already figured out it's then down to the underlying storage to write it out. If it can't keep up you'll see exactly this kind of reduced write speed after a while.

What's your RAID 10 array made up of?