10T, 10Million 8K files, 10 servers, 50 users,nfs lockd problem

[unicorncn]

Storage:
FreeNAS-8.3.1-Release-X64
Supermicro X9DRI-LN4F Intel E5-2620
DDR3-1600 ECC LSI 9211-8i 32GB
Intel 520 120G 7.2K Drives*17
Intel Pro 1000MT Quad Port
LAN: 6 port with mode 0 bonding
Servers:
RHEL 5.5 x64
LAN: 2 port with mode 0 bonding
switch:
H3C 5700 Gigabyte switch

Storage pool
16 2T HDD RAID10 with 16TB size, zfs, compress=lzjb, atime=off
snapshot every day and keep 2weeks， every weekend keep 2 months, total about 600
snapshot.

Total is about 10 million files, 200 thousands folders, 80% data is about 1~4K small files, ave size is about 8K. Total size is about 10TB. 60% write & 40% read.

Every user have their datasheet, and mount on every servers separate.

The read and write is seem OK, IOPS can top to 5K, but there have lockd problem several times. The client have no respond when access their folder, and there has a large "Server Ret-Failed" when show "nfsstat -s" on storage, and error "kernel: lockd: server not responding, timed out " on servers side.

After increate "numbers of servers" , it can clam the no respond problem, now the numbers of servers is 200.

If any way to resolve such problem?
If there any problem when close lockd with -o nolock? I have set this on one server, but some application need lock function.
If any advice to improve performance with such surrounding more. We need more 30% performance last half year.

Thanks a lot.

BTW, I have another pool hardware backup storage , it work well with zfs and gzip. It can do full backup in 24Hour when old backup need 3 days when use ext4.

 

[dlavigne]

It looks like that this is a known issue in FreeBSD and a fix may get MFC'd to FreeBSD 9.1. Restarting statd can usually unwedge it. If that doesn't, a reboot will.

 

[unicorncn]

Yes, I have to reboot to calm it . Cannot even restart statd to resolve it, it will hang when restart nfs.

When the FreeNAS 9.1 stable release?

Thanks a lot, dlavigne.

 

[dlavigne]

The beta period starts this month. Barring any major bugs, we're hoping for a fairly quick beta/RC cycle prior to release.

 

[unicorncn]

Good new, I'll test it on my backup server first when beta release. Thanks.

 

[dlavigne]

Just to be clear, this fix is not in 9.1 yet, but may get MFCd to it (merged from FreeBSD current). Though, once FreeNAS 9.1 is released, it should provide many other benefits. If 9.1 does not fix your issue, it is worthwhile to create a support ticket to let the devs know that the problem still affects users.

 

[unicorncn]

Get it, thanks a lot. I maybe do some test with nfsv4 to avoid such problem, and try to get more performance.