Strange problem with ZFS

[gswa]

Hello.

I've got some strange things happen on my new server.
Hardware is Intel C202 based SuperMicro 1U server + Intel Pentium G620 + 8GB of RAM + 4x3TB SATA Seagate 7200rpm HDD + 8GB usb stick with FreeNAS
FreeNAS 8.04 64-bit

So when I create volume from all 4 disks with ZFS RAIDZ1 and share it I get good speed (around 800Mbps) but with transfer stops: http://gswa.ru/files/050412/rzzfs.png .
I make some tests and looks like it's not samba and not sata subsystem.
There is graph of two 4GB+ transfers (writes) to UFS stripe volume of those 4 disks: http://gswa.ru/files/050412/stripeufs.png near to ideal.

I thinked that is an ZFS problem, but RAIDZ1 of 3 disks on this system works good.

Really need 4x3TB raid. Please help to find what to do else.

 

[gswa]

here is write test results:

3xhdd @ RAIDZ write:
dd if=/dev/zero of=/mnt/vz/tmp.dat bs=2048k count=50k 51200 0 records in
51200 0 records out
107374182400 bytes transferred in 351.831403 secs (305186466 bytes/sec)

4xhdd @ RAIDZ write:
dd if=/dev/zero of=/mnt/vz/tmp.dat bs=2048k count=50k
51200 0 records in
51200 0 records out
107374182400 bytes transferred in 1240.136733 secs (86582535 bytes/sec)

 

[Hexland]

If you're making a vdev of ada0,1,2 and its working fine, then repeating with ada0,1,2,3 and its slow... try making a 3 disk vdev of ada1,2,3

See if disk ada3 is the problem thats slowing stuff down...

Make sure that all the disks that need it are formatted for 4k sectors (my speed halved because I was using an advanced format drive with 512k formatting)

You could also try making 4 x ZFS volumes separately, and profiling the read/write speed to see if you can identify if the drives themselves have any problems...

also check dmesg | grep ada to see if theres any oddities in the ATA transfer modes (one drive might be seen as PIO instead of DMA, or something like that)

 

[gswa]

thanks for your reply. i've checked this possibility earlier. it's not any of disks. look at my ufs stripe of 4 disks graph.

 

[Hexland]

My bad... I didn't read the original post properly :)

Interesting though -- I am experiencing something similar, and just posted my quick tests last night

http://forums.freenas.org/showthrea...cro-Tower-Server&p=25461&viewfull=1#post25461

I didn't try less than 6 disks with ZFS -- I might give that a go tonight.
I was working under the assumption that the problem was that the CPU was being hammered during write cache flushes, but your experiences with 3 disks doesn't back that assumption (at least in no way I can think of)

I'll try the same configuration as you tonight, to see if I get the same results.

 

[gswa]

update on my problem.

now testing same system with FreeBSD 9.0 x64.

4xHDD RAIDZ - dd write test - 385MB/s !!!
samba write around 80MB/s and stable

need some time for more tests

 

[ProtoSD]

I wonder if the FreeBSD 9.0 results are ZFS v28 related. Have you upgraded your pool, or did you just create a v28 pool for testing?

I have a FreeNAS 8.03-p1 build based on FreeBSD 9.0 x64 available if you want to give that a try.

 

[gswa]

I think it's in the way FreeNAS partitioning drives. When I installed FB9 I looked at partiton map on one of ZFS pool drives. There was 2GB freebsd-swap partition and ZFS partition. This with kernel tuning and older ZFS version can be the reason for this problem.

I created new pool in FB9. Sorry but currently I can't do more tests on that system - it's already charged with 3TB of data and continues to fill. Some time later I hope we will build another one with 4TB drives :)))