RaidZ2 write performance < 10MB/s

[marcevan]

Now that I destroyed a bad-design (well, bad implementation - spare 1TB was inadvertantly put in as full striped) ZFS pool, I re-did the pool (wiping disks first) as a 6 drive RaidZ2.

I now have only 6 disks attached to the 6 SATA ports as AHCI and pool status is great, and no discernable issues from SMART on any drive. They are all WD Red 5400rpm 3-TB drives.

Running 9.1.1. X64 with 16GB RAM.

The data that I'm moving onto the pool is on a drive attached to Ubuntu client I use for work. It's on the same switch as the FreeNAS box and both have verified Gigabit cards in them and the switch is gigabit as well.

Speedtest.net from ubuntu gets my paid for 80MB/s but I'd expect within the house, I would be closer to 100MB/s box-to-box with gigabit ports.

I'm pushing ~80GB of music to the FreeNAS at a underwhelming 7.4MB/sec.

I have confirmed write caching is on in BIOS and on for each drive thru camcontrol identify on each of the 6 drives.

Not really sure what to test next, since individually each box reports it's got full duplex on the gigabit port and I'm rediscent to perform DD or ifperf while the transfer is going on.

 

[cyberjock]

write caching is a bad idea for ZFS. You should disable that as ZFS does its own much smarter write cache.

I will tell you that moving small files is MUCH slower than moving large files. I'd think that if I were moving my mp3 folder I'd get about the same speed.

What CPU do you have and what NIC?

 

[marcevan]

cj,

Thanks as I think I mis-read in other posts that one should turn ON write cache. I'll do a reboot when the mp3 transfer finishes and kill write cache in the BIOS.

For CPU it's an AMD Phenom X2 550
For NIC it's an Intel card on em0 thru PCI-E. Confirmed it's 1000 full duplex.

 

[cyberjock]

Hmm... that CPU should be good enough. The NIC obviously is. I don't have any other ideas. Small file transfers are always a pain as throughput is significantly lower. My guess is you're just going to have to tough through it. :(

 

[Joo Chung]

Have you considered breaking up the 80GB of files into multiple smaller sets of concurrent transfers? e.g. 2 or 4 concurrent transfers of 40GB or 20GB? And see if you can get a higher aggregate throughput?

 

[marcevan]

I've got about 17GB to go and MP3 are the smallest files I have. About 43 more minutes so I'll finish, reboot, BIOS off the write-cache, and see how the rest goes:

I have a number of decent sized video files to transfer next as I already did my photos and that ran 17GB at ~13.4 MB/sec.

 

[cyberjock]

For small files that will normally hurt performance even more. The issue is that the head is spending more time seeking than reading, so your total throughput is crap. All you'll do is add more concurrent seeking which will hurt performance even more. I'd just wait it out.

 

[Joo Chung]

Thats assuming his ZFS spool is the problem and not the network or the protocol. Has he definitively ruled out either of those?

 

[cyberjock]

Actually, its probably not his pool but the drive he's having to read off of plus the overhead of sharing protocols. ZFS intelligently caches and commits writes. ;)

 

[Joo Chung]

I just reread the opening post, and I think you are probably right. Any bets the drive on his Ubuntu client is a USB 2.0 drive?

 

[cyberjock]

That's very possible. I hadn't really thought of that because I know moving small files even on internal disks just sucks... I've done it too many times to count and I hate it every time. The "time remaining" fluctuates like crazy and you have no idea if it has an hour or 8 hours to go.

 

[Joo Chung]

I agree about small files. But audio files, while they are typically not huge, they are also not exactly small.

 

[cyberjock]

No, but reading 3MB and it taking multiple seeks because you must do file system data really sucks.

 

[SmallGuy]

Are you using rsync?
How is the cpu load?

 

[marcevan]

I'm not using rsynch and CPU load on both Ubuntu and FreeNAS are light (~10%).

Disk on Ubuntu is SATA connected to the motherboard.

Network: Both machines have gigabit ethernet confirmed and are both on the same gigabit switch.

Now I rebooted and BIOS absolutely does not have any settings for turning off disk caching.

Is there a command line I can use?

 

[cyberjock]

No. You probably don't have a disk cache then.

 

[marcevan]

Rats, because camcontrol showed disks with cache on.

And my videos - around 1.4GB each, are now doing 7.2MB/sec

 

[marcevan]

Camcontrol inquiry on ada0:

[Code]
Feature Support Enabled Value Vendor
read ahead yes yes
write cache yes yes
flush cache yes yes[/Code]

 

[Joo Chung]

So how are you copying the files over? NFS?

 

[marcevan]

Ubuntu shows SMB:\ to the directory.

So I grab my files and drag them over to the appropriate destination folder.