Strange performance with a pool of 4 disks

[potope]

Hello,
I need some help to understand why the speed of data transfer from a windows client to FreeNAS (8.0.4 p x64 p2 ) varies significantly if I change the number of disks in a zfs pool.

The hardware I use is an hp* microserver populated with 4 2TB hard drive (Seagate ST2000DM001)

If I create a zfs pool (raidz1) which consists of 4 hard disk and start a copy from a windows client to FreeNAS (smb or iscsi) of a file of about 15 GB I see an average transfer rate of about 1 megabyte per second.

However, if I create a zfs pool (raidz1) composed of 3 hard drives and I make the same test I see an average transfer rate of approximately 80 megabytes per second.

I find the results of the second test much more satisfactory than those offered by the first and I would like to understand what's wrong with a raidz1 zfs pool made of 4 hard drives.

In other words where I'm wrong, what rule I have not have taken into account?

*http://h10010.www1.hp.com/wwpc/me/e...16-4237917-4237917-4248009-5153477.html?dnr=1

 

[gpsguy]

Just go with an odd number of disks for your RAIDZ configuration.

See this thread for details: http://forums.freenas.org/showthread.php?16-Getting-the-most-out-of-ZFS-pools

 

[paleoN]

You get an 80:1 improvement going to a 3 disk from a 4 disk raidz1 pool? There is definitely something else going on. Have you looked at the smart data on that 4th drive? You could always benchmark the drives individually when they are not part of any pool or mirror.

 

[Brand]

Just go with an odd number of disks for your RAIDZ configuration.

See this thread for details: http://forums.freenas.org/showthread.php?16-Getting-the-most-out-of-ZFS-pools

That would not account for such drastic differences in speed.

 

[potope]

@Brand
Thank you for the link you gave me it's worth reading, I hope to finish reading the solaris best practice guide over the week end, I will learn somethinng for sure!

@paleoN
I've checked the SMART data and every hard disk seems fine, in this regard I can give you more detail in order to exclude more confidently that it could be a disk fault.

- I've setup two machine (2 HP with ~8TB of raw data available, 4 hd 2TB ) and the test results are consistent between those two.

- Since I've some doubt too about the hard disks during the tests in zfs1 (3 hdd) I 've mixed the hard disks to see if there would be some change in performance but remain the same however I make the pool. My bad I didn't writ it first.

 

[paleoN]

What's the actual hardware in those servers? Do they have onboard raid? If so how are you configuring it?

@paleoN
I've checked the SMART data and every hard disk seems fine, in this regard I can give you more detail in order to exclude more confidently that it could be a disk fault.

- I've setup two machine (2 HP with ~8TB of raw data available, 4 hd 2TB ) and the test results are consistent between those two.

- Since I've some doubt too about the hard disks during the tests in zfs1 (3 hdd) I 've mixed the hard disks to see if there would be some change in performance but remain the same however I make the pool. My bad I didn't writ it first.

I take it you are still testing then? If both servers are the same then it's not the disks themselves. Try the following.

• On one server destroy your zpool.
• Open 4 separate SSH sessions.
• su to root in all of them.
• Run the following dd commands in the first 3 SSH sessions, one per session at the same time.
  [size=+1]This will destroy data on your disks.[/size] Only run it on disks not in an array.
  Code:

  dd if=/dev/zero of=/dev/ada0 bs=2048k count=50k
  dd if=/dev/zero of=/dev/ada1 bs=2048k count=50k
  dd if=/dev/zero of=/dev/ada2 bs=2048k count=50k

Assuming your drives are adaX otherwise change as appropriate. I have the same hard drives and it should take more than 500 seconds & less than 600 seconds to run against the raw drives. In other words about 10 minutes. Once it is finished, in the fourth SSH session run this all by itself:

Code:

dd if=/dev/zero of=/dev/ada3 bs=2048k count=50k

Wait until finished. If it takes about the same time as the others then try running all the dd commands all at the same time.

 

[potope]

What's the actual hardware in those servers? Do they have onboard raid? If so how are you configuring it?

The server use a Turion II neo n40l dual core with 4096 mb of ram have an on board raid but I don't use it.

I take it you are still testing then? If both servers are the same then it's not the disks themselves.

Yes I'm still testing I really like to understand the performance I can expect from this hw + freenas.

Test in 3 ssh concurrently

ada0
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada0 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 622.074732 secs (172606565 bytes/sec)

ada1
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada1 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 622.243872 secs (172559646 bytes/sec)

ada2
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada2 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 630.333907 secs (170344925 bytes/sec)

---
ada0 alone

ada3
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada3 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 656.253918 secs (163616825 bytes/sec)

-----------------------------
Four disk concurrently

ada0
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada0 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 744.453586 secs (144232205 bytes/sec)

ada1
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada1 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 746.280764 secs (143879070 bytes/sec)

ada2
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada2 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 749.480188 secs (143264871 bytes/sec)

ada3
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada3 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 758.663547 secs (141530699 bytes/sec)


Today I've found a bios option that allow to enable a cache on write.
I've done more test in Windows with a file of 15 980 266 496
3.34 windows cache on (71mb/s) zfs1 3 disk
5.47 Windows cache on (43mb/s) zfs1 4 disk

Thank you for reading !!

 

[William Grzybowski]

Did you check 4k sector size while creating the pool?

 

[potope]

Did you check 4k sector size while creating the pool?

Yes I've checked that but I don't see an improvement when I force 4096 bytes sector size, and the documentation says that the settings is rarely needed (http://doc.freenas.org/index.php/Volumes#Volume_Manager) since freenas uses 4K sector for ZFS if the underlying hard drive is advanced format.

Thanks

 

[gpsguy]

Since your n40l supports 8Gb of RAM, I would bump at least one of them up to 8Gb and retry your tests.

The server use a Turion II neo n40l dual core with 4096 mb of ram

From the doc's

"The best way to get the most out of your FreeNAS™ system is to install as much RAM as possible. If your RAM is limited, consider using UFS until you can afford better hardware. ZFS typically requires a minimum of 6 GB of RAM in order to provide good performance; in practical terms (what you can actually install), this means that the minimum is really 8 GB. The more RAM, the better the performance, and the Forums provide anecdotal evidence from users on how much performance is gained by adding more RAM. For systems with large disk capacity (greater than 6 TB), a general rule of thumb is 1GB of RAM for every 1TB of storage.

NOTE: by default, ZFS disables pre-fetching (caching) for systems containing less than 4 GB of usable RAM. Not using pre-fetching can greatly reduce performance. 4 GB of usable RAM is not the same thing as 4 GB installed RAM as the operating system resides in RAM. This means that the practical pre-fetching threshold is 6 GB, or 8 GB of installed RAM. You can still use ZFS with less RAM, but performance will be affected. "

 

[potope]

Since your n40l supports 8Gb of RAM, I would bump at least one of them up to 8Gb and retry your tests.

You're completly right, every time I reboot freenas I see a message which states that ZFS is disabling pre-fetching on my system.
Previously I've done some test with 8gb of ram but since I've not seen any improvement I've taken out the ram of freenas.
As soon as I can I'll redo all the test with 8GB of RAM.

 

[paleoN]

Did you check 4k sector size while creating the pool?

potope I would listen when William suggests something. Did you confirm that they are formatted 4k even if you don't check the box?

From the command line do:

* zpool set cachefile=/data/zfs/zpool.cache tank (change tank to your pool name)
* zdb -U /data/zfs/zpool.cache | grep ashift

* If the answer = 9, you have 512 byte sectors.
* If the answer = 12, you have 4k byte sectors.

It is a significant performance hit to use 512 sectors on 4k sector drives.

 

[paleoN]

The server use a Turion II neo n40l dual core with 4096 mb of ram have an on board raid but I don't use it.

What's the motherboard & raid controller?

Four disk concurrently

ada0
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada0 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 744.453586 secs (144232205 bytes/sec)

ada1
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada1 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 746.280764 secs (143879070 bytes/sec)

ada2
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada2 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 749.480188 secs (143264871 bytes/sec)

ada3
[root@hpnas] ~# dd if=/dev/zero of=/dev/ada3 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 758.663547 secs (141530699 bytes/sec)

That's actually not nearly as bad as I thought it was going to be. Ideally they would have been as fast as a single disk though. Do you have a spare controller card you can test with?

 

[potope]

potope I would listen when William suggests something. Did you confirm that they are formatted 4k even if you don't check the box?It is a significant performance hit to use 512 sectors on 4k sector drives.

Here's a test forcing 4096 bytes sector size
[root@hpnas] ~# zpool set cachefile=/data/zfs/zpool.cache rama
[root@hpnas] ~# zdb -U /data/zfs/zpool.cache | grep ashift
ashift=12


Here's the same test without forcing
[root@hpnas] ~# zpool set cachefile=/data/zfs/zpool.cache tilli
[root@hpnas] ~# zdb -U /data/zfs/zpool.cache | grep ashift
ashift=12

So disk are formatted 4k even if I don't check the box since the disks have 4096 byte per Sector

Anyway I'm not dismissing anyone help, if it seems so to you could be because I'm not a native english speaker and every post is a battle with google translate.

 

[potope]

What's the motherboard & raid controller?That's actually not nearly as bad as I thought it was going to be. Ideally they would have been as fast as a single disk though. Do you have a spare controller card you can test with?

Unfortunately I do not have a spare controller to do other tests, I hope the upgrade to 8gb of ram can bring some improvement but I have not yet had the time to do the test.
The raid controller is integrated into the motherboard but disabled from bios, and the motherboard is the one provided by default with the microserver n40l hp, I have not changed, what should I check?

 

[paleoN]

Anyway I'm not dismissing anyone help, if it seems so to you could be because I'm not a native english speaker and every post is a battle with google translate.

You are using google translate admirably. I don't think you are being dismissive. William is one of the FreeNAS developers so I usually try whatever he says.:)

I don't know what motherboard comes with the microserver n40l hp or what the integrated controller is. Is AHCI enabled for the hard drives in the bios?

 

[ntshad0w]

potope, mates,

I'm not a zfs or unix master but...
I observe probably same behaviour in freenas 8.2.0 (x64) on the hp microserver with 5x disks raid-z (best performance ratio from raid perspective), write transfers was around 1MB/s (with raid controller without BBWC), all was good on earlier freenas versions, I don't test 3 disks, but for me it looks like some bug in Freenas v8.2.0 x64, evidently situation was ok on some earlier versions, probably 8.0.2 x64, but when I add BBWC, write transfers jump 50-75MB/s (50 for NFS, 75 for CIFS), when I disable zil (not recommended, tunable adv option vfs.zfs.zil_disable = 1 ) transfers go up to 90-110MB/s.
I have 8 GB of ram in my hp microserver and (now) hp p410 1gb with fbwc/bbwc (without RAID controller the situation is the same)
additionally, can you test 5x hdd?, 5'th one to hp microserver you can add with little invention on the bottom in 5.25" case box or just buy such case and use a sata cable from mainboard :))

regards
NTShad0w