Slow RAIDZ1 with 4xWD Advanced Format drives

[offender]

I have the following setup:

Proliant N40L (AMD Turion Neo2 - dual core 1.5ghz)
8Gb ECC RAM (2x4Gb Kingston 10600 1333Mhz sticks
8Tb HDD (4x WD20EARX Western Digital Advanced Format 2Tb drives. *No* jumpers)
BIOS settings have the drive write cache disabled.
(Gigabit LAN with TPLink switches and Realtek 8111B on the machine I am testing from and a BroadCom NIC on the Proliant)

I am running FreeNAS 8.0.2

I have set up a RAIDZ1 pool which has given me 5.2Tb of usable space. I checked the 4k advanced format box and get this when I check:
zdb -U /data/zfs/zpool.cache | grep ashift
ashift=12


I am finding the following write speeds *before* doing the basic tuning from the freenas wiki (approx 57.8 Mb/s)
dd if=/dev/zero of=/mnt/datashare/ddfile bs=1024k count=16000
16000+0 records in
16000+0 records out
16777216000 bytes transferred in 290.403970 secs (57771993 bytes/sec)

FileZilla transfers a 21.4Gb file in 599 seconds (very bursty) - approx 35Mb/s

I am finding the following write speeds *after* doing the basic tuning from the freenas wiki (approx 60.7 Mb/s)
dd if=/dev/zero of=/mnt/datashare/ddfile bs=1024k count=16000
16000+0 records in
16000+0 records out
16777216000 bytes transferred in 276.172377 secs (60749073 bytes/sec)

FileZilla transfers a 21.4Gb file in 525 seconds (fairly bursty) - approx 40Mb/s

(Read speeds are approx 175Mb/s before tuning and 279Mb/s after tuning using dd locally)
(I didn't do a Filezilla transfer before tuning but after tuning the read was 21494Mb in 212 seconds - 101Mb/s - very smooth)

I am wondering if this write issue could be due to the Advanced Format drives I have setup? I have done nothing other than use the GUI to setup the RAIDZ

 

[kashiwagi]

I have exactly the same drives in an N36L without any tuning, which gets about 200MBps, so there is definitely something odd going on here.

[***@***] /mnt/volume1# dd if=/dev/zero of=/mnt/volume1/ddfile bs=1024k count=16000
16000+0 records in
16000+0 records out
16777216000 bytes transferred in 81.582093 secs (205648266 bytes/sec)

 

[Milhouse]

@offender: Careful with your Mb/s (bits) and MB/s (bytes) - I'm assuming you mean mostly MB/s in your post...

Either way you're right to be concerned, those disk I/O figures are low - why though, not sure.

Did you perform manufacturer diagnostic tests on all your disks before installation? Are they all working fine without errors? What's the status of the pool, are all the disks online?

 

[Digidoc]

As silly as this may sound, try replacing the SATA cables. I had a similar experience on both the desktop I am currently writing this on, and also on my own FreeNAS box when I first built it.

Not all SATA cables are created equal. When I first built this HP DC7900 system (w/two drives in RAID-0), it would work fine if I left the transfer speed set to SATA-I (1.5Gb) speeds. If I changed the BIOS settings to enable SATA-II, everything went to hell in a hand basket even though the drives were capable of 3Gb speeds. The machine would randomly blue-screen and/or corrupt files (that's when it would actually make it into Windows). It turns out that even though this is (or at least was when it was initially purchased) a brand new system, HP cheaped out on the SATA cables and put cables that were only good up to SATA 1.5Gb in it. I replaced both cables with new SATA cables that were rated to SATA-III speeds and it's been running like a banshee since.

As for my FreeNAS box, when I first built it I noticed that randomly it would freeze up during transfers as well as bog down for no apparent reason whatsoever. I replaced all the SATA cables in it and it resolved the problem. I've since ended up installing an 8-port SAS/SATA card with it's own cabling, but it's still running great.

Now.... I'm not saying that this is the same problem you're experiencing, but it'd be the first thing I'd check.

 

[offender]

Thanks for the replies.

kashiwagi
- do you have 'drive write cache' turned on in the BIOS for your N36L?
- I notice you have 5x2Tb drives (presumably in RAIDZ1?). I have noticed threads here and there talking about how 4x2Tb drives is bad because ZFS spreads 128kb segments across the drives inefficiently except in multiples of 3/5/9 drives for RAIDZ1 on 4k sector layouts.
- How did you get the fifth drive in there - in the ODD bay?

I have turned drive caching in the BIOS on (the server is connected to a UPS) and now I get around 148Mbytes/s
dd if=/dev/zero of=/mnt/datashare/ddfile bs=1024k count=16000
16000+0 records in
16000+0 records out
16777216000 bytes transferred in 113.092272 secs (148349801 bytes/sec)

Transfers via CIFS and FTP seem to saturate the gigabit ethernet rather than being jumpy. An FTP'ed 28gb file transferred at 90MBytes/s

Digidoc
- The microserver comes with a fairly proprietary SAS to SATA cable system from what I can tell. I will have a closer look to see whether the mini SAS cable can have new SATA cables put into it though.

Milhouse
- I have only run the quick tests on each drive with the WD Data Lifeguard tool. I'll run the full tests this weekend.

 

[Digidoc]

Digidoc
- The microserver comes with a fairly proprietary SAS to SATA cable system from what I can tell. I will have a closer look to see whether the mini SAS cable can have new SATA cables put into it though.

If that's the case it's pretty unlikely that it's the culprit to be honest, but if it is, you can get them pretty cheap from Monoprice.com. Most likely it's SFF-8087 to 4xSATA forward cable. I just bought two .75 meter SAS cables for about $8 each. You can see what the connector looks like here:

http://en.wikipedia.org/wiki/Serial_attached_SCSI

 

[Milhouse]

The cable can't easily be replaced - it's pretty much a custom SFF8087 to 4 Amphenol SATA data/power connectors that each bolt to the rear of the drive cage. Some users have attempted to replace the cable, however.

 

[kashiwagi]

I can't remember if I changed the write cache or not. I don't have a monitor available to hook up right now. Is there any way to check it from the OS?

As for the fifth drive, I just put it in the ODD bay. Just remember to flash to a patched bios so that you can get full speed on the ODD data port. Also, you have to buy both power and sata cables, as they are not included.

 

[offender]

I can't remember if I changed the write cache or not. I don't have a monitor available to hook up right now. Is there any way to check it from the OS?

Yes there is (and I have confirmed the difference by disabling drive write cache in the BIOS and re-running the command(s).

You can do either of the following:
ataidle /dev/ada0

...
SMART Enabled: yes
Write Cache Supported: yes
Write Cache Enabled: yes
...


or

camcontrol identify /dev/ada0

...
Feature Support Enabled Value Vendor
read ahead yes yes
write cache yes yes
...

I've run that on all four drives with and without write-caching enabled in the BIOS and the 'enabled' part for either ataidle or camcontrol is set accordingly with the BIOS setting.

I am interested to know what it is for your drives?

Here is the total camcontrol output for one of my drives:

Code:

camcontrol identify /dev/ada0
pass0: <WDC WD20EARX-00PASB0 51.0AB51> ATA-8 SATA 3.x device
pass0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)

protocol              ATA/ATAPI-8 SATA 3.x
device model          WDC WD20EARX-00PASB0
firmware revision     51.0AB51
serial number        ***************
WWN                   ***************
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 4096, offset 0
LBA supported         268435455 sectors
LBA48 supported       3907029168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6

Feature                      Support  Enabled   Value           Vendor
read ahead                     yes      yes
write cache                    yes      yes
flush cache                    yes      yes
overlap                        no
Tagged Command Queuing (TCQ)   no       no
Native Command Queuing (NCQ)   yes              32 tags
SMART                          yes      yes
microcode download             yes      yes
security                       yes      no
power management               yes      yes
advanced power management      no       no
automatic acoustic management  no       no
media status notification      no       no
power-up in Standby            yes      no
write-read-verify              no       no
unload                         no       no
free-fall                      no       no
data set management (TRIM)     no

 

[kashiwagi]

camcontrol identify /dev/ada0
pass0: <WDC WD20EARX-00PASB0 51.0AB51> ATA-8 SATA 3.x device
pass0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)

protocol ATA/ATAPI-8 SATA 3.x
device model WDC WD20EARX-00PASB0
firmware revision 51.0AB51
cylinders 16383
heads 16
sectors/track 63
sector size logical 512, physical 4096, offset 0
LBA supported 268435455 sectors
LBA48 supported 3907029168 sectors
PIO supported PIO4
DMA supported WDMA2 UDMA6

Feature Support Enabled Value Vendor
read ahead yes yes
write cache yes yes
flush cache yes yes
overlap no
Tagged Command Queuing (TCQ) no no
Native Command Queuing (NCQ) yes 32 tags
SMART yes yes
microcode download yes yes
security yes no
power management yes yes
advanced power management no no
automatic acoustic management yes no 254/0xFE 128/0x80
media status notification no no
power-up in Standby yes no
write-read-verify no no
unload no no
free-fall no no
data set management (TRIM) no

 

[offender]

ok cheers for that - it looks like you do have write caching enabled in the BIOS. I guess that the difference in speed (my 150 mb/s vs your 200 mb/s) may possibly be due to the modified BIOS you're using and/or the 4 vs 5 drives you're using.

I am thinking of adding an extra drive in and re-doing my RAIDZ but at the moment with 150mb/s writes and 260-280mb/s reads I am fairly happy with things.

So basically the solution for me was enabling write caching in the BIOS. (and I have the server on a UPS which should help - though technically HP also recommend having a redundant power supply setup though I am not sure if the microserver can accommodate that or not)

 

[jgreco]

No, the ProLiant Microserver does not support a redundant power supply.

 

[offender]

No, the ProLiant Microserver does not support a redundant power supply.

OK thanks for that. I would assume most people would run their home-based FreeNAS8 boxes with write caching on the drives enabled though regardless of redundant power supply installation?