3 different HDDs max out at 10mb/s while SSD's are 300mb/s using dd if=/dev/zero without compression

[plator93]

New to FreeNAS, thank you all for any help.

Specs:
Xeon 5607 2.26 x 4
16GB DDR3 ECC
2 x 128gb SSD
1 x 512gb SSD
3 x WDC WD40EZRX

All drives are SATA 3. The onboard controller is SATA 2.

Using the SSD drives, the performance is as expected. Somewhere around 300mb/s per drive locally (the SATA 2 limit) and they max out the 2 gigabit lan connections as expected. On the HDD's however, the system cannot seem to get above 10MB/s per drive. raid, stripe and single drive performance match.

I set up the drives as a 3x mirror initially noticing very poor performance in the 10-30mb/s range. Worried about a single drive failure or performance issue, I deleted the mirror and created three single drive volumes to test. ZFS compression is off.

WD40EZRX striped: (all three drives in a stripe)
[root@freenas] /mnt/stripe# dd if=/dev/zero of=./test bs=4M count=1k
4294967296 bytes transferred in 136.939569 secs (31,363,961 bytes/sec)

WD40EZRX single drives: (tested each drive on its own, performance is identical on each drive)
[root@freenas] /mnt/a0# dd if=/dev/zero of=./test bs=4M count=1k
4294967296 bytes transferred in 341.708772 secs (12,569,087 bytes/sec)

SSD single 128gb:
[root@freenas] /mnt/ssd/test# dd if=/dev/zero of=./test bs=4M count=1k
4294967296 bytes transferred in 14.545375 secs (295,280,614 bytes/sec)

SSD 2 x 128gb striped:
[root@freenas] /mnt/ssdstripe# dd if=/dev/zero of=./test bs=4M count=1k
4294967296 bytes transferred in 7.370052 secs (582,759,421 bytes/sec)


Read performance is identical. CPU usage is 99% idle.The web gui reporting tab shows a consistent 10mb/s on the interface when dd is being run. Image attached.

Looking in to the SMART data, there are no faults. The drives were performing as expected on this server using windows server os at somewhere around 90mb/s sequential. These WD green drives aren't very fast but I don't have a lot of data and would like to use them for sequential transfers only (backup archives) and prefer reliability to performance.

At this level of performance the drives act as if they do not have their write cache enabled. I would like to run the drives in a 3 x mirror configuration.

Any advice on what to try? There is no data on the system so I am up for anything.

 

[enemy85]

Hi, the sata2 vs sata3 thing doesn't matter, so is nor that the problem. My first guess would have been a faulty disk, but u said u already checked them so neither that is a possibility...
Just a try: change the ssd sata ports with the hdd ones and see if something change. U didn't say anything about the mobo and how disks are connected...

 

[cyberjock]

How about instead of writing to the disks, reading from them. And instead of reading from the zpool of the disks (which none of us are able to validate aren't flawed in some fashion) you read from /dev/adaX and write to /dev/adaX.

Not to hound you for inaccuracies, but your title says '3 different HDDs max out at 10mb/s' and you are wrong on two fronts:

1. The do not do 10mb/s. They do 10MB/sec. BIG DIFFERENCE when talking about benchmark values. BIG difference.
2. Your test are not testing the hard drives. You are testing a zpool, which may or may not be a single disk. Even if it is a single disk, you are still not testing a hard drive. You are quazi-testing the hard drive, but through the zpool.

Please be accurate with your claimed tests and their results. Testing a zpool of a single disk is not the same as testing a single disk. We can't help you if you can't accurately and precisely convey your problem and back it up with information that we can verify.

 

[plator93]

So many great ideas!

Just a try: change the ssd sata ports with the hdd ones and see if something change. U didn't say anything about the mobo and how disks are connected...

I swapped one of the bays from a SSD to HDD and the problem stayed with the drive.

How about instead of writing to the disks, reading from them.

And instead of reading from the zpool of the disks (which none of us are able to validate aren't flawed in some fashion) you read from /dev/adaX and write to /dev/adaX

Oooh I didn't think of using the block device. Without the file system in the way, we don't have to worry about cache either.

Write test
[root@freenas] ~# dd if=/dev/zero of=/dev/ada0 bs=4M count=250
1048576000 bytes transferred in 95.896767 secs (10,934,425 bytes/sec)

Read test
[root@freenas] ~# dd if=/dev/ada0 of=/dev/null bs=4M count=250
1048576000 bytes transferred in 7.388218 secs (141,925,425 bytes/sec)

It looks like the drives are reading at the correct speed. It kinda feels like the hard drive block write cache is disabled for some reason. There aren't any options on the BIOS for write cache control. It's an ICH10 controller. The hard drives are under devices ada0 - ada2.

[root@freenas] ~# camcontrol identify /dev/ada0
pass0: <WDC WD40EZRX-00SPEB0 80.00A80> ATA-9 SATA 3.x device
pass0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)

protocol ATA/ATAPI-9 SATA 3.x
device model WDC WD40EZRX-00SPEB0
firmware revision 80.00A80
serial number *
WWN *
cylinders 16383
heads 16
sectors/track 63
sector size logical 512, physical 4096, offset 0
LBA supported 268435455 sectors
LBA48 supported 7813971633 sectors
PIO supported PIO4
DMA supported WDMA2 UDMA6
media RPM 5400

Feature Support Enabled Value Vendor
read ahead yes yes
write cache yes no
flush cache yes yes
overlap no
Tagged Command Queuing (TCQ) no no
Native Command Queuing (NCQ) yes 32 tags
NCQ Queue Management no
NCQ Streaming no
Receive & Send FPDMA Queued no
SMART yes yes
microcode download yes yes
security yes no
power management yes yes
advanced power management no no
automatic acoustic management no no
media status notification no no
power-up in Standby yes no
write-read-verify no no
unload no no
general purpose logging yes yes
free-fall no no
Data Set Management (DSM/TRIM) no
Host Protected Area (HPA) no

It looks like write cache is disabled. This setting shows up on each of the spinners. The SSDs show their write cache enabled.

For fun I added "kern.cam.ada.0.write_cache=1" to /boot/loader.conf + rebooted with no effect.

Does anyone know how to enable write cache? (my idea is to install an alternate controller card)

 

[mav@]

I may be wrong, but I have feeling that /boot/loader.conf may not work after switching to GRUB loader in 9.3. Set those tunables via respective WebUI menu. In any case that is more reliable from the point of further upgrades.

 

[plator93]

Added:
Variable: kern.cam.ada.0.write_cache
Value: 1
Type: Loader
Enabled: X

Rebooted and checked. Write cache remains off.

I am probably wrong but I read in the documentation for the kernel that write cache is enabled by default. I guess that would be why it is enabled on the SSD's. Not sure where to look to find out why the kernel (or something else?) has disabled it on the spinners.

 

[plator93]

For fun I installed a different drive in the same slot and write cache works. It looks to be specific to this model of drive.

Is this a bug?

[root@freenas] ~# camcontrol identify /dev/ada0
pass0: <ST3000DM001-1CH166 CC46> ATA-8 SATA 3.x device
pass0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)

protocol ATA/ATAPI-8 SATA 3.x
device model ST3000DM001-1CH166
firmware revision CC46
serial number *
WWN *
cylinders 16383
heads 16
sectors/track 63
sector size logical 512, physical 4096, offset 0
LBA supported 268435455 sectors
LBA48 supported 5860533168 sectors
PIO supported PIO4
DMA supported WDMA2 UDMA6
media RPM 7200

Feature Support Enabled Value Vendor
read ahead yes yes
write cache yes yes
flush cache yes yes
overlap no
Tagged Command Queuing (TCQ) no no
Native Command Queuing (NCQ) yes 32 tags
NCQ Queue Management no
NCQ Streaming no
Receive & Send FPDMA Queued no
SMART yes yes
microcode download yes yes
security yes no
power management yes yes
advanced power management yes yes 128/0x80
automatic acoustic management no no
media status notification no no
power-up in Standby no no
write-read-verify yes no 0/0x0
unload no no
general purpose logging yes yes
free-fall no no
Data Set Management (DSM/TRIM) no
Host Protected Area (HPA) yes no 5860533168/5860533168
HPA - Security no
[root@freenas] ~#

 

[cyberjock]

You are wrong.

From my FreeNAS Mini running the current build:

[root@mini] ~# sysctl -a | grep write_cache
kern.cam.ada.write_cache: 1
kern.cam.ada.0.write_cache: -1
kern.cam.ada.1.write_cache: -1
kern.cam.ada.2.write_cache: -1
kern.cam.ada.3.write_cache: -1
kern.cam.ada.4.write_cache: -1
kern.cam.ada.5.write_cache: -1

Some of the above disks are spinners, others are SATA DOMs, and others are SSDs. ;)

Besides, its enabled for all ada device via the first line. Besides that, that's not the write cache you should be worried about. The ZFS cache is what you should be concerned with, and it is most definitely enabled by default. If it weren't enabled then nobody in this forum that didn't have $10k worth of hardware wouldn't be able to write at more than about 10MB/sec to a given pool. ;)

The caching you are talking about is different, and you don't really want that enabled if its disabled by default. Some disks allow you to claim writes were committed when they weren't. So if you have a power loss or the box reboots suddenly, you'd lose writes that were allegedly made to the given disk. This would totally screw up ZFS as you'd have transactions that are supposed to be there that aren't there. So ZFS has to try to figure out what is and isnt consistent. Once it finds consistency it will rollback to that point. Unfortunately, if you have inconsistency before the most recent consistent point, your zpool will be trashed and you'll be one of those "please help me recover my data" threads. ;)

You *really* shouldn't be playing with the settings on the disk by default. You stand a very strong chance of introducing problems you are not prepared to deal with, and will be nearly impossible to recover from later when it manifests itself as some nasty problem. I can tell you that I'd never be able to prove it was a write cache being enabled on a disk no matter how much evidence you provide. It's just too deep of a level to make identification remotely possible.

 

[cyberjock]

Ok, let me add a little revision. My Reds show write cache is supported and enabled. They are a different disk (WDC WD20EFRX) so this may still be normal for you. But I think it's important to prove what the setting is by default. I'm wondering where you've used these disk before and what would have changed the setting. FreeNAS definitely does not play with these settings, so you'd have to investigate what and where these drives were used before on your own.

 

[plator93]

So we've finally figured it this out. These WD green drives, for some reason had their write cache disabled. We're not sure why or how this happened but turning the write cache back on using the ATA commands and controller commands all failed to produce a result. Eventually, we contacted WD who was of no assistance. These are OEM drives and have no warranty.

So we put the drives in a USB 3.0 WD My Book enclosure to test the performance there. And it was 35 mbyes/sec sequential write. After power cycling the enclosure, the performance more than doubled. Now, placing the drives back in the FreeNAS system results in full write performance.

The hard drive is still flagged as having its write cache turned off, however.

Hope this is handy for someone else. Obviously this is no fault of FreeNAS at all.

 

[Ericloewe]

So we've finally figured it this out. These WD green drives, for some reason had their write cache disabled. We're not sure why or how this happened but turning the write cache back on using the ATA commands and controller commands all failed to produce a result. Eventually, we contacted WD who was of no assistance. These are OEM drives and have no warranty.

So we put the drives in a USB 3.0 WD My Book enclosure to test the performance there. And it was 35 mbyes/sec sequential write. After power cycling the enclosure, the performance more than doubled. Now, placing the drives back in the FreeNAS system results in full write performance.

The hard drive is still flagged as having its write cache turned off, however.

Hope this is handy for someone else. Obviously this is no fault of FreeNAS at all.

That's probably to reduce the probability of data loss, since they came from external enclosures, prone to be abused by Joe Clueless Consumer.
Honestly, you got me thinking about that little detail - how many USB HDDs have their write caches enabled and how can one be sure that they were really flushed?

 

[plator93]

I'm not sure if every one of the enclosure sold WD green drives have their write cache disabled when directly attached. The 4TB green drive we pulled new out of the enclosure showed write cache supported and enabled when connected directly.

The write speed does change from 110mbytes/sec to about 158mbytes/sec when the drives are directly attached - this may be due to the computer we used for testing. Read speed is the same.

In the end, the only way to get the WD green drives to turn their write cache back on was to put them in a WD My Book enclosure. There must be some vendor-specific commands that it runs to configure the drive, outside of the ATA spec. Who knows? We now have a full set of working spares! :)

Thank you for your help