Hard drive testing performance

[OiD]

Hi all,

I'm playing around in feenas. Creating and destroying things... Currently I'm testing my hard drives (4TB WD Purples) with Smart and Badblocks. I've noticed a pattern in the disk reporting tab where ther IO speed steadily decreases over time, be it read or write. I've also noticed that this happens to pairs of drives. I know I'm not using sufficient RAM (6GB) and the onboard RAID is disabled. Could this be lack of RAM? or some other issue? Could it be relative to the head position on the platers?

I'm attaching a screen capture of what I'm seeing. Any feedback for this will be appreciated : )

Thanks.

 

[saikee]

I personally believe the reduction of read/write speed is related to the head position of the platers. I got the same behavior when writing 92% full a 8TB disk last night. The write speed dropped from about 200mb/s to about 130mb/s from start to finish. The outer most track is first to be read but at constant rotating speed the head must pick up a longer circumference than from an inner most tracks.

 

[OiD]

Thanks Saikee, I thought it could be that as the disk rotation is constant. Glad to hear that.

Regarding the other issue where two drives ADA1 and ADA4 are both just shy of 150MB/s and ADA2 and ADA3 are flat lining at 100MB/s for a while, any ideas what could be the cause?

I'm definatly thinking its related to the motherboard as I'm doing all 4 drives in parallel, but that might be me trying to convince myself that the drives are good...

 

[nightshade00013]

The speed over time changes because the heads are moving from the outside edge of the drive to the inside but the disk rotational speed stays the same.

Think of it this way. A tire that has a circumference of 92.6 inches will take 704 revolutions to travel 1 mile. A tire that has a circumference of 68.3 inches will have to take 956 revolutions to travel the same difference. If both are spinning at the same RPM the larger tire will travel farther in the same time period.

Optical drives are different as they speed up or slow down to keep the read/write speed the same past the laser.

As far as the flat area my best guess is that something is inhibiting the full read/write speed somewhere. May want to check cables or disconnect the drives that were able to run at max speed and swap to see if the ports being used are slower. You say you have raid turned off so it sounds like you are using a consumer board, if so you know this is considered a no no right. And if it is a consumer board the ports could be under a different controller like a marvel or who knows what. Possibly could also be an older version of SATA that maxes at a certain speed.

 

[saikee]

OID,

I could not comment on why you have 150MB/s read/write peak on one set of disks while 100MB/s on the other. However I do know if you read from a source vdev and write the data on another vdev or external target your read operation can be governed by the speed of the writing disks.

As an example I recorded about 30MB/s read from a 2x3 disks mirror Vdev, all 7200rpm, and about 200MB/s write speed on a single disk mounted as a separate volume. This confirms ZFS was reading all 6 disks evenly to write on a single disk which was spinning at 5400rpm. Thus my zpool is capable of much faster reading speed but the bottle neck is the slow single the data was written on.

Generally you need to correlate the read speed disks with the write speed disks to make sense out of the ZFS operation.

 

[OiD]

Well, finished the test, everything looks fine!

Results below;

Code:

[root@freenas] ~# smartctl -A /dev/ada1 ; smartctl -A /dev/ada2 ; smartctl -A /dev/ada3 ; smartctl -A /dev/ada4 ;
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x002f  200  200  051  Pre-fail  Always  -  0
  3 Spin_Up_Time  0x0027  173  170  021  Pre-fail  Always  -  8333
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  38
  5 Reallocated_Sector_Ct  0x0033  200  200  140  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x002e  200  200  000  Old_age  Always  -  0
  9 Power_On_Hours  0x0032  100  100  000  Old_age  Always  -  228
10 Spin_Retry_Count  0x0032  100  253  000  Old_age  Always  -  0
11 Calibration_Retry_Count 0x0032  100  253  000  Old_age  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  34
192 Power-Off_Retract_Count 0x0032  200  200  000  Old_age  Always  -  12
193 Load_Cycle_Count  0x0032  200  200  000  Old_age  Always  -  25
194 Temperature_Celsius  0x0022  119  116  000  Old_age  Always  -  33
196 Reallocated_Event_Count 0x0032  200  200  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0032  200  200  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0030  100  253  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  0
200 Multi_Zone_Error_Rate  0x0008  200  200  000  Old_age  Offline  -  0

smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x002f  200  200  051  Pre-fail  Always  -  0
  3 Spin_Up_Time  0x0027  179  178  021  Pre-fail  Always  -  8016
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  38
  5 Reallocated_Sector_Ct  0x0033  200  200  140  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x002e  200  200  000  Old_age  Always  -  0
  9 Power_On_Hours  0x0032  100  100  000  Old_age  Always  -  228
10 Spin_Retry_Count  0x0032  100  253  000  Old_age  Always  -  0
11 Calibration_Retry_Count 0x0032  100  253  000  Old_age  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  34
192 Power-Off_Retract_Count 0x0032  200  200  000  Old_age  Always  -  12
193 Load_Cycle_Count  0x0032  200  200  000  Old_age  Always  -  25
194 Temperature_Celsius  0x0022  119  116  000  Old_age  Always  -  33
196 Reallocated_Event_Count 0x0032  200  200  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0032  200  200  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0030  100  253  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  0
200 Multi_Zone_Error_Rate  0x0008  200  200  000  Old_age  Offline  -  0

smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x002f  200  200  051  Pre-fail  Always  -  0
  3 Spin_Up_Time  0x0027  172  169  021  Pre-fail  Always  -  8358
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  38
  5 Reallocated_Sector_Ct  0x0033  200  200  140  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x002e  200  200  000  Old_age  Always  -  0
  9 Power_On_Hours  0x0032  100  100  000  Old_age  Always  -  228
10 Spin_Retry_Count  0x0032  100  253  000  Old_age  Always  -  0
11 Calibration_Retry_Count 0x0032  100  253  000  Old_age  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  34
192 Power-Off_Retract_Count 0x0032  200  200  000  Old_age  Always  -  12
193 Load_Cycle_Count  0x0032  200  200  000  Old_age  Always  -  25
194 Temperature_Celsius  0x0022  122  120  000  Old_age  Always  -  30
196 Reallocated_Event_Count 0x0032  200  200  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0032  200  200  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0030  100  253  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  0
200 Multi_Zone_Error_Rate  0x0008  200  200  000  Old_age  Offline  -  0

smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x002f  200  200  051  Pre-fail  Always  -  0
  3 Spin_Up_Time  0x0027  175  174  021  Pre-fail  Always  -  8216
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  38
  5 Reallocated_Sector_Ct  0x0033  200  200  140  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x002e  200  200  000  Old_age  Always  -  0
  9 Power_On_Hours  0x0032  100  100  000  Old_age  Always  -  228
10 Spin_Retry_Count  0x0032  100  253  000  Old_age  Always  -  0
11 Calibration_Retry_Count 0x0032  100  253  000  Old_age  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  34
192 Power-Off_Retract_Count 0x0032  200  200  000  Old_age  Always  -  12
193 Load_Cycle_Count  0x0032  200  200  000  Old_age  Always  -  25
194 Temperature_Celsius  0x0022  119  116  000  Old_age  Always  -  33
196 Reallocated_Event_Count 0x0032  200  200  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0032  200  200  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0030  100  253  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  0
200 Multi_Zone_Error_Rate  0x0008  200  200  000  Old_age  Offline  -  0

[root@freenas] ~#

So now I tried some DD read and writes to each drive individually, three of them have consistent read/writes of 150MB/s while one of them will read at about 125MB/s but write at 150MB/s. Attached is a report of the data from FreeNAS. I'm running the test again with a larger file. Any input if this is a warning for drive issues? I'll try swapping them around after this test and see if it's the drive of the controller.

ada1 R:123MB/s W:153MB/s
ada2 R:153MB/s W:153MB/s
ada3 R:154MB/s W:154MB/s
ada4 R:152MB/s W:152MB/s

acording to below data:

Code:

[root@freenas] ~# dd if=/dev/ada1 of=testfile bs=1M count=25000;
25000+0 records in
25000+0 records out
26214400000 bytes transferred in 202.625745 secs (129373491 bytes/sec)
[root@freenas] ~# dd if=testfile of=/dev/ada1 bs=1M count=25000;
25000+0 records in
25000+0 records out
26214400000 bytes transferred in 162.482077 secs (161337179 bytes/sec)
[root@freenas] ~# dd if=/dev/ada2 of=testfile bs=1M count=25000;
25000+0 records in
25000+0 records out
26214400000 bytes transferred in 162.466563 secs (161352586 bytes/sec)
[root@freenas] ~# dd if=testfile of=/dev/ada2 bs=1M count=25000;
25000+0 records in
25000+0 records out
26214400000 bytes transferred in 162.769960 secs (161051830 bytes/sec)
[root@freenas] ~# dd if=/dev/ada3 of=testfile bs=1M count=25000;
25000+0 records in
25000+0 records out
26214400000 bytes transferred in 162.230274 secs (161587596 bytes/sec)
[root@freenas] ~# dd if=testfile of=/dev/ada3 bs=1M count=25000;
25000+0 records in
25000+0 records out
26214400000 bytes transferred in 162.441051 secs (161377927 bytes/sec)
[root@freenas] ~# dd if=/dev/ada4 of=testfile bs=1M count=25000;
25000+0 records in
25000+0 records out
26214400000 bytes transferred in 164.471586 secs (159385585 bytes/sec)
[root@freenas] ~# dd if=testfile of=/dev/ada4 bs=1M count=25000;
25000+0 records in
25000+0 records out
26214400000 bytes transferred in 164.729469 secs (159136068 bytes/sec)

Thanks again!

 

[OiD]

Ok, I've swapped the drives ada1 with ada2 and ada3 with ada4. Same scenario, ada1 reads around 120MB/s and writes at 150MB/s.
The drives are okay then.

I've checked the BIOS settings as well. There are two "SATA Emulation" modes, IDE or SATA+AHCI (Raid 0,1,2,5,10). I'm using IDE as the documentation states that the RAID has JBOD but it's not clear if it's true JBOD (ACH10).

Edit; I've touched something... now I'm getting 150MB/s R/W on all drives.. I went though all the BIOS settings. I must have touched something...
Well, i'll put the drives back in the correct order, and start up FreeNAS and start copying my files over.

 

[nightshade00013]

Generally SATA AHCI mode is used. Raid is a different step and requires more configuration. IDE mode has to do some other fancy stuff and will run slower.

 

[OiD]

Generally SATA AHCI mode is used. Raid is a different step and requires more configuration. IDE mode has to do some other fancy stuff and will run slower.

So ideally I'd want the SATA+AHCI(RAID) and make sure I'ts a true JBOD. Any Idea on how to test it?

The controller is either a 3041E or 8888ELP HBA. I'm now using the SATA+AHCI(RAID) mode and smart works. Any other tests for true JBOD or pass through mode?

Thanks!

 

[nightshade00013]

If you didn't make any changes in the config and the drives are still working and smart is working it is likely just AHCI without any raid happening. A way to truly test this would be to setup a pool with all drives with the controller in IDE mode and then switch it to AHCI and see if the pool is recognized. I am not 100% sure but it sounds like it is probably the same as most consumer boards which have a raid option.

And just so you know that board uses broadcom ethernet. so you may want to get an intel ethernet card.

https://www.cnet.com/products/hp-workstation-z600-xeon-e5620-2-4-ghz-series/specs/

 

[saikee]

I believe in using zfs filing system all other raid should be switched off. zfs itself has its own raidz and works best or most efficiently on disks not already configured with raid outside zpool.

 

[OiD]

If you didn't make any changes in the config and the drives are still working and smart is working it is likely just AHCI without any raid happening. A way to truly test this would be to setup a pool with all drives with the controller in IDE mode and then switch it to AHCI and see if the pool is recognized. I am not 100% sure but it sounds like it is probably the same as most consumer boards which have a raid option.

And just so you know that board uses broadcom ethernet. so you may want to get an intel ethernet card.

https://www.cnet.com/products/hp-workstation-z600-xeon-e5620-2-4-ghz-series/specs/

I'll give it a test and see!

Damn! I must of been looking at some old documentation... i read that the max RAM size was 24GB... There is a C2 revision that supports 48GB... Ah well, good to know I can upgrade the RAM down the road at least!

Edit; Just checked my version and its a C2. FreeNAS seems to work fine in both IDE and AHCI+RAID mode. As long as I don't make an array it should be a simple passthrough. Double yay!

I believe in using zfs filing system all other raid should be switched off. zfs itself has its own raidz and works best or most efficiently on disks not already configured with raid outside zpool.

That is what I've understood. I want to make sure I have no RAID vodoo going on thinking that I'ts deactivated :D

 

[OiD]

I'm in a bit of a pickle. My 2TB backup drive just bit the dust. It won't spin up.
I'm currently importing from the original 2TB drive into FreeNAS, doing about 100GB per hour, should be done in the morning. Still I'm not that comfortable yet of trusting FreeNAS as im totally new to it, what is a good way to check if the importing process has succeded?

Meanwhile I'm trying not to panick...

Thanks!

 

[saikee]

I thought the upper right icon of FreeNAS GUI gives out warning on bad disk or storage problems. Did you not get a prior warning?

 

[OiD]

I thought the upper right icon of FreeNAS GUI gives out warning on bad disk or storage problems. Did you not get a prior warning?

The Drive was from my main computer and connected via usb 3.0 to FreeNas. So no warnings. I managed to get the data from the backup onto FreeNas so all is well. Now I'm sorting all my files out and will make another backup when done.

I'll also check the hard drive and see if I can find out what failed. Might just be a polyfuse/regulator or the whole board could be dead... One drive with dying sectors and one that just died on me. Good thing I had backups :D