solnet-array-test

Russell Coight · Aug 22, 2017

I'm running the solnet-array-test, I realise that it never terminates but should it complete the 'Awaiting completion: initial parallel seek-stress array read' step?

It's been going for about 48 hours and hasn't progressed any further but there is disk activity. 'initial parallel array read' finished OK.

Rookie mistake but I've had to run it twice because I accidentally pulled the cable out (broken clip of a temporary network cable) the first time. I started the test again and it skipped the initial parallel array read and went straight to seek-stress array read?
The thread says to run it for a pass or two but a pass doesn't seem to finish for me?

Thanks

Russell Coight · Aug 22, 2017

Saw this in /var/log/messages:

Code:

Aug 21 22:41:53 freenas (ada4:ahcich5:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 00 d  1 df 40 99 01 00 01 00 00
Aug 21 22:41:53 freenas (ada4:ahcich5:0:0:0): CAM status: Uncorrectable parity/C  RC error

Doesn't sound good. badblocks test was OK.

Code:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x002f  200  200  051  Pre-fail  Always  -  0
  3 Spin_Up_Time  0x0027  100  253  021  Pre-fail  Always  -  0
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  2
  5 Reallocated_Sector_Ct  0x0033  200  200  140  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x002e  200  200  000  Old_age  Always  -  0
  9 Power_On_Hours  0x0032  100  100  000  Old_age  Always  -  84
 10 Spin_Retry_Count  0x0032  100  253  000  Old_age  Always  -  0
 11 Calibration_Retry_Count 0x0032  100  253  000  Old_age  Always  -  0
 12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  2
192 Power-Off_Retract_Count 0x0032  200  200  000  Old_age  Always  -  0
193 Load_Cycle_Count  0x0032  200  200  000  Old_age  Always  -  12
194 Temperature_Celsius  0x0022  125  119  000  Old_age  Always  -  25
196 Reallocated_Event_Count 0x0032  200  200  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0032  200  200  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0030  100  253  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  0
200 Multi_Zone_Error_Rate  0x0008  200  200  000  Old_age  Offline  -  0

Russell Coight · Aug 22, 2017

I found another thread with a similar problem and the OP was asked to provide SMART logs.

There's also the same errors for another drive.

ada1

Code:

root@freenas:~ # smartctl -a -q noserial /dev/ada1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family:  Western Digital Red
Device Model:  WDC WD40EFRX-68N32N0
Firmware Version: 82.00A82
User Capacity:  4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:  512 bytes logical, 4096 bytes physical
Rotation Rate:  5400 rpm
Form Factor:  3.5 inches
Device is:  In smartctl database [for details use: -P show]
ATA Version is:  ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:  Tue Aug 22 21:09:03 2017 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
  was never started.
  Auto Offline Data Collection: Disabled.
Self-test execution status:  (  0) The previous self-test routine completed
  without error or no self-test has ever
  been run.
Total time to complete Offline
data collection:  (45600) seconds.
Offline data collection
capabilities:  (0x7b) SMART execute Offline immediate.
  Auto Offline data collection on/off support.
  Suspend Offline collection upon new
  command.
  Offline surface scan supported.
  Self-test supported.
  Conveyance Self-test supported.
  Selective Self-test supported.
SMART capabilities:  (0x0003) Saves SMART data before entering
  power-saving mode.
  Supports SMART auto save timer.
Error logging capability:  (0x01) Error logging supported.
  General Purpose Logging supported.
Short self-test routine
recommended polling time:  (  2) minutes.
Extended self-test routine
recommended polling time:  ( 483) minutes.
Conveyance self-test routine
recommended polling time:  (  5) minutes.
SCT capabilities:  (0x303d) SCT Status supported.
  SCT Error Recovery Control supported.
  SCT Feature Control supported.
  SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x002f  200  200  051  Pre-fail  Always  -  0
  3 Spin_Up_Time  0x0027  100  253  021  Pre-fail  Always  -  0
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  2
  5 Reallocated_Sector_Ct  0x0033  200  200  140  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x002e  200  200  000  Old_age  Always  -  0
  9 Power_On_Hours  0x0032  100  100  000  Old_age  Always  -  167
10 Spin_Retry_Count  0x0032  100  253  000  Old_age  Always  -  0
11 Calibration_Retry_Count 0x0032  100  253  000  Old_age  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  2
192 Power-Off_Retract_Count 0x0032  200  200  000  Old_age  Always  -  0
193 Load_Cycle_Count  0x0032  200  200  000  Old_age  Always  -  11
194 Temperature_Celsius  0x0022  122  119  000  Old_age  Always  -  28
196 Reallocated_Event_Count 0x0032  200  200  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0032  200  200  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0030  100  253  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  0
200 Multi_Zone_Error_Rate  0x0008  200  200  000  Old_age  Offline  -  0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num  Test_Description  Status  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline  Completed without error  00%  83  -
# 2  Extended offline  Completed without error  00%  8  -
# 3  Conveyance offline  Completed without error  00%  0  -
# 4  Short offline  Completed without error  00%  0  -
SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
  1  0  0  Not_testing
  2  0  0  Not_testing
  3  0  0  Not_testing
  4  0  0  Not_testing
  5  0  0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

ada4

Code:

root@freenas:~ # smartctl -a -q noserial /dev/ada4
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family:  Western Digital Red
Device Model:  WDC WD40EFRX-68N32N0
Firmware Version: 82.00A82
User Capacity:  4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:  512 bytes logical, 4096 bytes physical
Rotation Rate:  5400 rpm
Form Factor:  3.5 inches
Device is:  In smartctl database [for details use: -P show]
ATA Version is:  ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:  Tue Aug 22 21:13:02 2017 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
  was never started.
  Auto Offline Data Collection: Disabled.
Self-test execution status:  (  0) The previous self-test routine completed
  without error or no self-test has ever
  been run.
Total time to complete Offline
data collection:  (45480) seconds.
Offline data collection
capabilities:  (0x7b) SMART execute Offline immediate.
  Auto Offline data collection on/off support.
  Suspend Offline collection upon new
  command.
  Offline surface scan supported.
  Self-test supported.
  Conveyance Self-test supported.
  Selective Self-test supported.
SMART capabilities:  (0x0003) Saves SMART data before entering
  power-saving mode.
  Supports SMART auto save timer.
Error logging capability:  (0x01) Error logging supported.
  General Purpose Logging supported.
Short self-test routine
recommended polling time:  (  2) minutes.
Extended self-test routine
recommended polling time:  ( 483) minutes.
Conveyance self-test routine
recommended polling time:  (  5) minutes.
SCT capabilities:  (0x303d) SCT Status supported.
  SCT Error Recovery Control supported.
  SCT Feature Control supported.
  SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x002f  200  200  051  Pre-fail  Always  -  0
  3 Spin_Up_Time  0x0027  100  253  021  Pre-fail  Always  -  0
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  2
  5 Reallocated_Sector_Ct  0x0033  200  200  140  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x002e  200  200  000  Old_age  Always  -  0
  9 Power_On_Hours  0x0032  100  100  000  Old_age  Always  -  167
10 Spin_Retry_Count  0x0032  100  253  000  Old_age  Always  -  0
11 Calibration_Retry_Count 0x0032  100  253  000  Old_age  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  2
192 Power-Off_Retract_Count 0x0032  200  200  000  Old_age  Always  -  0
193 Load_Cycle_Count  0x0032  200  200  000  Old_age  Always  -  14
194 Temperature_Celsius  0x0022  120  119  000  Old_age  Always  -  30
196 Reallocated_Event_Count 0x0032  200  200  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0032  200  200  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0030  100  253  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  0
200 Multi_Zone_Error_Rate  0x0008  200  200  000  Old_age  Offline  -  0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num  Test_Description  Status  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline  Completed without error  00%  83  -
# 2  Extended offline  Completed without error  00%  8  -
# 3  Conveyance offline  Completed without error  00%  0  -
# 4  Short offline  Completed without error  00%  0  -
SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
  1  0  0  Not_testing
  2  0  0  Not_testing
  3  0  0  Not_testing
  4  0  0  Not_testing
  5  0  0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

In the thread I found it was suggested that SATA cables or the PSU could be the problem. I bought a box of 10 Amphenol SATA cables from eBay, same model as these so I'm hoping they're OK.

Forgot to post earlier but specs are:

Supermicro A1SAI-2750F
2 x 8GB Crucial ECC RAM
4 x 4TB WD Reds
Seasonic G550

MrToddsFriends · Aug 22, 2017

Russell Coight said:

I never saw such messages on my system. Supermicro A1SAi-2750F (old version with Avoton/C2000 bug), all SATA ports populated (4 x WD Red 3TB on SATA2 ports, 2 x SanDisk X300s on SATA3 ports). I even have the same cables (6 x CBL-0044L).

Your SMART results look o.k. for me.

Russell Coight · Aug 22, 2017

I've got 2 disks in the SATA3 ports and the other 2 in SATA2 ports, this is OK isn't it? I had to do it this way because the SATADOM is wider than a SATA port so I had to put it in the port that is away from the other ports.

It actually did finish though, 173,829 seconds is just over 48 hours so that explains that. Should I be worried that 3 of the 4 disks are slow?

Code:

Completed: initial parallel seek-stress array read

Disk's average time is 126540 seconds per disk

Disk  Bytes Transferred Seconds %ofAvg
------- ----------------- ------- ------
ada0  4000787030016  173829  137 --SLOW--
ada1  4000787030016  136359  108 --SLOW--
ada3  4000787030016  147327  116 --SLOW--
ada4  4000787030016  48646  38 ++FAST++

MrToddsFriends · Aug 22, 2017

Russell Coight said:
I've got 2 disks in the SATA3 ports and the other 2 in SATA2 ports, this is OK isn't it?

There's no problem with that.

Russell Coight said:
Should I be worried that 3 of the 4 disks are slow?

AFAICT no, but it's been a while since I last used solnet-array-test. Please do a forum search.

Russell Coight · Aug 28, 2017

I changed the SATA cables for ada1 and ada4, ran the test again and got no errors.
As for the slow disks, it changes each time I run the test so I'm not worried about it. Each disk as showed fast at least once in the 3 times I've run the test.

Important Announcement for the TrueNAS Community.

solnet-array-test

Russell Coight

Explorer

Russell Coight

Explorer

Russell Coight

Explorer

MrToddsFriends

Documentation Browser

Russell Coight

Explorer

MrToddsFriends

Documentation Browser

Russell Coight

Explorer

Similar threads