Scary SMART Values on New Seagate Enterprise Drives?

[mattlach]

Hey all,

So, I'm 4 disks into my "swap all 12 drives in my NAS with larger drives and resilver to grow my ZFS pool" project, and decided to go with 10TB Seagate Helium Enterprise drives (ST10000NM0016)

The four drives I have thus far come from two orders of two drives each. Two from Newegg and two from Amazon.

All drives passed the following tests before being resilvered into the pool:
- SMART Short test
- SMART Conveyance Test
- Badblocks write test (all four test patterns, taking ~5 days)

I've done some reading on this in the past where it was suggested that one pretty much ignore the "RAW VALUES" in SMART readouts from Seagate drives as they probably don't mean what you think they mean, and instead use Seatools for any diagnostics.

The problems I have with Seatools:

1.) The Linux version is old and not maintained and didn't appear to give me any useful information.

2.) The Windows version might be more fully featured, but it doesnt seem to recognize Seagate hard drives as true Seagate hard drives when sitting in a USB dock, and I don't have a box with easily accessible SATA ports I can stick one in to right now

3.) There is a DOS version, but that requires taking my server offline and booting from a Freedos USB stick. My server may not be production, but it is more "home production" than lab, so this would be really inconvenient.


Why I am concerned:

I know "ignore the RAW VALUE field" is what I've found when googling inthe past, but what about the three digit weighted values? Are those to be ignored as well?

Just look at some of these, notably the "Hardware_ECC_Recovered" looks pretty scary on all of these:

Disk 1:

Code:

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   075   064   044	Pre-fail  Always	   -	   32160040
  3 Spin_Up_Time			0x0003   096   096   000	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   4
  5 Reallocated_Sector_Ct   0x0033   100   100   010	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000f   081   060   045	Pre-fail  Always	   -	   129401214
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   494 (1 186 0)
10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   3
184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
188 Command_Timeout		 0x0032   100   100   000	Old_age   Always	   -	   0
189 High_Fly_Writes		 0x003a   088   088   000	Old_age   Always	   -	   12
190 Airflow_Temperature_Cel 0x0022   071   055   040	Old_age   Always	   -	   29 (Min/Max 26/35)
191 G-Sense_Error_Rate	  0x0032   098   098   000	Old_age   Always	   -	   4187
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   5
193 Load_Cycle_Count		0x0032   100   100   000	Old_age   Always	   -	   30
194 Temperature_Celsius	 0x0022   029   045   000	Old_age   Always	   -	   29 (0 19 0 0 0)
195 Hardware_ECC_Recovered  0x001a   027   003   000	Old_age   Always	   -	   32160040
197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x003e   200   200   000	Old_age   Always	   -	   0
240 Head_Flying_Hours	   0x0000   100   253   000	Old_age   Offline	  -	   491 (144 32 0)
241 Total_LBAs_Written	  0x0000   100   253   000	Old_age   Offline	  -	   83916735051
242 Total_LBAs_Read		 0x0000   100   253   000	Old_age   Offline	  -	   83452883922

Disk 2:

Code:

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   080   066   044	Pre-fail  Always	   -	   91957936
  3 Spin_Up_Time			0x0003   096   096   000	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   4
  5 Reallocated_Sector_Ct   0x0033   100   100   010	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000f   079   061   045	Pre-fail  Always	   -	   81905155
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   283 (213 63 0)
10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   4
184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
188 Command_Timeout		 0x0032   100   100   000	Old_age   Always	   -	   0
189 High_Fly_Writes		 0x003a   090   090   000	Old_age   Always	   -	   10
190 Airflow_Temperature_Cel 0x0022   072   067   040	Old_age   Always	   -	   28 (Min/Max 20/33)
191 G-Sense_Error_Rate	  0x0032   098   098   000	Old_age   Always	   -	   4777
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   2
193 Load_Cycle_Count		0x0032   100   100   000	Old_age   Always	   -	   16
194 Temperature_Celsius	 0x0022   028   040   000	Old_age   Always	   -	   28 (0 20 0 0 0)
195 Hardware_ECC_Recovered  0x001a   006   002   000	Old_age   Always	   -	   91957936
197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x003e   200   200   000	Old_age   Always	   -	   0
240 Head_Flying_Hours	   0x0000   100   253   000	Old_age   Offline	  -	   273 (12 162 0)
241 Total_LBAs_Written	  0x0000   100   253   000	Old_age   Offline	  -	   83844942895
242 Total_LBAs_Read		 0x0000   100   253   000	Old_age   Offline	  -	   82966201892

Disk 3:

Code:

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   079   064   044	Pre-fail  Always	   -	   70937936
  3 Spin_Up_Time			0x0003   098   098   000	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   2
  5 Reallocated_Sector_Ct   0x0033   100   100   010	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000f   077   060   045	Pre-fail  Always	   -	   53737869
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   158 (50 181 0)
10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   2
184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
188 Command_Timeout		 0x0032   100   100   000	Old_age   Always	   -	   0
189 High_Fly_Writes		 0x003a   098   098   000	Old_age   Always	   -	   2
190 Airflow_Temperature_Cel 0x0022   071   067   040	Old_age   Always	   -	   29 (Min/Max 24/33)
191 G-Sense_Error_Rate	  0x0032   098   098   000	Old_age   Always	   -	   4720
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   1
193 Load_Cycle_Count		0x0032   100   100   000	Old_age   Always	   -	   14
194 Temperature_Celsius	 0x0022   029   040   000	Old_age   Always	   -	   29 (0 21 0 0 0)
195 Hardware_ECC_Recovered  0x001a   007   006   000	Old_age   Always	   -	   70937936
197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x003e   200   200   000	Old_age   Always	   -	   0
240 Head_Flying_Hours	   0x0000   100   253   000	Old_age   Offline	  -	   157 (185 136 0)
241 Total_LBAs_Written	  0x0000   100   253   000	Old_age   Offline	  -	   83754676019
242 Total_LBAs_Read		 0x0000   100   253   000	Old_age   Offline	  -	   78155955092

Disk 4:

Code:

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   083   065   044	Pre-fail  Always	   -	   193684680
  3 Spin_Up_Time			0x0003   098   098   000	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   2
  5 Reallocated_Sector_Ct   0x0033   100   100   010	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000f   077   061   045	Pre-fail  Always	   -	   54155047
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   158 (142 118 0)
10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   2
184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
188 Command_Timeout		 0x0032   100   100   000	Old_age   Always	   -	   0
189 High_Fly_Writes		 0x003a   100   100   000	Old_age   Always	   -	   0
190 Airflow_Temperature_Cel 0x0022   071   067   040	Old_age   Always	   -	   29 (Min/Max 25/33)
191 G-Sense_Error_Rate	  0x0032   098   098   000	Old_age   Always	   -	   4822
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   0
193 Load_Cycle_Count		0x0032   100   100   000	Old_age   Always	   -	   10
194 Temperature_Celsius	 0x0022   029   040   000	Old_age   Always	   -	   29 (0 21 0 0 0)
195 Hardware_ECC_Recovered  0x001a   011   009   000	Old_age   Always	   -	   193684680
197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x003e   200   200   000	Old_age   Always	   -	   0
240 Head_Flying_Hours	   0x0000   100   253   000	Old_age   Offline	  -	   157 (133 76 0)
241 Total_LBAs_Written	  0x0000   100   253   000	Old_age   Offline	  -	   83781193947
242 Total_LBAs_Read		 0x0000   100   253   000	Old_age   Offline	  -	   78158256547

So, am I concerned for nothing. Does one just ignore SMART readouts on Seagate Enterprise drives, or did I get 4 drives from 2 different retailers that are all going bad within only a couple of weeks?

Much appreciated.

Crossposted here and here for more eyeballs, as server/enterprise/*nix stuff tends to get less traffic.

 

[rs225]

Not scary. It is well known the Seagate SMART values on those elements always look big. I think they are actually two values in each one, which is supposed to form a ratio.(Hence 'rate') I don't know where the other value kicks in, but it would be something where you see a number that suddenly is using more than 32-bits, or some other very large value. That's when you actually have activity.

If you have something like CrystalDiskInfo, it obviously knows how to decode the value, as it doesn't show any concern on a normal Seagate drive.

 

[cobrakiller58]

my seagate enterprise drives have similar numbers and are over 40k hours on them I personally ignore 1,7 and 195 value unless it actually says it's failed at some point. here are a couple of them and have been performing flawlessly.

Code:

Device Model:	 ST32000645NS
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   081   063   044	Pre-fail  Always	   -	   150883171
  3 Spin_Up_Time			0x0003   095   093   000	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   97
  5 Reallocated_Sector_Ct   0x0033   100   100   036	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000f   081   060   030	Pre-fail  Always	   -	   4413827885
  9 Power_On_Hours		  0x0032   052   052   000	Old_age   Always	   -	   42708
 10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   96
184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
188 Command_Timeout		 0x0032   100   100   000	Old_age   Always	   -	   0
189 High_Fly_Writes		 0x003a   096   096   000	Old_age   Always	   -	   4
190 Airflow_Temperature_Cel 0x0022   071   041   045	Old_age   Always   In_the_past 29 (23 79 30 29 0)
191 G-Sense_Error_Rate	  0x0032   100   100   000	Old_age   Always	   -	   0
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   81
193 Load_Cycle_Count		0x0032   100   100   000	Old_age   Always	   -	   97
194 Temperature_Celsius	 0x0022   029   059   000	Old_age   Always	   -	   29 (0 22 0 0 0)
195 Hardware_ECC_Recovered  0x001a   021   004   000	Old_age   Always	   -	   150883171
197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x003e   200   200   000	Old_age   Always	   -	   0

Code:

Device Model:	 ST32000645NS
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   078   063   044	Pre-fail  Always	   -	   81499827
  3 Spin_Up_Time			0x0003   096   093   000	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   101
  5 Reallocated_Sector_Ct   0x0033   100   100   036	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000f   081   060   030	Pre-fail  Always	   -	   121533533
  9 Power_On_Hours		  0x0032   052   052   000	Old_age   Always	   -	   42706
 10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   99
184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
188 Command_Timeout		 0x0032   100   099   000	Old_age   Always	   -	   1
189 High_Fly_Writes		 0x003a   100   100   000	Old_age   Always	   -	   0
190 Airflow_Temperature_Cel 0x0022   072   041   045	Old_age   Always   In_the_past 28 (36 17 30 28 0)
191 G-Sense_Error_Rate	  0x0032   100   100   000	Old_age   Always	   -	   0
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   86
193 Load_Cycle_Count		0x0032   100   100   000	Old_age   Always	   -	   101
194 Temperature_Celsius	 0x0022   028   059   000	Old_age   Always	   -	   28 (0 21 0 0 0)
195 Hardware_ECC_Recovered  0x001a   018   003   000	Old_age   Always	   -	   81499827
197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x003e   200   200   000	Old_age   Always	   -	   0

 

[mattlach]

Not scary. It is well known the Seagate SMART values on those elements always look big. I think they are actually two values in each one, which is supposed to form a ratio.(Hence 'rate') I don't know where the other value kicks in, but it would be something where you see a number that suddenly is using more than 32-bits, or some other very large value. That's when you actually have activity.

If you have something like CrystalDiskInfo, it obviously knows how to decode the value, as it doesn't show any concern on a normal Seagate drive.

Thanks, but as I mentioned, I know the RAW values are to be ignored.

I'm concerned about the interpreted normalized value to the left.

for the HARDWARE_ECC_RECOVERED field, the lowest of the four is at 006, and the highest only at 029, with a threshold of 000. These are usually up near 100 for new drives. When I see a value of 006 and a threshold of 000, that looks like "failure imminent" to me.

That is, unless the interpreted SMART values are best ignored as well for these drives.

Thanks,
Matt

 

[mattlach]

my seagate enterprise drives have similar numbers and are over 40k hours on them I personally ignore 1,7 and 195 value unless it actually says it's failed at some point. here are a couple of them and have been performing flawlessly.

Thank you, that is good to know, and reassuring.

 

[rs225]

Your raw values for ECC and Read show an equal number, so we can infer that each is tracking zero events(ecc or raw read) on the same number of reads. But why the differing normalized values? That would be a question for Seagate. I would say it is best to look at the raw values, and ignore the normalized values. If an attribute actually says it is failing, then pay attention.

The concept of ignoring the raw values (for ordinary persons) means that unless you are putting the values in a binary/hex calculator to look for the upper 16 bits (32 thru 47) and see that they are all zero, they are just going to cause worry. This is especially true when the number seems to increase every time you look at it. :)

 

[mattlach]

Your raw values for ECC and Read show an equal number, so we can infer that each is tracking zero events(ecc or raw read) on the same number of reads. But why the differing normalized values? That would be a question for Seagate. I would say it is best to look at the raw values, and ignore the normalized values. If an attribute actually says it is failing, then pay attention.

The concept of ignoring the raw values (for ordinary persons) means that unless you are putting the values in a binary/hex calculator to look for the upper 16 bits (32 thru 47) and see that they are all zero, they are just going to cause worry. This is especially true when the number seems to increase every time you look at it. :)

Good to know, thank you.


Yeah it can be quite disconcerting to look at, especially if you are expecting SMART to work like every other SATA drive of every type from every other manufacturer out there.

With literally every other SATA hard drive or SSD I've come across the raw values may or may not be human readable, but the interpreted value starts out at 100 for a perfectly new drive, and slowly degrades down towards a threshold value which indicates possible failure.

I seriously don't understand what possessed them to be different in this regard.

 

[rs225]

At the time, they were probably looking at the information only being evaluated by diagnostic programs, not people. If their method had won out, nobody would think there is anything wrong with it. It actually is better information; perhaps the problem is their reluctance to document it.

 

[mattlach]

At the time, they were probably looking at the information only being evaluated by diagnostic programs, not people. If their method had won out, nobody would think there is anything wrong with it. It actually is better information; perhaps the problem is their reluctance to document it.

Ahh, so you are saying this dates back to when SMART was new in 2002 or so?

I guess I haven't used Seagate drives in a while :p

Come to think of it, the last ones I used were probably the two obscenely loud 7200rpm 100GB PATA drives I had in a striped RAID0 configuration in my desktop in college ~2001 which replaced my old 34gig IBM Deskstar. I got them so that I had enough sequential write speed to record analogue video compressed with lossless huffyuv codecs from my video capture card. Amazing how difficult Thai stuff used to be, not THAT long ago.

After that I think I had a single 7200rpm WD drive in my desktop which was replaced with a small boot SSD and a WD green for data storage in 2010.

Then the WD green left my desktop when I foolishly bought a 5 bay drobo that same year and filled it with WD Greens.

Within a year, I realized the drobo was a mistake and sold it, and built my first ZFS system with WD greens which one by one died after they racked up hundreds of thousands of head parks because I never ran across the articles about WDIDLE.exe.

One by one I swapped in 4TB WD reds until I got a proper server case and expanded it to 12 4TB WD Reds (2x 6 drive RAIDz2) and now I am swapping those out one by one with 10TB Seagate Enterprise drives.

Anyway, I digress. Moral of the story, I haven't played with Seagate drives in a while.

 

[Chris Moore]

One by one I swapped in 4TB WD reds until I got a proper server case and expanded it to 12 4TB WD Reds (2x 6 drive RAIDz2) and now I am swapping those out one by one with 10TB Seagate Enterprise drives.

You have all the 10 TB drives in place or is that still a work in progress?
There have been some reports of people having trouble with them.

 

[mattlach]

You have all the 10 TB drives in place or is that still a work in progress?
There have been some reports of people having trouble with them.

Hmm. I didn't come across any of this in my research. I'm aware of the rather poor track record Seagate had in the 2TB and 3TB era, but the backblaze reports seem to suggest that Seagate has improved their quality.

What kind of reports of Trouble have you seen?

As I mentioned above, I've got 4 of them resolved this far, so 8 to go. I'm trying to spread them out a bit so they are at least a little bit apart when it comes to date codes...

 

[Chris Moore]

There is a thread here on the forum where a couple people are having errors and, if I recall, they are thinking it is down to a firmware or driver issue. You might want to search the forum, it is just in the last week or so.

It could be a different model, but I thought it was the Seagate 10 TB.

 

[cobrakiller58]

This thread is the one @Chris Moore is talking about, same drive model and same controller. I'm watching that thread because I have the same controller as you and I'm watching the 10TB seagate drives prices

 

[mattlach]

This thread is the one @Chris Moore is talking about, same drive model and same controller. I'm watching that thread because I have the same controller as you and I'm watching the 10TB seagate drives prices

Thanks for that. Please see my response in the other thread.

TLDR version, I have had no issues like this yet, but my pool is on ZFS on Linux these days, so may differences.

 

[rs225]

Ahh, so you are saying this dates back to when SMART was new in 2002 or so?

I don't know for certain, but I would assume it has never changed. According to Wiki, Seagate had a forerunner technology since at least 1995, so it could date to that time.