Hard Drive Burn-In Testing - Discussion Thread

[Mirfster]

"inappropriate ioctl"? Can anyone explain?

https://forums.freenas.org/index.php?threads/badblocks-testing-inappropriate-ioctl-for-device.26015/

 

[Deleted47050]

For what it's worth, I have always received that error message every time I have run badblocks but it has always completed successfully, so nowadays I just ignore it.

 

[VladTepes]

OK it seems I have some 'ïssues"

These are the results of my SMART tests, after having performed badblocks testing on the drives:

Show

Code:

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       2
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       234
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       2
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       105
194 Temperature_Celsius     0x0022   122   115   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

[root@Remus] ~# smartctl -A /dev/ada2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       9
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       5
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       323
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       5
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       2
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       187
194 Temperature_Celsius     0x0022   122   115   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

[root@Remus] ~# smartctl -A /dev/ada3
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   194   194   021    Pre-fail  Always       -       5275
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       6
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       1184
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       6
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       4
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1025
194 Temperature_Celsius     0x0022   122   108   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

[root@Remus] ~# smartctl -A /dev/ada4
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   198   182   051    Pre-fail  Always       -       9293
  3 Spin_Up_Time            0x0027   184   182   021    Pre-fail  Always       -       5800
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2232
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       2046
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   098   098   000    Old_age   Always       -       2053
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       14
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2217
194 Temperature_Celsius     0x0022   122   104   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

[root@Remus] ~# smartctl -A /dev/ada5
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   180   179   021    Pre-fail  Always       -       5958
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2231
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       2042
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   098   098   000    Old_age   Always       -       2052
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       13
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2217
194 Temperature_Celsius     0x0022   121   103   000    Old_age   Always       -       29
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

[root@Remus] ~# smartctl -A /dev/ada6
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       100
  3 Spin_Up_Time            0x0027   179   179   021    Pre-fail  Always       -       6008
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2231
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       2040
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   098   098   000    Old_age   Always       -       2052
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       13
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2217
194 Temperature_Celsius     0x0022   120   105   000    Old_age   Always       -       30
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       1

I suppose at this point the question is - are these drives OK to use, or cactus?

 

[rogerh]

OK it seems I have some 'ïssues"

These are the results of my SMART tests, after having performed badblocks testing on the drives:


I suppose at this point the question is - are these drives OK to use, or cactus?

I assume there were no badblocks errors and you have done long smart tests which completed satisfactorily on all disks since you did the badblocks. You did not actually mention this above, but assuming it is true then I would go ahead and use the drives.

It is definitely a good idea to do regular long smart tests and regular scrubs, perhaps once or twice a month each, and make sure errors in either will be reported to you by email. In which case your server should be as reliable as any comparable FreeNAS setup. You also probably need backups if your data is at all valuable.

 

[VladTepes]

Sorry in the badblocks thread I started I mentioned that a couple of drives showed badblocks results 3/0/0 and the others 0/0/0
The results I posted above of the long smartctl tests were conducted after that badblocks testing.

Noted re the backups issue - this data also sits on my 6TB drive on my PC, as well as much of it in external USB drives which usually remain unattached.

 

[rogerh]

That will teach me to read properly. We need an expert to say if those badblocks results should disqualify the drives.

 

[Stux]

When running badblocks shouldn't there be a flag or something to output the bad blocks to a file so that the system knows not to use those blocks on the drives?

The very first post says badblocks is destructive and says it in big bold scary letters.

It then also shows the non destructive version and it's flags.

 

[Stux]

Yep, you are correct. No, I didn't put any data on the pool. The only reason a pool was there was because I followed the installation guide in the documentation. Had I know to wait, I would have. In any case, I unmounted the pool (and based on the prompts I got, the console deleted it...?). I'm going to rerun badblocks now but having the pool mounted has generated a crap tonne of read errors in the SMART output.

Code:

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   113   100   006    Pre-fail  Always       -       51493616
  3 Spin_Up_Time            0x0003   091   091   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       18
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   075   060   030    Pre-fail  Always       -       34074229
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       294
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       19
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   070   067   045    Old_age   Always       -       30 (Min/Max 19/33)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       14
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       18
194 Temperature_Celsius     0x0022   030   040   000    Old_age   Always       -       30 (0 19 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

I'm hoping that once I rerun badblocks those numbers will change to more realistic values. They are seagate drives though and I know they index the errors differently....

Seagate drives include the raw read rate in the raw read error rate.

Funny, but sorta cool too.

Same with seeks/writes

Point is, the raw value is not necessarily real. but depending on your drive they can be a sign of early failure.

The when failed is more important.

 

[Ericloewe]

Seagate drives include the raw read rate in the raw read error rate.

Funny, but sorta cool too.

Same with seeks/writes

Point is, the raw value is not necessarily real. but depending on your drive they can be a sign of early failure.

The when failed is more important.

More than that, it's a funky coding that represents two separate values, both in floating point. So, their decimal translation is completely meaningless.

 

[Stux]

More than that, it's a funky coding that represents two separate values, both in floating point. So, their decimal translation is completely meaningless.

Yes. I've actually decoded it before for s and giggles

 

[angelus249]

Thanks for the guide, very helpful.

Just one thing. It's been mentioned before here, but I really think the OP should be adjusted for the usage "screen" instead of "tmux". It's so much better.

Mini-Tutorial for whoever might be interested.

Open new screen:
screen -m

Start command for particular hdd inside screen, e.g.
badblocks -b 4096 -ws /dev/ada1

Detach that screen with "CTRL + A + D". Then create more screens for more disks.

List available screens:
screen -ls
e.g.

Code:

There are screens on:
		57951.pts-8.FreeNAS	 (Detached)
		57958.pts-8.FreeNAS	 (Detached)
		57916.pts-8.FreeNAS	 (Detached)

Re-open screen with
screen -r 57951

Cheers

 

[SweetAndLow]

Thanks for the guide, very helpful.

Just one thing. It's been mentioned before here, but I really think the OP should be adjusted for the usage "screen" instead of "tmux". It's so much better.

Mini-Tutorial for whoever might be interested.

Open new screen:
screen -m

Start command for particular hdd inside screen, e.g.

badblocks -b 4096 -ws /dev/ada1


Detach that screen with "CTRL + A + D". Then create more screens for more disks.

List available screens:
screen -ls
e.g.

Code:

There are screens on:
		57951.pts-8.FreeNAS	 (Detached)
		57958.pts-8.FreeNAS	 (Detached)
		57916.pts-8.FreeNAS	 (Detached)

Re-open screen with
screen -r 57951

Cheers

Tmux is screen with more features. Doesn't matter which you use but tmux would be the better choice if you are going to use those skills somewhere else.

Sent from my Nexus 5X using Tapatalk

 

[Meyers]

Thanks for this guide - it was very helpful. I used it last year to burn in drives for my first FreeNAS system (a FreeNAS Mini) and just burning in new drives for a FreeNAS Mini XL system now. I had never considered doing this before, but it makes total sense.

I bought 6TB WD Reds last year and this year I bought the 8TB drives (and another four 6TB drives to expand into the Mini XL). One thing I noticed is that the 8TB models lack a conveyance test, so I ran the short and long tests, badblocks, and will run another long after that. Otherwise I haven't had a single issue with the Reds running 24x7 for the past year (just doing remote backups so low utilization, scrubs 2x a month, SMART tests several times a month).

So far this has taken about six days to do and badblocks is nearly done (90ish% at 121.5 hours). Plan on about 7-8 days for a full burn-in procedure on 8TB drives!

 

[koerni]

Thank you for the guide. I used screen instead of tmux because I'm more familiar with but everything worked like a charm. Would be nice, if the runtime of these test on different HD sizes could be added in the start post, so new users can gues how long the badblocks test takes (mine is taking 57 h and it is not even completed on a WD 4 TB Red drive).

Is there a tool which monitors the changes of the smart values over time?
For example: remembers the output of the smarttests, writes these in a database and shows me the changes over the past, lets say 50 runs. A nice and fancy webgui would be a bonus but I think christmas is over... :)

 

[Deleted47050]

Thank you for the guide. I used screen instead of tmux because I'm more familiar with but everything worked like a charm. Would be nice, if the runtime of these test on different HD sizes could be added in the start post, so new users can gues how long the badblocks test takes (mine is taking 57 h and it is not even completed on a WD 4 TB Red drive).

Is there a tool which monitors the changes of the smart values over time?
For example: remembers the output of the smarttests, writes these in a database and shows me the changes over the past, lets say 50 runs. A nice and fancy webgui would be a bonus but I think christmas is over... :)

I don't know of any such tools but I would be interested to see if someone else knows something more about this. Still, it shouldn't be too difficult to implement it, if I had more time I could have a go at this.

In any case, you could do this manually in several ways. For example, you could use cron to save smart tests to file and then compare them using diff so you would see what changed compared to the previous smart test (which is probably what you will be looking for 99% of the times).

 

[Kris Heslop]

I have been using this with some mixed results after one of my 3 drives started showing signs of failure after only 11-months. I have run into strange, to my mind at least, issues as follows:

I have looked for these items below, but without success, perhaps because I have not mastered the search syntax 100%.

1: "Smartcl" not working
When I attempt to run "smartcl" I get error messages saying essentially not installed.
I can get "smartd" to work, but the flags for long and short tests are unrecognized.

I have checked and SMART is enabled, any thoughts on this?

2: What happens if you run SSH, tmux, but do Cntrl b +", instead of the uppercase? (Yes i did it, and it looks like I have combined things and have a hard time confirming which disk I am testing, since the "/dev/adax" is no longer showing.)

3: When I was only running one badblocks on the new drive (ada4) it stopped after 21+ hours, and then again after under 15 minutes (Run through shell with tmux.)

4. Where am I supposed to go to find the smart status reports on the disks? Perhaps related to #1 above...

 

[Kris Heslop]

Oh, two others, (1) how do you stop running badblocks? (2) How do you restart badblocks? When I tried it started again at zero, from what I read above, it seems like for some it resumed.

Thank you for helping out yet another newbie.

 

[SweetAndLow]

Oh, two others, (1) how do you stop running badblocks? (2) How do you restart badblocks? When I tried it started again at zero, from what I read above, it seems like for some it resumed.

Thank you for helping out yet another newbie.

Might be easier to start your own thread since you have so many problems. Make sure to read the forum rules before posting and always include hardware and freenas version.

Sent from my Nexus 5X using Tapatalk

 

[Ruff.Hi]

I've just picked up some new HDDs and I want to run the various SMART and bad blocks tests mentioned here. My plan was to plug the new drives into my production machine via some open SATA ports, boot up FreeNAS and run the tests on the new drives. My current 8 HDDs in my zpool would be active and connected to my prod machine.

Any issues with that?

I am expecting no issues re the SMART tests ... not sure about the badblocks part ... especially the 'we need to enable the kernel geometry debug flags' part.

 

[Spearfoot]

I've just picked up some new HDDs and I want to run the various SMART and bad blocks tests mentioned here. My plan was to plug the new drives into my production machine via some open SATA ports, boot up FreeNAS and run the tests on the new drives. My current 8 HDDs in my zpool would be active and connected to my prod machine.

Any issues with that?

I am expecting no issues re the SMART tests ... not sure about the badblocks part ... especially the 'we need to enable the kernel geometry debug flags' part.

Well... it's up to you, of course - your machine, your rules.

Badblocks is destructive when used to burn-in disks as advised in this thread, so there's always the chance of making a fat-fingered mistake and inadvertently erasing one or more drives in your pool. Everything will be okay, as long as you make absolutely sure you're working with the new drives and not the ones in your pool!

I'm lucky enough to have an old PC equipped with a Dell H200 HBA, just for this kind of work. Handy!