Hard Drive Burn-In Testing - Discussion Thread

[joeschmuck]

If you were me, would you be satisfied with this result or would you play it safe and run smartctl?

I'd be fine with that result but once you get your system built, you can verify the smart data for each drive and ensure it states you have a pass.

 

[Deleted47050]

Also, if you plan on burning in your drives, you will have to run a second set of smart tests on the drives anyway, and you can do it straight on your freenas box then.


Sent from my iPhone using Tapatalk

 

[neto_hugo]

@Spearfoot

Before running your bash script I need to ask you a question:

I'm going to test the script on a WD RED 8TB. Is it advisable for me to edit the values from 4096 to 8192 before? Or can I leave it as it is?

Thank you!

 

[Deleted47050]

@Spearfoot

Before running your bash script I need to ask you a question:

I'm going to test the script on a WD RED 8TB. Is it advisable for me to edit the values from 4096 to 8192 before? Or can I leave it as it is?

Thank you!

That's the block size, not the hard drive size. You can very likely leave that value as it is.

 

[Spearfoot]

@Spearfoot

Before running your bash script I need to ask you a question:

I'm going to test the script on a WD RED 8TB. Is it advisable for me to edit the values from 4096 to 8192 before? Or can I leave it as it is?

Thank you!

It shouldn't hurt to use a larger block size, and it may even speed up testing of your huge hard drive... which is going to take a very, very long time!

The badblocks '-b' and '-c' options work together, but there are limits on how large you can set them:

• -b block-size : Specify the size of blocks in bytes. The default is 1024.
• -c number of blocks : The number of blocks which are tested at a time. The default is 64.

You might try leaving the block size at 4096 and use the '-c' option to make badblocks test a larger number of blocks at a time. Something like 128 or 256 might be a good place to start.

For large drives such as yours, I would skip the initial 'extended' test. Just comment it out by putting an '#' in front of the command, like this:

Code:

# Run the test sequence:	
run_short_test	
#run_extended_test	
run_badblocks_test	
run_short_test	
run_extended_tes

Good luck!

 

[Deleted47050]

If I remember correctly, when I tested my 8 TB drive it took badblocks 3 or 4 days to complete. Just so you know what you are getting into :)


Sent from my iPhone using Tapatalk

 

[Stux]

I have written a Bash script ('disk-burnin.sh') to automate the disk burn-in process , available on GitHub:

https://github.com/Spearfoot/disk-burnin-and-testing

I've tested the script on both FreeNAS/FreeBSD and Linux (Ubuntu). It uses smartctl to query the disk for its short and long test durations in minutes. From these it calculates sleep intervals (in seconds) for the two tests, adding a little extra time for a 'fudge factor'. Then it:

• Executes a short SMART test
  
• Sleeps until the short test finishes, then echoes SMART info to the logfile
  
• Executes a long SMART test
• Sleeps until the long test finishes, then echoes SMART info to the logfile
• Executes badblocks -b 4096 -wsv -o [bad blocks data file] /dev/[device ID]
• Executes a short SMART test
• Sleeps until the short test finishes, then echoes SMART info to the logfile
• Executes a long SMART test
• Sleeps until the long test finishes, then echoes SMART info to the logfile

As described earlier in the thread, it's a good idea to use tmux sessions for each individual drive, and on FreeNAS/FreeBSD you must run sysctl kern.geom.debugflags=0x10 to enable full disk write access.

I hope some of you will find this script useful.

Just testing out this script on 6 Seagate IronWolf 8TB drives.

I think it would be a good idea to start a thread for this script, and then back reference it from the github readme, it took me a while to find this post ;)

Minor thing, and completely my fault, I changed the log directories to a non-existant subdirectory in pools log directory.

/mnt/tank/server/logs/hdd-tests

Predictably resulting in directory not found errors

Code:

tee: /mnt/tank/server/logs/hdd-tests/burnin-ST8000VN0022-2EL112_ZA17HA93.log: No such file or directory

Might want to consider a mkdir -p on the log directory if its not "."?

Also, I noticed it hasn't officially been tested on Seagate drives, is there some obvious way I can tell that its interpreting the SMART output correctly?

Thanks

 

[Spearfoot]

Just testing out this script on 6 Seagate IronWolf 8TB drives.

I think it would be a good idea to start a thread for this script, and then back reference it from the github readme, it took me a while to find this post ;)

There already is a thread in the Resources section ("Github repository for FreeNAS scripts, including disk burnin") which I should have referenced, instead of linking directly to the repo.

Minor thing, and completely my fault, I changed the log directories to a non-existant subdirectory in pools log directory.

/mnt/tank/server/logs/hdd-tests

Predictably resulting in directory not found errors

Code:

tee: /mnt/tank/server/logs/hdd-tests/burnin-ST8000VN0022-2EL112_ZA17HA93.log: No such file or directory

Might want to consider a mkdir -p on the log directory if its not "."?

Good idea! I don't address log file location other than to stub it in. I assume users already have such a directory on their system and will edit the script to use it.

Also, I noticed it hasn't officially been tested on Seagate drives, is there some obvious way I can tell that its interpreting the SMART output correctly?

Thanks

I haven't tested any Seagate disks because I don't own any, but I'm fairly sure that other users have used the script successfully with them.

The script doesn't use the SMART attribute output in any way; it just dumps it out to the log for your perusal. It does query for the test duration and status, but that output from smartctl is the same for all disks.

In case you missed my reference to it in the script, you'll need to run sysctl kern.geom.debugflags=0x10 on FreeBSD systems (including FreeNAS) before burning in disks. It's also a good idea to use a separate tmux session for each disk.

 

[majerus]

How long should it take for the disk burn in script to run? I start it by doing a ./disk-burnin.sh ada1 which returns within seconds which seems wrong. (drive is 4tb)I am missing something?

 

[Stux]

How long should it take for the disk burn in script to run? I start it by doing a ./disk-burnin.sh ada1 which returns within seconds which seems wrong. (drive is 4tb)I am missing something?

The script is configured in dry run mode by default. You need to edit it a little bit.

 

[ViciousXUSMC]

Could I boot to say Ubuntu Live on my laptop and run this on my 8TB WD External drives via USB?

My Server is still waiting for the HBA to come in so that it can see the drives, and I would like to test the drives still in the enclosure to see if they are good before I shuck them.

Right now running 2 drives on 2 computers doing full format and WD Full Erase but each computer can only run one drive at a time.

The big benefit I see in this method is running more than one drive at a time.

 

[Stux]

Could I boot to say Ubuntu Live on my laptop and run this on my 8TB WD External drives via USB?

My Server is still waiting for the HBA to come in so that it can see the drives, and I would like to test the drives still in the enclosure to see if they are good before I shuck them.

Right now running 2 drives on 2 computers doing full format and WD Full Erase but each computer can only run one drive at a time.

The big benefit I see in this method is running more than one drive at a time.

Well, its a simple script... but it probably wouldn't work.

But you could boot to FreeNAS on your desktop and run it. Just grab a flash drive and install FreeNAS to that, then boot off it.

 

[ViciousXUSMC]

Well, its a simple script... but it probably wouldn't work.

But you could boot to FreeNAS on your desktop and run it. Just grab a flash drive and install FreeNAS to that, then boot off it.

Just tried it out, not the precanned script but the manual step by step instructions.
The FreeNAS box has USB ports so tossed two of the 8TB Externals on there and it works.

Just have to manually specify the device type with the -d flag and for the WD 8TB its specifically -d sat for the smartctl commands.

Question on the whole tmux thing.

If you wanted to run 2+ badblocks at once you can just open more than one terminal window/session in like say Putty and get the same results (other than if the session is closed the job terminates)
If its taking about 80 hours to do a full erase or a full format on these disks is BadBlocks going to take about the same or is it going to be 4x longer as it does the 4x pass thing?

All 4 of the 8TB WD Externals I got from Best Buy were the 256mb cache model and not a single error so far, so looking good!

I'll maybe still do my badblocks test on my laptop instead of the NAS using Ubuntu once the current full format finishes that its doing in windows I'll boot it to Ubuntu and see if that works and report back.

 

[Stux]

Well, tmux allows you to resume a console session which has been interrupted/disconnected.

So you can close your ssh and just let the process finish, then check in a week later.

And yes, it takes a long time to burnin an 8TB disk. Why it's a good idea to do all of them at the same time ;)

 

[Journer]

For those worried about tmux or running things in parallel, things getting killed, etc:
just use 'nohup' and send the command to the background (&). You'll want to send all the output to a log instead of screen. If you want to get fancy, after everything is done send a message to wall and/or the pts session you logged into. Or just be lazy and look at the log files every now and then :)


Anyhow - anyone doing any kind of burn-in on SSDs? Probably not a good idea to run through write cycles... but some amount of testing should be in order.

 

[Deleted47050]

I have never done burn in on SSDs, for the reason you said: I don't feel comfortable throwing so many writes on them. Plus, there is less risk for them to have been damaged during transport, which is one of the things that bother me the most with mechanical drives.


Sent from my iPhone using Tapatalk

 

[joeschmuck]

I would not run a write test on a SSD. I would run a SMART Long Test and let it read all the data blocks. If there is a write failure, the SSD should overcome it. You could come up with a nice script to run to watch for specific SMART data to watch the wearlevel value, ensure it warns you each time it changes.

 

[ViciousXUSMC]

Curious about this:
sysctl kern.geom.debugflags=0x10

I just kicked of the badblocks on two disks and forgot to set this, but not seeing any issues in forgetting to do so.

I googled it and it seems like it turns off protection for writing to the MBR of a disk. However since any disks you would run this on are not your OS/Boot disks and pure storage, there should be no MBR to write to no?

Is that part of the guide true that it needs to be done or am I misinterpreting something?

 

[joeschmuck]

That is a good question but since I don't know how badblocks interprets this, it could be just to skip the first 512 bytes and not test those which would not be terrible but it would still leave that section untested. The MBR contains not just a small boot code but equally important the partition table. The good thing is you can wait until your current testing is complete and if you want to, run badblocks on the first few sectors to test those vice redoing the entire drive. Just lookup to parameters to set here.

 

[qwertymodo]

I don't know what that parameter does exactly, I just know that when I didn't set it, badblocks completely failed to run, and when I did set it, it worked.

Sent from my m8wl using Tapatalk