1st storage server - FreeNAS!

[Sir SSV]

I run my fan controller script. It works by putting the fans in max mode and then controlling their duty cycles directly.

Standard and optimal modes control the fans based on CPU temperature. Heavy IO is the same as Standard, except FANA is set to 80% (iirc)

You will want to run a fan controller script if you want the drive fans to spin up/down to cool your HDs.

https://forums.freenas.org/index.php?threads/script-hybrid-cpu-hd-fan-zone-controller.46159/

I will definitely be running your fan controller script :) big thanks for all your hard work! I might have a few questions in regards to installing but that is later. Still need to install my hard drives and carry out burn in/ badblocks testing.

If I run Memtest for another say 2 - 3 hours then I'd be safe to say the ram has been tested sufficiently?

Thanks for all your help in the build thread mate, really do appreciate it :)

 

[Stux]

I think Standard won't spin below 50% and Optimal will drop to below 30% which you can't do on a Noctua.

 

[Stux]

I will definitely be running your fan controller script :) big thanks for all your hard work! I might have a few questions in regards to installing but that is later. Still need to install my hard drives and carry out burn in/ badblocks testing.

If I run Memtest for another say 2 - 3 hours then I'd be safe to say the ram has been tested sufficiently?

Thanks for all your help in the build thread mate, really do appreciate it :)

Start those HDs burning already ;)

While you do it, keep an eye on their temps.

 

[Sir SSV]

Start those HDs burning already ;)

While you do it, keep an eye on their temps.

I still haven't installed the hard drives in their enclosures yet lol

For the burn in process, I found this thread. Should cover everything I need to do?

https://forums.freenas.org/index.php?threads/how-to-hard-drive-burn-in-testing.21451/

 

[Stux]

Yes

 

[Sir SSV]

Yes

Thanks. Time to start reading up on what I need to do ;)

 

[Sir SSV]

Does anyone know how to upgrade the firmware on the BPN-SAS2-EL1 backplane?

I've read a post here but have no idea how to carry out the upgrade?

https://forums.freenas.org/index.php?threads/supermicro-x10srl-f-3-846-chassis-72-disks.30431/page-9

 

[nightshade00013]

Does anyone know how to upgrade the firmware on the BPN-SAS2-EL1 backplane?

I've read a post here but have no idea how to carry out the upgrade?

https://forums.freenas.org/index.php?threads/supermicro-x10srl-f-3-846-chassis-72-disks.30431/page-9

The below would probably be a good read, should explain how the upgrade is done with the tool Supermicro provides.
ftp://supermicro.com/utility/ExpanderXtools_Lite/Windows/ExpanderXtools_Lite_1%205.pdf

 

[Sir SSV]

Memtest has been completed and no ECC errors were reported in Supermicro ipmi log file so that's a relief. One thing, having 128Gb of ram certainly takes a while to test!

I know the below screenshot only has 2 passes completed, but before that 2 passes were completed on Memtest86 so 4 passes in total and no errors in log file is a good result for me

On to flashing of the Supermicro backplane. This was somewhat tricky as there is not a wealth of information out there on how to execute this procedure. However, now I know how to do it, it is dead easy.

I did try to start the firmware upgrade on my Windows 10 machine but to no avail. Both Supermicro ExpanderXtools Lite v1.5 and Xflash were reporting no HBA detected. Spending the best part of last night trying to diagnose this got me thinking... my personal desktop and laptop run Windows 10 Enterprise ltsb which doesn't have a load of drivers and I couldn't be bothered loading the Windows 8 drivers from the Supermicro website so I had a brain wave... borrow a desktop computer from work that runs Windows 7! Talk about dedication to FreeNAS :D

Once I installed the M1015 card in the Windows 7 machine, it was detected correctly and this is the procedure I did to flash my backplane. Lucky for me, my backplane was already on 55.14.18.0 but at least I know how to do it if I ever need to in the future :)

Flashing BPN-SAS2-846EL1 backplane

Install M1015 card into a Windows computer running Windows 7 (might work with Windows 8/8.1/10 but I didn’t have the time to check) and connect SFF-8087 cable from HBA in Windows machine to backplane of server. Make sure server is powered on!

Note: There is also a procedure to do this on Linux but I am more familiar with Windows hence the guide written below :)

Download SMC ExpanderXtools Lite v1.5 program and firmware files from

• ftp://211.214.88.34/bios/BP_FW/

Extract to C:\ and create a path within windows (when running Command Prompt, this will be able to search for folders/files without doing cd \)

Create a path in Windows by doing the following

• Right click on My Computer and navigate to properties
• Press Advanced System Properties in left pane
• Press Environment Variables
• Under System Variables, double click Path
• On Windows 7, place semicolon ( ; ) after last entry then type C:\SMCTools
• On Windows 10, click New and type C:\SMCTools
• Then OK on the windows to exit
• Restart system to ensure changes are active

Run command prompt (Windows key + r) then type cmd

In command prompt, type smc then enter

SMC ExpanderXtools Lite v1.5 will open

Under COM, select In Band

Details of HBA will appear in left pane labelled Expanders as well as your backplane details (Firmware, Version, Region, etc)

Note down SAS address then exit

OR

In command prompt, type

• xflash -i get avail

This will list SAS expander address as <xxxxxxxxxxxxxx>. Note this down

Back in command prompt, type

• xflash -i <sas address> down fw sas2xfw_55.14.18.0.fw 0 -y

Xflash will initialise interface, state current firmware, replacement version and then download to backplane

Shutdown server, remove HBA card from Windows machine and reinstall in server then power server back on... Process complete :D

 

[Sir SSV]

Small update, have just completed running short self test and all 12 drives passed.
When I tried to run a conveyance test, I received an error saying "failed: scsi error aborted command". Not quite sure what happened there? I replaced the SFF-8087 cable and still received the same error...
Anyway, I have commenced a long self test on all drives, the system notified me it would take 1250 minutes
One question, will the hard drive lights flash on the front whilst the short & long tests are being conducted?
Also, how can I check the progress of the long tests and is there a way for me to monitor hdd temps?

Edit: after reading this thread looks like the WD 8tb reds aren't supported in conveyance test
https://forums.freenas.org/index.ph...t-running-on-wd-reds-8tb-advice-please.55713/

 

[Stux]

Activity lights won't flash. There is no activity across the interface.

You can run smartctl -a /dev/adaXX to see the progress.

 

[Sir SSV]

Thanks Stux, server has been running over night completing smart long tests.

I did manage to find how to check progress remaining of test and also temperature of hdds using the

smartctl -a /dev/daX

Until this point, my hdds have been varying in temperature between 32 - 36C so quite happy with that and the Noctua fans seem to be doing their job well. However, once again I have changed the fan speed to Super I/O in Supermicro ipmi. Looking forward to running your script :)

Considering I am using an Intel X540-T1 10Gbe ethernet card, I have used 2 cat cables. 1 connected into the Intel NIC to connect/configure/run tests on the FreeNAS and 1 connected into the ethernet port so I can still access the Supermicro ipmi.

 

[Sir SSV]

Small update today, server has been running completing smart short tests and long tests with 0 errors so far :D

For those that may not have seen the burn in thread, a lot of great information is here

https://forums.freenas.org/index.php?threads/how-to-hard-drive-burn-in-testing.21451/

I used Putty to SSH into my server then executed the following commands.

To see a list of harddrives and their location, type:

camcontrol devlist

All my harddrives were allocated da0, da1, etc, da11

Then run smart short test:

smartctl -t short /dev/daX

Unfortunately, the conveyance tests don't play well with WD Red 8Tb hdds as found over here

https://forums.freenas.org/index.ph...t-running-on-wd-reds-8tb-advice-please.55713/

Run long smart test:

smartctl -t long /dev/daX

Note: this test took approximately 18 - 23 hours for all 12 of my WD Reds

Because no errors were reported, I moved on to testing for badblocks.

To perform raw disk I/O, disable a safety that prevents raw I/O to devices that are in use by the GEOM system, type:

sysctl kern.geom.debugflags=0x10

Note: The above command should not be run on a production system. Once all tests have been completed, please reboot system to disable

As the console is unresponsive during badblock testing (it runs in the foreground) and considering I was using SSH to conduct the tests, then type:

tmux

This means if my SSH connection is closed or disconnected, I can return to the badblock testing by simplying opening a new instance of Putty and typing

tmux attach

Due to my WD Reds being greater than 2Tb, I ran the following for badblocks

badblocks -b 4096 -ws /dev/daX

Because you can run badblocks simultaneously on all harddrives, after the first test has initiated, press

Ctrl + B, then Shift + ' (for ") to split pane horizontally. This allows you to initiate more badblock tests on the other harddrives.

However, one small issue I encountered when running multiple instances of badblocks was Putty threw me an error saying "Error: Pane too small"

To circumvent this, press Ctrl + B then spacebar. This will change the view of tmux. You will have to press this several times to return to the original window however the tests will be closer together. Unfortunately, when running tmux, each extra test you initiate will cut the workable screen by half thus causing you to run out of room. By changing the view, this essentially "resets" the view and allowed me to run 12 instances of badblock testing

This website is pretty cool for shortcuts whilst using tmux. I'm not sure how all commands work (as I wanted to commence badblock testing asap), but it would be interesting to issue some of the other commands :)

https://tmuxcheatsheet.com

Anyway, I will provide an update once badblock testing has completed. I'm anticipating this may be in say 5 years or so :p

So, now that badblocks testing has been completed (with 0 errors), complete 1 more smart long test by typing:

smartctl -t long /dev/daX

Once test has completed, check results

smartctl -A /dev/daX

Pay careful attention to the following entries:

• Reallocated_Sector_Ct
• Current_Pending_Sector
• Offline_Uncorrectable

These should all have a raw value of 0. Any result greater is means for an RMA on your drive

Ensure once these tests (and results) have been completed/viewed the system is restarted to re-enable the safety for kernel geometry debug flags

 

[Sir SSV]

Is there a way to check on hdd temps whilst running bad blocks? Can't seem to find any information regarding this

 

[danb35]

Sure--detach from the tmux session (Ctrl-B d), then smartctl -a /dev/ada0 (or whatever the drive ID is).

 

[Sir SSV]

Sure--detach from the tmux session (Ctrl-B d), then smartctl -a /dev/ada0 (or whatever the drive ID is).

Awesome! Thanks for the quick reply :)

 

[Sir SSV]

Just checked on all the hdd temps... they range from 32-37 degrees Celsius. Must say I'm very impressed with the Noctua fans. Even with the fanwall (120mm fans) the case is quiet considering the hdd's are under quite a bit of stress at the moment.

Next lot of 12 WD Red 8Tb on their way. Looks like more hdd burning in to come :D

 

[Sir SSV]

Whilst badblocks continue on my WD Red 8tb hdds... look what the courier was kind enough to deliver today ;)

 

[Sir SSV]

Finally finished badblock testing now on to final smart long tests.

For reference, the below is how long each individual test took just as a fyi on these new 8tb hdds

Smart short test: 2 minutes
Smart long test: 21 - 23 hours
Badblock testing: 136 hours

Hopefully this gives people an idea how long these burn-in tests take on new 8tb hdds

 

[Stux]

Are you sure the smart short tests took an hour :/