BUILD New Build - Seeking Opinions and a few clarifications

[Ericloewe]

I prefer Reds, at least, since they're guaranteed to support TLER (the Greens may reject having their timers shortened).

Greens work if you're willing to live with insanely long IO while a bad drive is in the system.

 

[Gilley7997]

I prefer Reds, at least, since they're guaranteed to support TLER (the Greens may reject having their timers shortened).

How long have your reds been running in your system now since I was looking at getting the same 3TB drives.

 

[Ericloewe]

~5200 hours.

 

[Gilley7997]

Well this is what I decided to go with and it's almost all put together.

• Motherboard: Supermicro X10SL7-F
• Processor: Xeon E3-1231v3
• Memory: Crucial 2x8GB DDR3-1600 Unbuffered ECC - CT2KIT102472BD160B
• Storage: 4 x 3TB Western Digital Reds
• USB: 2x32GB USB3.0 SanDisk Cruzer Fits
• Power Supply: EVGA 120-G1-0650-XR 650W Power Supply (It was a better deal than the Seasonic at the time)
• Case: Fractal Design R4

I am in the process of getting this all together and have ran 19 hours and about 6 passes on MemTest 86+ with no issues and ran all 4 hard drives through an extended test in DLGDIAG. No issues there either.

I will be flashing the LSI to IT mode here in a few moments.

Before I start experimenting more with FreeNAS itself, are there any other burn in tests you would recommend prior to even putting FreeNAS on the system. I searched the site a bit, but didn't find anything definitive. Was kind of shocked to not see a sticky on this, as good as the community is about putting stickies out there.

Any other guidance on things to do before I actually put the software on here, have a little time yet as I have to tidy up the cabling as the case is still sitting on the workbench with the doors off.

 

[Bidule0hm]

There's a thread about badblocks for drives burn-in here ;)

 

[Gilley7997]

There's a thread about badblocks for drives burn-in here ;)

Perfect Thanks!

Just a question, is it normal for WDLDIAG Utility to not find a drive if it was attached to the SAS Controller? They are being detected I can see them inside the configuration for the LSI. I tested them on the other controller for now, just a little quirky thing.

 

[Rilo Ravestein]

There's a thread about badblocks for drives burn-in here ;)

And it's a sticky... In the hardware forum... Go figure [emoji6]

 

[Gilley7997]

Okay so, I ran into a hiccup and some odd behavior from the Motherboard as well as FreeNAS and using these SanDisk Ultra Fit 32GB USB Drives. I am using the STABLE version of FreeNAS that was available as of last night to attempt this install.

Upon searching I found that partly I am experiencing the issue described in this bug.

The odd part about this is I am seeing the same odd behavior in the BIOS as described by Jason Gould in the discussion of the bug.

Upon initial install both SanDisk drives able to be selected as a boot device, (They are Both Listed with Device Name "SanDisk" so can't really tell them apart in the Bios screen) along with my install media (a 4GB OCZ USB Flash Drive).

Note the OCZ drive never disappears as a Boot Option.

This process below is 100% repeatable.
Take special note of the delay in System Initialization time as the drives begin to disappear from the BIOS.

Also note that Power off and Power on will reset the cycle.

1. Boot from Power Off
2. System Initializing...32 time = 3.5 sec.
3. Enter BIOS - Select OCZ Drive as Boot
4. Restart
5. System Initializing...32 time = 3.5 sec.
6. Boot From the OCZ.
7. Install FreeNAS to both Sandisks.
8. Install completes without issue
9. Restart the system. (Not Power Off)
10. System Initializing...32 time = 3.5 sec.
11. Check the BIOS all 3 devices are still reported and selectable as boot media.
12. Attempt to boot into FreeNAS. (Receive USB Errors described in bug).
13. FreeNAS Launches (Reports Boot As Degraded)
14. Restart the system (Not Power Off)
15. System Initializing...32 time ~ 1 Min and 20 sec.
16. Check the BIOS. Now only one SanDisk is selectable as boot device as well as the OCZ
17. Select the SanDisk as the boot device. Save Settings. Restart System
18. System Initializing...32 time ~ 1 Min and 20 sec.
19. Attempt to load FreeNAS. (Receive USB Errors described in the bug)
20. FreeNAS load stops at a "MountRoot" prompt
21. Restart the system (Not Power Off)
22. System Initializing...32 time ~ 2 min 34 sec.
23. Check the BIOS. Now the only selectable USB Boot device is the OCZ.
24. Let the system boot with no changes. Boots FreeNAS Install from OCZ Attempts to run. Reports USB Error from bug report During Install this time. (Note that this did not happen on first install). Only device listed for Install here is the OCZ Media.
25. Restart the system.
26. System Initializing...32 time ~ 2 min 34 sec.
25. Power Off System
26. Power On System
28. System Initializing...32 time = 3.5 sec.
27. All three USB Devices are selectable in the BIOS again as bootable.

So after this research I can only come to a few conclusions and someone can correct me if I'm off my rocker.

1.) The FreeNAS OS Boot and the USB_ERR_IOERROR leaves the SanDisk in a state that the system cannot initialize it after a system reset and it takes a power cycle to clear this state. The increasing time for System Initialization would lead me to believe that there is a timeout for attempts to initialize a detected hardware device and this is why the time doubles as the second device disappears.

Does anyone know if that System Initializing messages number 32 is the Device Detection/Initialization Phase? I'm just guessing here based on the data.

2.) There is some type of problem here with these devices and how they are ID'd. I can't distinguish the difference in the BIOS, is FreeNAS having the same issue during it's initialization? One drive (The first SanDisk attempted) is assigned a device ID successfully after the reboot, and then the errors start, and it varies whether the Error Message is Addr3 or Addr4 depending on which device gets mounted first.

3.) Any other thoughts?

I have a few screenshots of the messages that I get when FreeNAS tries to boot if they would help, but I think for the most part the bug report details that.

 

[JayG30]

That all seems inline with what I see. I think your conclusions make sense.
I would add that you don't actually have to reboot the system to fix the other USB stick as showing UNAVAILABLE. Simply removing the stick and plugging it back in makes it show back up and ZFS picks the drive up sets it back "ONLINE", restoring the freenas-boot pool. It seems to even assign it the original gptid as well.

At this point I'm at the point that I think this is an issue with GRUB, which will take someone from that project to resolve. From what I'm reading though, it sounds like they are sort of "unwilling" and as Jordan mentioned, people are pointing fingers. So I have little faith that this will get resolved in a timely manner. So now I'm looking for KNOWN WORKING USB sticks for running FreeNAS 9.3. Something I can feel confident won't magically stop working when the next major release happens.


PS: Tomorrow I'm going to attempt installing the latest 9.2 release and performing an upgrade to 9.3, just to check.

 

[Gilley7997]

So now I'm looking for KNOWN WORKING USB sticks for running FreeNAS 9.3. Something I can feel confident won't magically stop working when the next major release happens.

I'm with you here, I have been looking at signatures when I have been reading posts to see if anyone is running a pair of USB drives that work. Doesn't appear that many people put the boot devices in there sig's though.

Want to RMA these and get some new ones on the way to maybe play with them this weekend.

Any suggestions? Would like to at least have 16GB sticks.

 

[Rilo Ravestein]

The UltraFits work for me. I have four of them and tested them all. No problems so far.

 

[Ericloewe]

My Toshibas have been working fine.

 

[Gilley7997]

The UltraFits work for me. I have four of them and tested them all. No problems so far.

So your running a Super Micro board with a C226 chipset. The SuperMicro X10SL7 boards we have are running the C222.

Just curious are you running those UltraFit's in the 3.0 Ports or the 2.0 Ports on the board?

 

[Gilley7997]

My Toshibas have been working fine.

And your running the C224 chipset on that board, anyone have a set of drives that work on the C222 chipset?

 

[Ericloewe]

All the chipsets are the same. The C22x CPUs only differ in feature set.

 

[Gilley7997]

All the chipsets are the same. The C22x CPUs only differ in feature set.

The only reason I am looking at the difference is Rilo is using the exact same USB drives on the C226 chipset apparently without issue, but that JayG30 and I are attempting to use on the X10SL7 boards and experiencing boot issues.

I'm not saying that the Toshiba's wouldn't work for me, but before I purchase another set, just was looking for something on at least some matching hardware.

 

[Rilo Ravestein]

So your running a Super Micro board with a C226 chipset. The SuperMicro X10SL7 boards we have are running the C222.

Just curious are you running those UltraFit's in the 3.0 Ports or the 2.0 Ports on the board?

On the 3.0 ports. Had to use a 2.0 port for the install media, but installed to and running from the usb3.0 ports without problems. But i do think usb3 was disabled in BIOS by default (thus making it 2.0 ports effectively), but i'll have to check to be sure. This usb2/3 issue is mentioned several times on the forum already btw. I also thought this was covered in the manual, but i can't find it now (anymore).

 

[Gilley7997]

On the 3.0 ports. Had to use a 2.0 port for the install media, but installed to and running from the usb3.0 ports without problems. But i do think usb3 was disabled in BIOS by default (thus making it 2.0 ports effectively), but i'll have to check to be sure. This usb2/3 issue is mentioned several times on the forum already btw. I also thought this was covered in the manual, but i can't find it now (anymore).

Yep, so I wasn't even attempting to use the 3.0 ports on the board, but as you somehow got these devices to work, I'm willing to give it a shot and disable XHCI in the bios and drop them into the 3.0 ports just to see what happens. I am fearing I will have the same result. Won't be able to do that until after work tonight though.

 

[Ericloewe]

All the chipsets are the same. The C22x PCHs only differ in feature set.

 

[Gilley7997]

Yep, so I wasn't even attempting to use the 3.0 ports on the board, but as you somehow got these devices to work, I'm willing to give it a shot and disable XHCI in the bios and drop them into the 3.0 ports just to see what happens. I am fearing I will have the same result. Won't be able to do that until after work tonight though.

Doesn't make a difference in the 3.0 ports either. Will just send these back and get the SanDisk 16GB Fits instead. Time to get this up and running on an old drive just to continue getting the system burnt in.