FreeNAS 8 reboots continuously when all 10 SATA ports filled

[Mr. Bungle]

I just put together my first system to run FreeNAS:

• Asus M4A89GTD motherboard (http://www.newegg.com/Product/Product.aspx?Item=N82E16813131631)
• Kingston KVR1333D3E9SK2/8G DDR3-1333 8GB (2x4GB) ECC memory (http://www.newegg.com/Product/Product.aspx?Item=N82E16820139262)
• AMD Athlon II X2 260 Regor 3.2GHz CPU (http://www.newegg.com/Product/Product.aspx?Item=N82E16819103873)
• (10x) HITACHI Deskstar 0S03230 3TB SATA 6.0Gb/s hard drive (http://www.newegg.com/Product/Product.aspx?Item=N82E16822145493)
• (2x) HighPoint Rocket 620 PCI-Express 2.0 SATA 6.0Gb/s controller card (http://www.newegg.com/Product/Product.aspx?Item=N82E16816115072)
• IDE to Compact Flash CF Adapter (http://www.monoprice.com/products/p...=10407&cs_id=1040705&p_id=2105&seq=1&format=2)
• Transcend 4GB Compact Flash TS4GCF133 (http://www.newegg.com/Product/Product.aspx?Item=N82E16820208339)
• Intel EXPI9301CTBLK PCI-E network adapter (http://www.newegg.com/Product/Product.aspx?Item=N82E16833106033)
• Antec EarthWatts Green EA-380D 380W PSU (http://www.newegg.com/Product/Product.aspx?Item=N82E16817371033)
• Antec Three Hundred Illusion ATX case (http://www.newegg.com/Product/Product.aspx?Item=N82E16811129066)
• Cooler Master STB-3T4-E3-GP 4-in-3 HDD cage (http://www.newegg.com/Product/Product.aspx?Item=N82E16817993002)

It's all assembled now, and I loaded FreeNAS 8.0.1-RC2 onto the CF card and was able to boot up successfully and log in via the web interface. However, I noticed while booting that only 9 of my 10 HDDs was recognized in the BIOS, which was also reflected in the FreeNAS UI. After some troubleshooting, I narrowed it down to a bad SATA cable, so I swapped that out, and now all 10 drives are recognized in the BIOS.

However, now FreeNAS won't boot. I get through the POST and BIOS steps just fine, but right when I'd normally start to see FreeNAS booting up, the machine just resets and goes through the process again, and does this continuously.

Not sure what had happened, I tried everything I could think of to narrow this down:

• changing the boot device settings in the BIOS
• removing the Intel NIC (I saw the Intel Boot screen, which I hadn't noticed before, and thought maybe the NIC was trying to initiate a network boot)
• installing FreeNAS on a USB stick and booting from that
• reinstalling FreeNAS on the CF card and trying again

None of those steps fixed the problem, but then I remembered that everything had worked fine when only 9 HDDs were connected at first. I removed one of the drives, and it worked! FreeNAS booted again.

So, I've narrowed the problem down to FreeNAS perpetually rebooting only when I have all 10 SATA ports filled. This happens whether I have the IDE port filled with the IDE/CF adapter and am booting off of a CF card, or whether I remove that adapter and boot from USB.

Does anyone have any ideas about what may be happening, and possible steps to fix it? Thanks in advance!

 

[cubix]

Hi there,

Is the suspect SATA port onboard or on one of the PCI-E cards? Can you disconnect any of the drives for FreeNAS to boot?

 

[Mr. Bungle]

Hi there,

Is the suspect SATA port onboard or on one of the PCI-E cards? Can you disconnect any of the drives for FreeNAS to boot?

Good question - so, it appears that FreeNAS will boot successfully only if I disconnect one or more of the motherboard's SATA ports (it doesn't appear to matter which one; I tried three different ones, and it booted each time).

If I disconnect one or more drives from either of the Rocket 620 HBAs, the computer still reboots continuously at the point where FreeNAS should come online (after the POST and BIOS tasks). So the problem appears to occur when all 6 native SATA ports on the motherboard are filled (at least with HDDs; I haven't tried an optical drive or anything else yet).

I'll check my motherboard's manual now to see if it says anything about this. Please let me know if this helps narrow it down in any way. Thanks!

 

[cubix]

It definitely helps, but I was thinking it might have been the addon cards - your motherboard has different speed PCI-E slots and thought that might have been the issue. But just to make sure, are you able to pull both addon cards and leave the 6x onboard sata drives connected and report back?

 

[Mr. Bungle]

Hey, good call! I removed both of the Highpoint HBAs, and the system booted fine. What do you think that means?

The slots I have available on my motherboard (as listed in the manual) are:

• PCIe 2.0 x16_1 slot (gray, at x 8 link)
• PCIe 2.0 x1_1 slot
• PCIe 2.0 x4_1 slot
• PCIe 2.0 x16_2 slot (blue, single at x 16 or dual at x 8 link)

I have the HBAs plugged in to the two x16 slots. Should I try moving them? If so, where to?

Also, this reminded me that I'd seen some talk of somehow updating the HBAs (maybe via firmware?) to be effectively transparent to the system; in other words, they'd function more or less like native motherboard SATA ports. Do you know anything about that? Could the fact that I'm seeing both of them POST individually during the boot process mean that they might not be set up ideally since I just need them for extra SATA ports and nothing else?

 

[cubix]

Bit of a long shot, but possibly because of shared IRQ assignments. It looks like your cards are PCIe x1 cards, so I'd be using the PCIe x1_1 and PCIe x4_1 (if they fit).

 

[Mr. Bungle]

Thanks - I tried that, but no joy. Does that mean it's not likely an IRQ issue, or does that not necessarily rule that out?

 

[Mr. Bungle]

FWIW, I'm reading through the motherboard manual now, and it says that when cards are installed in both PCIe 2.0 x16 slots, they both run at x8 speed - so my original configuration should have had them both running that the same speed.

 

[cubix]

Unfortunately I don't know. Hopefully one of the hardware pros stumble across this thread and provide some input. It does seem like something is up with the PCIe cards rather than the onboard sata ports.

FWIW, I'm reading through the motherboard manual now, and it says that when cards are installed in both PCIe 2.0 x16 slots, they both run at x8 speed - so my original configuration should have had them both running that the same speed.

I'm assuming that's for dual video cards? Your addon cards are only x1 so that shouldn't be an issue.

 

[Mr. Bungle]

Ah, OK. Yeah, that makes sense - the x8 is an upper bound, so two x1 cards shouldn't affect that.

Anyway, no worries - thanks for your help! You've helped me isolate it with some confidence to the HBA cards, so that definitely helps. I've read through my motherboard manual and didn't see any red flags about anything, so now I'm researching the HighPoint 620 adapters to see if there are alternative drivers that I need to install or anything like that.

I noticed that one person seemed to get very responsive support from a FreeNAS Trac ticket - do you know if that's a publicly accessible ticket system, or how he alerted the developers to the problem? I don't want to bother them if I can figure this out on my own, but as a last resort I might need to open a ticket, if possible.

 

[b0bje]

That's a whole bunch of hardware for a 380Watt PSU.
Have you checked the bios to see how your 12 volt rails are doing ?

Think it is quite plausible that adding that 10th disk is pushing it on your power.

 

[ProtoSD]

I noticed that one person seemed to get very responsive support from a FreeNAS Trac ticket - do you know if that's a publicly accessible ticket system, or how he alerted the developers to the problem? I don't want to bother them if I can figure this out on my own, but as a last resort I might need to open a ticket, if possible.

Anyone can create a ticket on the FreeNAS Trac ticket system, you just need to register for an account, it's free. It's generally for reporting problems and bugs, the developers are pretty responsive, but that doesn't mean they respond to every single problem. Give it a try if you want, no one will get annoyed with you.

 

[Mr. Bungle]

That's a whole bunch of hardware for a 380Watt PSU.
Have you checked the bios to see how your 12 volt rails are doing ?

Think it is quite plausible that adding that 10th disk is pushing it on your power.

Thanks, b0bje - that's definitely a possibility, but at this point, I'm not really suspected a power issue. However, what specifically can I look for in the BIOS to find any warning signs around that? The voltages all look OK - all are within about 1% of the nominal voltage - and my Kill-a-Watt indicates ~220W peak power draw (while the system is booting and all 10 drives are spooling up at once) and ~120W at idle.

Please see my next post for some more clues about what I've found via my latest round of troubleshooting.

 

[Mr. Bungle]

Anyone can create a ticket on the FreeNAS Trac ticket system, you just need to register for an account, it's free. It's generally for reporting problems and bugs, the developers are pretty responsive, but that doesn't mean they respond to every single problem. Give it a try if you want, no one will get annoyed with you.

Thanks, protosd! I just put in a ticket with FreeNAS, and also a ticket with Asus (motherboard) and HighPoint (Rocket 620 HBAs) since I don't really know which specific component/software is at the root of the problem.

Here's another clue - I found that the IDE/AHCI setting is critical. With all 10 disks attached an in AHCI mode, I can't boot. However, changing to IDE mode for all 6 of the onboard SATA ports allows me to boot just fine!

To summarize:

• If I set all 6 motherboard SATA ports to IDE mode, the system boots successfully. The BIOS allows you to specify AHCI/RAID/IDE separately for ports 1-4 (as a group) and ports 5-6 (as a group). You can set all 6 as AHCI, all 6 as IDE, or 1-4 as AHCI and 5-6 as IDE (the BIOS says something about using the latter to allow the system to recognize optical drives for installing the OS).
• If I set ports 1-4 to AHCI and ports 5-6 to IDE, the system also boots successfully.
• If I set all 6 motherboard SATA ports to AHCI mode, the system does not boot when all 10 drives are connected. To get it to boot, I can either (A) remove one or more SATA cables from the motherboard's ports, or (B) remove one of the Rocket 620 HBAs from its PCIe 2.0 slot (or just both SATA cables from it, still leaving the card in the slot). I was wrong about the latter - I thought I'd tried it with no success, but apparently that was not the case!
• One approach I kind of expected to work that does NOT work is to remove one SATA cable from each Rocket 620 HBA (8 drives total). I have to remove both SATA cables from either Rocket 620 HBA.

FWIW, I noticed that the BIOS for the 2 Rocket 620 HBAs list "Mode: CPU AHCI" so I assume that the Rockets can only run (or at least run by default) in AHCI mode.

So, with all that in mind, some new questions:

• Does this new information reveal anything more about the cause?
• Should I just give up and leave the motherboard SATA ports in IDE mode so I can boot this thing already? Or is AHCI worth more troubleshooting?
• Does this suggest any particular component(s) as the culprit, or safely rule anything out?

 

[Mr. Bungle]

Here's the FreeNAS ticket: http://support.freenas.org/ticket/838

 

[b0bje]

From your further investigation I would say it's 99% unlikely to be a power issue.
Seeing as ahci should normally make your drives use less power (AHCI ALPM) then in regular IDE Mode.

To completely rule power issues out, you can hook up a multimeter to a free molex and have a look if the 12v rail has any weird dropouts while in ahci mode. (if the PSU isn't able to sustain enough amperage on the 12v rail, you could have a 1000w PSU and it still won't work)