10GBe Card Prevents Boot

[cinemafunk]

I purchased two used HP 561T (717708-001) on eBay (link). When I add this card to my TrueNAS box (hardware below), it prevents the machine from booting. The has an indefinite blinking cursor after POST and SAS card initialization. Once the card is removed, the machine boots as expected. I also removed 4 HDD power to see if the PSU was under powered and experienced the same result. I am also unable to enter the BIOS.

Both cards work in my Proxmox Box (hardware also below). Both are immediately discovered and performant.

Not finding much helpful information with my Google-fu.

What is it about this card that is preventing my TrueNAS machine from booting? I still have a hunch that the PSU might be underpowered. Thoughts? Thankful for any help.

TrueNAS Machine

• SuperMicro X9SCM-F
• Intel Pentium G2140 3.30GHz Processor BX80637G2140
• 4x Kingston ValueRAM 8GB DDR3 SDRAM ECC Unbuffered KVR1333D3E9S/8G
• 4 WD RED WDC WD40EFRX-68N32N0 - 4TB
• 4 WD RED Plus WD80EFZZ - 8TB
• 2 PNY SSDs (boot drives)
• 2 PNY SDDs (slog)
• 1 nvme (cache) via PCIE card
• EVGA 450 B3, 80+ Bronze 450W
• Dell H310 6Gbps SAS HBA w/ LSI 9211-8i P20

Proxmox Machine

• Intel Core i7-9700
• GIGABYTE C246M-WU4
• 4x CORSAIR Vengeance LPX 32GB DDR4 DRAM 2400MHz C14
• Riitop M.2 to PCIe Adapter
• 4x 1tb nvme
• 2x 2TB HDD
• 4 SSDs
• Corsair CX450M — 450 Watt

 

[Ericloewe]

What's the chipset on those things?

 

[cinemafunk]

What's the chipset on those things?

Hello, it looks like it is a Intel X540 processor according to HPE.

 

[Ericloewe]

Ok, if this is happening before the OS takes over, as it sounds, it's likely to be a bad/incompatible/whatever boot ROM. Have you disabled all extension ROMs in the system firmware setup menu?

 

[cinemafunk]

Hmm. Not to sure where I may find that, and I'm entire sure I'm understanding either. I've attached some screenshots of areas of my BIOS that my be pertinent to this.

I found this article on Supermicro that might point to some workarounds. One of them is "Disable the Preboot Execution Environment (PXE) ROM of on-board NICs (Via BIOS configuration)." I believe that would be the "OnBoard Lan Option Rom Select" option which is currently set to "PXE".

 

[Ericloewe]

See all those PCIe OPROM entries? Disable them.

 

[cinemafunk]

After disabling Slots 4, 5, and 7, I was unable to boot. After re-enabling Slot 7 which houses my HBA card, I was able to boot again.

Once confirmed all operations with a booted TrueNAS, I again re-added the NIC, and received the same result: indefinitely blinking cursor and no boot, and unable to enter BIOS.

 

[Ericloewe]

Ok, so it's probably down to something on the SMBus. In which case, there's not much you can do. If you're feeling adventurous, you could mask out the SMBus pins on the PCIe card edge connector with appropriate isolating tape.

 

[cinemafunk]

Yuck. I'm not feeling comfortable with that sort of workaround Well, I'll keep this card around as a backup.

In the meantime, I don't want to give up on 10GBe. Do you recommend any dual port RJ45 10GB cards that are compatible with TrueNAS and my particular board: X9SCM-F? At this point, I'm open to anything under $200 new or used.

Thank you for all your help.

 

[Ericloewe]

Honestly, you had terrible luck because damn near anything should’ve worked better. You were probably bitten by HP customisations…

If you absolutely need twisted pair, get an Intel X540-T2, either stock or from an OEM with a smaller firmware team (e.g. Supermicro). They should be readily available and priced decently on the secondary market.

 

[Dice]

I ran into a similar (although I can't recall the exact details) issue when adding a 10GbE NIC.
I never got the card to show LINK at all, and the machine had troubles booting at times.

The problem at the time was not related to the NIC really, but instead an IRQ conflict that occured when PCIe slots were populated in a particular way.
I recall I tracked it down by shuffling around the extension cards, while taking not on what IRQ they were achieving depending on slot and configuration.

Somehow (vague memories) I got it to work by moving all cards around to a particular setup.

Have you tried (sorry if I missed it) booting the machine with only the NIC as extension card, and see if it boots in <any> of the slots?
Also with a BIOS reset?

I guess you can find the IRQ on lspci -v

 

[Ericloewe]

Might not have been an IRQ issue, since that stuff should mostly live in the past, at least as an issue. But the point is a good one, different PCIe slots might be wired up to different SMBuses and chances are if you can find a PCIe slot that is wired up to a different bus, then things might work okay.

 

[Dice]

Might not have been an IRQ issue, since that stuff should mostly live in the past, at least as an issue.

Yeah, so I was told too :)
When I got back and I mapped out the situation in detailed and asked for other conclusions - then there were.... a good number of "wtf moments" around the table.

It still may of course not have been the <actual> reason, but the correlation was stupid strong.
(Board was a X11-SSL).

 

[Ericloewe]

X11SSL-F, huh. Supermicro does seem to cheap out a bit on the SMBus layout in ways that typically don't cause problems until weird OEM-customized cards start showing up.

 

[cinemafunk]

As for trying out changing up the order of the PCI cards, I'm down for that perhaps this weekend, and report back.

Otherwise, I've been seeing that X540-T2 are recommended frequently. I'll take the advice with going with Supermicro board.