SOLVED X11ssm-f DIMMB2 disappears after adding PCIe SLOG

[GBillR]

I hope someone has seen this before. I have a typical x11ssm-f setup with FreeNAS on bare metal. I recently migrated from my ESXi to a simpler setup. Everything has been rock solid for at least 3 years.

I recently picked up a DC P3700 HHHL card to use as a SLOG. I loaded the card into another machine, booted and used the intel tools to overprovision the drive in preps for use as a SLOG. My problem is that when I insert the SLOG into either PCH slots 4 or 5 (does not matter) on my FreeNAS machine, I mysteriously lose a DIMM slot. Freenas recognizes the new SLOG (I can see it in the drives list in the GUI). Everything else is normal except I only have 1/2 my RAM.

I swapped sticks between slots with no change in results. The only way to gain DIMMB2 back was to remove the SLOG. I am considering moving the ram to other slots, but the supermicro manual says to populate slots in a specific order... B2 being the first one to use, followed by A2. The stick in A2 (either stick) works with the SLOG installed, but B2 does not. If I try to boot with a single stick in B2 with the SLOG also installed, the board will not boot and complains there is no RAM. Remove the SLOG, all is well.

I looked into my BIOS settings and everything I think could make a difference is set to default values. I did try to change a couple settings with no success. I toggled the following BIOS settings (one at a time) from their defaults (all have since been returned to default) in an attempt to resolve. Defaults are in bold:

Memory Configuration-->Fast Boot--> Enabled to Disabled​

PCIe/PCI/PnP Configuration-->Above 4G Decoding-->Disabled to Enabled​

PCIe/PCI/PnP Configuration-->PCH Slot 4(5) PCI-E X4 OPROM-->Legacy to Disabled​

Here are my specs:

x11ssm-f​

e3-1230v5​

2 x M391A2K43BB1-CPB 16GB DDR4-2133 ECC (populated in B2 and A2 per the manual)​

x520-DA2 (slot 6)​

LSI 9207-8i (slot 7)​

DC P3700 400 GB (slot 5)​

Seasonic SS-660XP2​

8 x 4 GB WD Red (4 mirrored vdevs stripped into 1 pool)​

250 GB 850 EVO boot drive​

FreeNAS 11.3 u5​

​

I am hoping someone has seen this, but my search of the board and google has not been helpful....

Thank you in advance.

​

 

[GBillR]

UPDATE -

I've done some more troubleshooting and have uncovered some interesting results. First I updated the x11 bios to the latest version (2.5). I was on 2.4, so did not expect this to fix anything, but figured it was a good place to start. Reset bios to optimized defaults and rebooted with no change. DIMMB2 is seemingly not recognized. To try to rule out a bad or overloaded power supply, I unplugged all peripheral plugs from the power supply and booted with just the motherboard connected... no change.

I then moved the ram to the other slots (black slots, B1 and A1) and the bios now sees both sticks. Since supermicro specifically lists slot order of preference when installing ram to optimize performance (as shown below), this seems like a "workaround" for whatever my problem is.

Very interesting is my IPMI sensor readings in this configuration. The system is reporting DIMM temps as if 3 slots are filled! B1 and A1, which are populated with ram and ipmi shows both at 21 degrees. Strangely, DIMMB2 shows a temp of 32 degrees, as if ram was installed.... but the slot is empty. DIMMA2 (the other empty slot) shows its temp as N/A. Additionally, it seems as if a new sensor labeled NVMe_SSD Temp has appeared in my list of IPMI sensors with a status of N/A.

So it seems as if the PCIe card is being somehow recognized as a stick of ram by the bios and therefore disabling the DIMMB2 slot??

Has anyone ever seem a supermicro board (or any) behave this way, or does anyone have any suggestions? I am going to submit this to supermicro support, but I have no experience doing so, and I'm not sure what the response time or quality might be....

 

[GBillR]

UPDATE -

So Supermicro support has come thru. I now just would like to know why what they instructed me to do works, and if there are any negative consequences from implementing their solution:

The simple solution was this: Disabling the MB's Jumpers: JI2C1/JI2C2.

Per the manual, these jumpers disable the SMBus to PCI Slots.

After digging around some more now that I had a lead, I ran across this post, along with many others, were tape was used to cover pins B5 and B6 on a HBA to get the motherboards to POST... I suspect that this solution is similar to disabling those 2 jumpers on the X11 board, but effectively doing so for ALL PCI slots.

I am a little perplexed as to how no one else seemingly has had this problem before me on this forum... I think I am using pretty typical hardware, that many before me have used. In my searches I can't find anyone with an PCI SSD that had similar symptoms.... mostly just HBAs not being able to POST. My symptoms were a little different... my board at least booted fine, but I suspect that to be because I also had another ram slot populated. Maybe I am not searching for the right terms. Or is it some firmware version problem? I updated the firmware on the ssd prior to installing into my FreeNAS machine, so maybe a newer firmware is causing the issue? If this is a more common issue, maybe it belongs in one of the resource docs?

I have to assume that since this affected the B2 DIMM slot (which is the first to be populated on this MB per the manual), that even those with a single stick of ram should be impacted if installing this ssd. Perhaps my searches were not efficient and someone can point me to a post where someone else had this problem before me. Or maybe I am one of the few to populate an X11SSM-F with and Intel DC P3700??? I can't believe that to be true.

So my questions to those who might know better than me:

1. Is the solution proposed by Supermicro equivalent to taping B5 and B6?
2. Would it be better for me to tape over B5 and B6 on the P3700 and restore the jumpers back to default? I have an LSI SAS and X520-DA2 installed as well, both were installed before these problems began.
3. Does disabling the SMBus to PCI cause any negative side effects?
4. What exactly has been disabled?

Appreciate any insights provided.

-Bill

 

[jgreco]

The SMBus is primarily (someone will argue I'm wrong) used for out-of-band communications, such as letting the IPMI subsystem inspect the status of your RAID controller. Highly integrated server systems like Dell and HP use this capability to analyze and monitor the system's hardware inventory, and stuff like that, reading back device serials, configurations, etc. In general, it does not seem to have much use on hobbyist-grade build-yer-own servers.

I take it that the Supermicro-provided solution is similar to taping B5 and B6, as you suspect. I see no value in worrying about this, however. You are probably fine as-is.

 

[GBillR]

Thanks for your insight. I appreciate the reply.

To close the loop on this for anyone who happens to run across this down the road, the official response I received from Supermicro regarding the why this was happening is as follows:

Code:

Hi Bill,

It is known issue of Intel DC P3700 AIC using same SMBUS address of DIMMB2 slot.

The disalbing I2C jumpers solution is to disble the I2C bus and MB won't detect the AOC's sensors.
The IPMI on this board does not have feature to detect third party AOCs temp, so it does not have aspects to IPMI.

Thanks,

TX.  

So it seems as if my suspicion that the MB was treating the P3700 as if it were a stick of ram in the first DIMM slot was not too far off base.

I still am surprised that I can't find a post here where someone else ran into this with the P3700 on an X11SSM-F.

EDIT: Link to Supermicro FAQ (which apparently I was unable to find in my search of their site, probably since it references another X11 board)
https://www.supermicro.com/support/faqs/faq.cfm?faq=23482

 

[Ericloewe]

In general, it does not seem to have much use on hobbyist-grade build-yer-own servers.

I've seen it used to wire up fan controllers on PCIe card to M.2 adapters, but I fully expect it to work with like one specific line of motherboards and cause a world of hurt on everything else.