New X9SCM-F-O build, 32gb Kingston kit crashing platform

[loganphyve]

Hi All,

I've been planning to set up a freeNAS box for a while now and finally pulled the trigger and ordered parts for a complete system after exhaustive reading on supported hardware. I'm having all sorts of stability issues using what I thought to be compatible hardware, which was purchased after reading the recommended hardware lists.

Board: SuperMicro X9SCM-F-O (New, latest BIOS)
Chip: Intel Xeon E3 1230 V2 1155 (New)
RAM: Kingston 32gb Kit: KVR16E11K4/32 , 4x 8gb sticks PC12800/DDR3 1600 ECC
HBA: IBM ServeRaid M1015 CrossFlashed to IT mode (New Pull)
Addin NICs: (2x) HP NC382T PCI Express4x Dual Port Server Adapter (New Pulls)

After searching around here and other forums, I am seeing some folks are having an issue with this Kingston Ram kit.

The system will POST and correctly recognizes all hardware. Bootable/Live OS's run fine under no-load circumstances, including the most recent FreeNAS USB image 9.2.1 which runs until zfs manager tries to create or manage a zpool. When I started playing around with FreeNAS to get familiar with it (before running hardware and burn-in testing), the box randomly crashes with Kernel panics. After a bit of troubleshooting, I think I have the issue narrowed down to RAM. I've run MemTest86 against the the RAM in question. The results are... unexpected.

Memtesting all 32GB halts the board in its tracks around the 18GB mark, several seconds in. Memtesting 1x 8gb and 2x 8gb in the Dimm 2a/b slots works like a champ. Memtesting each stick individually works as well. Adding in a 3rd (non-optimal but supported) stick halts the board in its tracks. It appears that this RAM is not working in amounts over 16gb.

I have plans to use this box for a few tasks. It's going to be hosting a large CIFS share for hosting video, audio, and picture data, sFTP, as well as some iSCSI LUNs for VMware datastores. So, I'd really like to use all 32GB of ram since the storage pools will be quite large. I eventually want to populate it with 12x 4+tb drives.

Has anyone else experienced this issue, what did you do to solve it, and if you had to replace memory, what do you have that works for certain? I've searched hi and lo looking for a solution but haven't found anything yet.

I think I'm going to have to return the memory for another brand. Oddly enough, Kingston says this kit works with the board in question...

http://www.kingston.com/us/memory/search?DeviceType=7&Mfr=SMI&Line=X9SCM-F&Model=70800

however SuperMicro says it doesn't... I only checked the RAM mfg site before purchase. (this is also the most-purchased 32gb kit for this board according to amazon)

http://www.supermicro.com/support/r...D650E3EC19339&prid=0&type=0&ecc=0&reg=0&fbd=0

I've looked all over the forums and have seen similar issues with other SuperMicro boards and Kingston Ram but haven't found anything helpful enough to solve the issue. First time post, please forgive me if I have this in the wrong area or it's been solved and I haven't found it yet.

Thanks in advance,
LoganPhyve

 

[cyberjock]

Well, I'd try swapping memory sticks around. If having a 3rd stick(regardless of which stick it is) causes the system to lockup, you are left with 4 basic possibilities:

1. That RAM slot is bad.
2. The RAM isn't compatible(I consider this unlikely since the problem has been on X10s and not X9s).
3. The motherboard is bad in some fashion.
4. Some outside influence(such as a PSU) is responsible for the issue.

To be honest, just by your comments that you want to use iSCSI for VMWare datastores and plan to use 12x4TB drives kind of tells me that 32GB won't be enough.. but I'll let you figure that out for yourself.

 

[loganphyve]

Well, I'd try swapping memory sticks around. If having a 3rd stick(regardless of which stick it is) causes the system to lockup, you are left with 4 basic possibilities:

1. That RAM slot is bad.
2. The RAM isn't compatible(I consider this unlikely since the problem has been on X10s and not X9s).
3. The motherboard is bad in some fashion.
4. Some outside influence(such as a PSU) is responsible for the issue.

To be honest, just by your comments that you want to use iSCSI for VMWare datastores and plan to use 12x4TB drives kind of tells me that 32GB won't be enough.. but I'll let you figure that out for yourself.

I've already swapped the mem sticks around. Like I mentioned, each one tested alright when run alone. In this case, I might want to consider replacing the board. Do you know of any utilities that can test the memory one stick at a time if all are populated?

I'm certain this is either a board or memory issue but would really like to know for sure. I'm going to run memtest against both pairs, one set at a time for endurance testing to see if any of the sticks fail out over a given period.

Any other ideas would certainly be appreciated... and thank you very much for the speedy response.

Not to go OT, but as far as RAM utilization goes, this board maxes out at 32gb. I don't need a full 12x 4tb disks, but it'd be nice for the room. Should I consider using a set of 4's for the CIFS zpool and a smaller set for the iSCSI zpool? I'd like to have 4+tb available for iSCSI access and 12+tb available for a network share. I'd be running a handful of VM's on a single ESXi host. I know zfs is RAM hungry and I've heard mixed things regarding whether it's enough to do what I want. Some folks report that 16+ is adequate for home use... but I'm not the expert here.

 

[cyberjock]

I'm not aware of any way to test one memory stick at a time since the memory controller has a definitive decision on how memory is mapped.

 

[panz]

Hi All,
Board: SuperMicro X9SCM-F-O (New, latest BIOS)
Chip: Intel Xeon E3 1230 V2 1155 (New)
RAM: Kingston 32gb Kit: KVR16E11K4/32 , 4x 8gb sticks PC12800/DDR3 1600 ECC
HBA: IBM ServeRaid M1015 CrossFlashed to IT mode (New Pull)

Memtesting all 32GB halts the board in its tracks around the 18GB mark, several seconds in. Memtesting 1x 8gb and 2x 8gb in the Dimm 2a/b slots works like a champ. Memtesting each stick individually works as well. Adding in a 3rd (non-optimal but supported) stick halts the board in its tracks. It appears that this RAM is not working in amounts over 16gb.
LoganPhyve

I have I setup like yours, using 32 GB of Kingston KVR16E11/8: I bought the sticks separately, not the kit, but should be the same.

I ran Memtest86+ and HCI memtest for about 12 days (!) with no issues at all.

Now my system has a first RAIDZ2 pool of 3TB hard disks and another set of 6 disks doing some intensive burn-in tests. This morning I just copied some 6 TB of data from one dataset to another (this is supposed to be disk intensive but RAM intensive too) without issues.

Some problems you're reading in the fora are affecting the X10 series.

Is the RAM seated correctly? Did you use all the ESD protections (wrist strap, antistatic mat, case grounding, ESD certified tools, etc.) when assembling the hardware?

As @cyberjock says you should investigate the other components too: a defective PSU could cause more damage than you could imagine.

A friend of mine did a build using a thermal compound that had some sort of metallic component in it and he accidentally dropped a tiny bubble of it into one RAM slot, resulting in a little short circuit and a consequently erratic behavior.

 

[loganphyve]

I have I setup like yours, using 32 GB of Kingston KVR16E11/8: I bought the sticks separately, not the kit, but should be the same.

I ran Memtest86+ and HCI memtest for about 12 days (!) with no issues at all.

Now my system has a first RAIDZ2 pool of 3TB hard disks and another set of 6 disks doing some intensive burn-in tests. This morning I just copied some 6 TB of data from one dataset to another (this is supposed to be disk intensive but RAM intensive too) without issues.

Some problems you're reading in the fora are affecting the X10 series.

Is the RAM seated correctly? Did you use all the ESD protections (wrist strap, antistatic mat, case grounding, ESD certified tools, etc.) when assembling the hardware?

As @cyberjock says you should investigate the other components too: a defective PSU could cause more damage than you could imagine.

A friend of mine did a build using a thermal compound that had some sort of metallic component in it and he accidentally dropped a tiny bubble of it into one RAM slot, resulting in a little short circuit and a consequently erratic behavior.

Yea, I've seated and reseated the RAM. ESD precautions were taken during setup. I just finished a first pass on memtest with two of the 4 sticks with no issues running in the primary slots. I'm testing the second set now to see if they pass (edit, the second set *just* finished. Passed).

The PSU is also new and has almost no runtime on it except for a few weeks testing with another supermicro board. Pretty sure I'm OK there. It behaved well before. I'll double check the voltages while re-creating the problem though, just to be sure.

It would not surprise me at all if there was a board issue. It was packaged very poorly from the vendor. I was kind of upset about that, actually. It could very well have been damaged in transit. I'm going to do some more testing but I'm pretty sure it's a bad board. It wouldn't be the first time I've had a DOA or damaged part show up.

Thanks :)

 

[loganphyve]

Looking at PSU voltages in BIOS, they appear within tolerable ranges. 12V is sitting a tad high at 12.498-12.551 V but it's within the (±0.60 V) acceptable range. Is this worth being concerned over? I generally accept circumstances like this if they're within tolerances.

All other values are within a few hundredths of expected. RAM voltage sitting at 1.528V, looks ok.

 

[panz]

I don't trust the BIOS for voltages check; I prefer old-style Fluke voltmeter ;)

Definitely could be a bad board. Some stores sell parts that are shown to other customers, touching them without any protection. Bad UPS handling is an issue too.

 

[cyberjock]

Voltage isn't all that matters for a PSU. Voltage fluctuations that you'd only see with an oscilloscope can cause system instability and actually damage hardware. So I wouldn't necessarily take those voltages as evidence the PSU is just fine. The easiest way to rule out a PSU with near 100% certainty is to simply plug in a spare PSU and see if the problem persists.

Also, I'm not sure what test you are running on your RAM, but RAM isn't generally considered 'good' without 3 passes with a program like memtest. I'm not sure you are actually getting to 3 passes because typically testing just 16GB of RAM with 3 passes takes 3+ hours...

 

[loganphyve]

I'll meter it just to be sure.

I'm not sure if you'd laugh or cry if you saw how the board was packaged... it wasn't good. It was intact, but there wasn't much to protect it.

 

[loganphyve]

Voltage isn't all that matters for a PSU. Voltage fluctuations that you'd only see with an oscilloscope can cause system instability and actually damage hardware. So I wouldn't necessarily take those voltages as evidence the PSU is just fine. The easiest way to rule out a PSU with near 100% certainty is to simply plug in a spare PSU and see if the problem persists.

Also, I'm not sure what test you are running on your RAM, but RAM isn't generally considered 'good' without 3 passes with a program like memtest. I'm not sure you are actually getting to 3 passes because typically testing just 16GB of RAM with 3 passes takes 3+ hours...

I wanted to get through at least one complete test cycle for both sets of sticks before putting in many more hours. I thought it best to do a single pass on each first just to see if one set was dead obvious. Guess I'll run a few more.

Unfortunately I don't have an oscilloscope, but I think I have a PSU I can swap out somewhere just to check.

 

[loganphyve]

FYI, I have RMA'd the board and re-purchased an X9SCM-IIF-O in its place. Memtest86 has been using the same batch of RAM now for over a week with 30 consecutive passes. I think I'm good to move on to the next steps.

Thanks again, contributors. The help was greatly appreciated.