How to fix Asrock c2750d4i with C2000 bug

[Constantin]

I wonder whether the 25MHz signal could be externally generated on a daughter board that plugs into the TPMI header, then cut the pcb trace next to the cpu, if it is accessible? Unfortunately, chances are this is a 6+ layer PCB. But hope springs eternal.

 

[Marcy20]

So my board came back from the RMA. there was a new one despite the fact that it was produced in 2014. However, now I have a problem with Truenas 12 U4. CPU stats doesn't show up correctly

 

[Samuel Tai]

CPU stats doesn't show up correctly

This is a known bug in 12.0-U4. See https://www.truenas.com/community/threads/cpu-stats-not-working-in-12-0-u4.93525/ for a user-developed work-around.

 

[dreamerns]

Hi there,
I have the same problem. I returned my board 4x for RMA and now the fifth might be saying goodbye to this unreliable garbage. But just before I throw it into trashcan I decided to check on the forum if there is some revive solution. I don't know which type of failure is this time. I can log in to IPMI, but when I start there is no post, the monitor is blank, nothing happens except that BMC led blinking green.
Thanks

 

[Constantin]

I’d try the proposed solutions that have worked for others. You literally have nothing to lose.

 

[bfarnam]

So I have a spare C2750D4I which apparently had a MFG Board Level Repair done. It is actually quite different from what is described here in this post. I have confirmed it is a B0 stepping CPU by checking the S number.

I tried to RMA it and they denied it as out of warranty with no repair option.

I am going to put my OScope on it later and see what I have. I will post more later on the results and the specifics of the MFG Board Repairs.

 

[bfarnam]

I have updated my thread over on STH... so I am not duplicating my efforts you can access it here:
https://forums.servethehome.com/index.php?threads/bug-in-intel-atom-c2000-series-processors.13173/

In essence this board appears to have had a "factory platform" fix which entailed "pulling up" ALL the timers on the CPU. I can confirm the LPC (Low Pin Count) Clock 0 and 1. The pull up was done with 120 ohm resistors to 3.3v which measures out on my mobo at 3.46v.

LPC Clock 0 and LPC Clock 1 are labeled as LAD 0 and LAD 1 at the TPMS header. Also LAD 3, LAD 4, and the PCI_CLK were pulled up at the TPMS. There were one, possibly 2, resistors added UNDER the CPU on the backside as well.

 

[cfcaballero]

Thanks so much for this post! (I can't seem to @mention you, though, OP).

I have a similar situation with my 2014 vintage C2750D4I having died in 2020 when I couldn't reach it. Now that I have my mitts on it, I put the oscilloscope probe on that PCICLK pin and I get what looks like open circuit. I can login to the BMC no problem, V0.35 like many others.

What should that pin look like before the resistor fix? Should it be a 33MHz square wave, just with the p2p voltage too low, or something else?

Thanks again, and once more in advance to you or anyone else that can answer and help me decide whether I should try and salvage this mobo.

 

[Ericloewe]

What should that pin look like before the resistor fix? Should it be a 33MHz square wave, just with the p2p voltage too low, or something else?

A clean signal would be a square wave. The frequency is less than fully clear: LPC nominally runs at 33 MHz, but Intel seems to have switched at least some chipsets to running it at 25 MHz.
Edit: The C2000 datasheets apparently do state 25 MHz.
(As nearly everything has moved off the LPC by now, bandwidth requirements are close to zero. I suspect that 25 MHz is marginally easier to generate from the 100 MHz base clock.). My C2758 measures at 25 MHz.

 

[Ericloewe]

Fake second edit: It turns out that C2000 can be configured for either 25 MHz (default) or 33 MHz. Or maybe not? This whole thing is confusing the hell out of me.

Edit to the fake edit: I get the feeling that it's actually the FLEX_CLK_SE0 pin that's being used for LPC clocks. That one supports 33 MHz, the LPC Clocks proper don't. Has anyone actually measured the frequency of the LPC clock on the ASRock C2x50D4I boards?

 

[Al Fuller]

Fake second edit: It turns out that C2000 can be configured for either 25 MHz (default) or 33 MHz. Or maybe not? This whole thing is confusing the hell out of me.

Edit to the fake edit: I get the feeling that it's actually the FLEX_CLK_SE0 pin that's being used for LPC clocks. That one supports 33 MHz, the LPC Clocks proper don't. Has anyone actually measured the frequency of the LPC clock on the ASRock C2x50D4I boards?

All: I have a FreeNAS mini that has the ASRock C2750D4I board, and it has died... I have tried to follow this thread, and even soldered together the resistors as indicated above in an effort to get the server back up - to no avail. As I have been trying to educate myself on what motherboards are available, this has still bothered me: I understand the issue to be that we need 3.3v on the PCICLK pin on the TPM header, but it is degraded.

The fix suggested is to take +3v from another pin on the TPM header, apply two resisters to make a voltage divider, and connect to the PCICLK pin.

It seems to me that we are trying to fabricate a power supply for this use case, and there is a perfectly good power supply already present that [conveniently] supplies 3.3v on one of the cables. Am I missing something, or can't we just pick off that 3.3v and use it? Thanks in advnce for your thoughts.

 

[Ericloewe]

It seems to me that we are trying to fabricate a power supply for this use case

More subtle than that: A hard pull-up. Shorting to +3.3V would, at best, result in the bus staying high all the time, which isn't useful. At worst, it would burn out the CPU entirely.

General consensus is that, for whatever manufacturing reason, the high side of the totem pole output driver on the CPU dies, while the low side keeps working. Adding a hard pull-up allows the low-side driver to (with some effort...) pull down the bus line to the low logic level.

 

[Al Fuller]

More subtle than that: A hard pull-up. Shorting to +3.3V would, at best, result in the bus staying high all the time, which isn't useful. At worst, it would burn out the CPU entirely.

General consensus is that, for whatever manufacturing reason, the high side of the totem pole output driver on the CPU dies, while the low side keeps working. Adding a hard pull-up allows the low-side driver to (with some effort...) pull down the bus line to the low logic level.

OK, thanks for the reply. I guess I understand better now why anyone / everyone didn't just go that route...

 

[eyocum]

I'm not neat with a soldering iron. If I breadboard up an edge connecter with the resistors, that'll work, right? as long as they complete the circit to the pins?

 

[Ericloewe]

To some extent. Keeping the wires as short as possible is important to keep inductance in check, as it is a 25 MHz signal.

 

[ses1984]

I had a power outage recently and had to shut down my unit. It failed to boot after that and it seemed to have basically the same symptoms described here, including the OEM timestamped logs. I tried to solder 100ohm and 470ohm resistors onto the TPM header as described in the OP and my system still seems to behave the same way. I guess I'm pretty stuck without an oscilloscope? Thanks

 

[Ericloewe]

Well, with a multimeter you can check if the clock line has activity. The other thread linked around here somewhere has a specific figure to look out for.

Beyond that, you can sort of try different things, but blind troubleshooting is not easy. Hell, even with an oscilloscope this can be quite a chore...

 

[nogates]

Hello!

First of all, I want to say thank you @online2 for starting this thread and sharing a possible solution!! Thank you, so, so much!

As many others, one day I turned off my truenas with the asrock c2750d4i, and then, it didn't want to start up again. Which led me to this thread :)

The original solution explained here worked like a charm for me. I connected the resistors and wires as described, and the motherboard was alive again. Though, it didn't last long. After 2 months, I had to power off the computer again, and it didn't want to start again.

The symptoms were exactly the same after removing the connection to the PCICLK pin. However, if I reconnected everything, the motherboard didn't even turn on. I could hear the fan for like a few milliseconds, and the power light blinking but it automatically powered off. After further inspection with the multi-meter, I discovered that the 3V pin wasn't providing any voltage anymore. Not sure if it has been burned in the process, but yeah, it looks dead :(

So, then I decided to try a different thing. Got myself a cheap voltage regulator (this to be precise). I connected the input to one of the spare PCI-E connector (like the one in this image, bottom right, 4 pins) that I am not using, which produces a voltage of 12V. Then, I used the voltage regulator to lower it to 3.3V. It still didn't want to start up. HOWEVER, it DID start when I increase the voltage up to 3.33V, which I guess might make sense since I've used longer wires during my testing that I had before.

One thing though, it doesn't start if I am supplying the voltage from the start. I have to first disconnect everything, then, start the compute, wait like 10 seconds, and then, activate the 3.33V power supply to the PCICLK pin.

So at the end, I've changed the BIOS setting so the computer doesn't start up automatically in the event of a power loss, and I've added a switch between the voltage regulator and the PCICLK pin, so I can easily start up the computer, wait for a few seconds, and then, switch on the power to the PCICLK.

This setup is currently working and the server starts up every single time

I am not sure if this will end burning any other part of the motherboard. BIOS and IPMI interfaces says all sensors are green, and CPU / Memory usage seems normal. Also, the computer is not overheating or doing anything weird. The PCICLK is consuming around 120mA constantly (+/- 5). Of course, if I disconnect the PCICLK power, the computer freezes completely

I will start looking for a different motherboard. I don't think I can trust this one anymore, but just wanted to write here my findings in case it's useful for anyone. Though remember, try this at your own risk :)

Thank you very much!

 

[sleestak666]

Just logged in to report that my board died about 2 months ago. Did my first solder job as outlined by the OP and my system now boots again.

Thanks!

 

[Al Fuller]

More subtle than that: A hard pull-up. Shorting to +3.3V would, at best, result in the bus staying high all the time, which isn't useful. At worst, it would burn out the CPU entirely.

General consensus is that, for whatever manufacturing reason, the high side of the totem pole output driver on the CPU dies, while the low side keeps working. Adding a hard pull-up allows the low-side driver to (with some effort...) pull down the bus line to the low logic level.

OK, I'm going to give reviving this server another go... Having put in resistors as listed in the original post, it still doesn't boot up. I had used pin headers instead of soldering directly to the pins, so my next effort will b trying that - which leads me to my confusion:

Can someone explain to me how it works to pick off 3v on the TPMS header, run it thru a voltage divider, and come out with the required 3.3v - which is greater than the input? I see online voltage divider calculators, and this one gives me 2.72 Volts - which makes intuitive sense to me. What am I missing? How does one start with 3v, divide it, and come out with 3.3v?