FreeNAS mini (C2750D4I) repair due watchdogd nand wearout

[pernils]

If you have a FreeNAS mini then you should check and consider to turn the watchdog function off in bios.

The mini's motherboard is ASRock C2750D4I and on bios version v2.90 it's by default turned off.
As I have encounter 2 failured motherboard it must have been set to on in bios before v2.90.

A motherboard with bios version 1.80 and BMC 00.14.00 only lasted 3 weeks before it stopped to boot. I only use freenas v9.3 due to the cifs performance problem.

http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I#Download.

From what I understand the official rumor is that this wearout only occurs on FreeNAS 9.10 and above.

https://bugs.freenas.org/issues/16190

But my experience is that you will encounter this problem if you have not disabled the watchdog in bios even on v9.3.

Last week I received a "repaired" board, and after a closer inspection the only part that is replaced is the small IC near the SATA connectors.

Its socket is the black one just bellow the throwhole in the upper left corner. -> http://www.asrockrack.com/Server/C2750D4I-1(L).jpg

The socket consist off 2 flaps that could unfold to remove the SOIC16.

The IC is a :

Windbond W25Q128FVFG.

Datasheet -> https://www.pjrc.com/teensy/W25Q128FV.pdf

It can be bought from ebay or aliexpress.

https://www.aliexpress.com/wholesale?catId=0&initiative_id=SB_20170123224711&SearchText=25q128fvfg

Will order and try out by my self. Is there someone out there who have done this process and is willing to share your experience ?

Another thread with more or less the same subject -> https://forums.freenas.org/index.php?threads/how-long-your-asrock-c2550d4i-or-c2750d4i-lasted.45445/

 

[James Morrison]

I'm also looking to repair my C2750D4I which suffered the watchdog flash burnout.

I don't think the problem is the W25Q128FVFG IC as I think that's where the firmware is stored. The storage IC is the 8 pin Winbond W25X40CLSIG which is soldered and near the DDR3_B1 slot. See
https://forums.freenas.org/index.ph...i-or-c2750d4i-lasted.45445/page-2#post-322635

According to Grzegorz Krzystek in his bug report he managed to fix his board but he's a bit vague as to the process. I tried to get him to clarify but he didn't go into any detail unfortunately:

it's easly replaceable, in case of AsrockRack C2750DI , there are tools on aspeed web site to flash clean chip.

https://www.aspeedtech.com/support.php

SOC Flash Utility

i used UEFI version (uflash) from this zip archive

keep in mind that after flash of clean chip , ipmi mac addresses will change, you can recover them by using Mac Recovery in bios.

https://bugs.freenas.org/issues/16190#note-40

I would really like to fix this board if possible.

 

[pernils]

Did a closer inspection of this repaired board and the only thing that seems to been tempered with is the SOIC16 IC in it's socket. The sticker was gone.

https://smppartsab-my.sharepoint.co...60c3422737fd9&authkey=ARE9CqAU9TfRQWiWMqOuOb8

The other populated dip socket next to the DIM seem's not to be untouched due it still have it's sticker on it. If the other windbond smd seems to not be resolder due to the perfection of the IC alignment and no signs of flux.

In this picture I have removed the sticker on the dip8 socket.

https://smppartsab-my.sharepoint.co...2e0da8fd6db28&authkey=AWWcggqSMqvqoWWP8WTAYfE

For now I have only ordered 5 pcs of the SOIC16. And at the moment I can't verify if this repaired board actually work due I have no case or ram for it.

So to summarize the IC's

Easily changeable socket SOIC16
Winbond W25Q128FVFG. Datasheet -> https://www.pjrc.com/teensy/W25Q128FV.pdf

Easily changeable socket DIP8
Winbond 25Q64FVA1Q. Datasheet -> https://www.winbond.com/resource-files/w25q64fw_revd_032513.pdf

Pita changeable 8 pin SMD
Winbond 25X40CLSIG. No datasheet found.

-------------------------------------------

Gathering other threads on the same topic

https://forums.freenas.org/index.php?threads/dead-asrock-c2750-rma-or-cut-losses.46162/

https://forums.freenas.org/index.php?threads/failed-asrock-c2750.36893/

https://forums.freenas.org/index.php?threads/how-long-your-asrock-c2550d4i-or-c2750d4i-lasted.45445/

https://forums.freenas.org/index.ph...2x50-d4i-boards-by-disabling-the-watchdog.16/ <- read this !!

 

[James Morrison]

The sticker just has the BMC firmware version on it. Perhaps it was removed because the firmware was updated, not because the IC was replaced.

Here's a datasheet for the Winbond 25X40CLSIG:
https://www.winbond.com/resource-files/w25x40cl_e01.pdf

I did come across most of those links and read about how to avoid the problem by disabling the watchdog in the BIOS. I've updated my BIOS to v2.30 and BMC firmware to v00.27.00 (http://www.asrockrack.com/general/productdetail.asp?Model=C2550D4I#Download). The latest BIOS update appears to disable the watchdog timer by default.

I've actually desoldered the (pita) Winbond 8 pin SMD 25X40CLSIG on my board and replaced it with an AT25SF041 which appears to be a like for like component. It didn't make any difference though as my board still has the same problems and I'm unable to boot anything to try the SOC Flash Utility.

I'm still messing around with it so I'll keep you updated on my progress.

 

[okaenrique]

i have same problem , how ican repair my ?

 

[pernils]

Finally I have received my Aliexpress package of soic16.

After spending serious amount of time fiddling with this ....

Every chip that is socketed must be programmed externally before inserting. I'm using a chipprog-48

A bit about the background.
I have 2 motherboards. One witch is out of warranty from iXsystem and one wich have been RMA here in sweden by Dustin.

I will here refer them as iXsystem and Dustin
iXsystem = broken
Dustin = RMA and working. The only visible thing done was the removed BMC firmware sticker.

I moved the BMC chip from broken iXsystem to Dustin.

Didn't boot
Used my chipprog and flashed a brand new soic16 with firmware 00.30 http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I#Download

Did boot but no mac address on nics.

Found this page for asrock mac address writer tool. http://www.asrock.com/support/download/mactool.asp
It didn't matter how I tried with mac c (mac address) or mac c (mac address) a+2 or mac c (mac address) a+2 /nofw etc etc ..
Tried both 1.61B and 2.07A.

Finally I gave up and read the working chip with my chipprog and flashed that image to a new chip.
The working chip had BCM v00.19 so I updated the BMC firmware via IPMI as I now had a mac address.

Seems that Asrocks mac tool don't work on C2750D4I.

Moved the 8 DIP bios chip from iXsystem to Dustin.

Wouldn't boot again. Used the chipporg and flashed the chip with 2.90 from http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I#Download
And once more i booted again.

I moved back all the working chip from Dustin board to iXsystem board and flipped the power. No boot just black screen.
The iXsystem board must be broken in another way.

Did some testing to see if there can be clue what is missing on a broken boards.
My findings is following..

Broken BMC chip will only make on led to be lit. Red arrow show the other led.

Nothing will show up on the monitor.

Broken bios chip (8 DIP). Both led is lit. The upper left one will start to blink after 30 sek. This is the same behavior as with a fully working system. This from what I have read this is the heartbeat LED for the IPMI/BMC controller. It's supposed to flash about once a second to indicate that it's booted and operational.

I had been testing so much but I think I ended up in PXE screen. I couldn't reproduce this later on so I can't take this for granted.

All the test is done with no boot media. A normal system will boot up into bios setup if no boot media is attached.


Summary:

I still have one board that's broken.

All chip must be flashed by a extern programmer.

Asrock mac writer tool may not work with C2750D4I

I have 5pcs windbond soic16 flashed and verified for disposal.

 

[pernils]

Strange ..

The still broken board iXsystem I went a head and cleared the cmos.
Remove the battery and short the solder pads with a paperclip. The pads is next to the battery.

Flipped the power and no boot screen but the board started to spin the disks. I have only the power cord to the disk connected and then I can see if the mother board is talking to the power supply at all.
Hmm I thought and made a nmap. And i found a ip with open port 80. Used the web browser and reach the IPMI login page. Logged in and updated the bmc firmware.
Rebooted and it's dead again ..bummer

 

[pernils]

After another cmos reset and flash of 8 DIP bios chip the IPMI address was found by nmap. No boot text on the monitor and no via the java verison.

I don't know .. This nand wearout seem not to be in any of the socketed ics. Must be the pita smd ic. My RMA bord to dustin seems to not be affected by this wereout. The board was only in production for 3 weeks before it stopped to boot, could this nand wereout come so fast?. Wouldn't be surprised it they just cleared the cmos after all.

 

[Arwen]

@pernils
Part of your problem may be the 8 PIN socketed chip is the wrong one to change.
I thought the soldered Winbond 25X40CLSIG 8 PIN gull wing chip was the one
that needed to be replaced.

 

[Stux]

Also, there are two faults with these boards, the nand wear-out and the avoton faulty clock generator

 

[Stux]

Ix systems extended the warranty to 3 years. Is your board still out of warranty?

https://support.ixsystems.com/index.php?/Knowledgebase/Article/View/289

iXsystems is proud to stand behind its products. We’re extending the warranty on all second generation FreeNAS Mini motherboards shipped before February 2017 to a total of three (3) years.