What does this email mean? Subject: SMART error (FailedReadSmartErrorLog) detected on host: freenas

[Nindustries]

Ah Mr Jock, thanks for replying even if you bring potentially bad news heh. I'll follow the cables and see where they are plugged in and try to see if theyall are all on one controller. If so that will give me around thinking it's a contcomit'sroller.

Do you think those errors warrant an rma?

Try swapping cables. If not, see if the problems come from a Marvell controller. If so, contact AsRock support at william@asrockamerica.com. (I've been there, but didn't want to go through the hassle and swapped it for something else.) They will probably provide you with a tool to update your Marvell controller firmware.

Good luck and keep us posted!

 

[cyberjock]

Do you think those errors warrant an rma?

It's very possible. AsRock will know more. As for a firmware update, they played that game with a few other people. It didn't help them. :/

If they offer a firmware update, feel free to give it a shot. There's nothing to lose from trying. But, I wouldn't expect miracles.

 

[phonoflux]

Oh my god my phone completely butchered my last reply. Was supposed to be something along the lines of "If I can identify they are all plugged into the same controller that will give me comfort about taking that avenue"

I'll have a crack at it tonight when I get home.

 

[phonoflux]

Status report time. After tracing the cables the 4 drives were plugged into the 4 ports on the motherboard labeled Sata III_M0 through to M3 (pictured bottom left here http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I)

From further investigation, the intel chipset runs the top 2 white (sata3) and top 4 blue (sata2), another marvel runs the bottom right 2 of the 6, and then the problem marvel runs the bottom left 4 of 6 (as described)

After moving the drives to the 4 blue ports running off the intel controller and resuming copying data all night I didn't get a single error whereas I was getting 2-3 an hour previously.

I emailed William from the address you provided anyhow with the detail I have found (would be about 18 hours ago now) and haven't heard back but will keep you all posted. I'll hook some spare drives up to those ports when I get more time to do further tests to see if they continue to throw those smart errors.

TL;DR seems whatever marvel controller runs the ports labelled SataIII_M0, SataIII_M1, SataIII_M2 and SataIII_M3 need to be avoided on my board.

 

[Matdif]

I am having trouble on this same motherboard and the same Marvel sat slots. The same errors in fact. Currently running badblocks to make sure its those ports but that wont complete for 3-4 days. I would like to know though if you were able to solve it by updating the firmware?

 

[phonoflux]

Ah, your reply reminds me I need to update my story :)

In short, had a reply from the ASRock team and their two suggestions were to disable those ports as being bootable in the bios and to update to a newer version of the bios which they attached to an email (v2.53). Those ports were already set to not be bootable and I wasn't too keen to use an unreleased bios so I said I would go down the path of RMA.

I've now got an RMA number but it sounds like for me to RMA they want me to send the board back to China and leave me without anything for the duration. That hasn't been confirmed yet but I suspected this might be the case and I have started asking (and will be insisting if that be the case) on a forward replacement before I send mine back.

I wish I could find the thread or the place I read that others were having issues with certain ports on this board too to see if it's the same issue, as they didn't seem to want to do any additional troubleshooting short of what I mentioned. If it's a basic incompatibility with FreeNAS/BSD then that's fine but if it's the start of potentially more issues then well, not so good.

Matdif, can you provide a bit more clarity on what you're getting? Is it exactly the same as me and only from those ports? Those 3 interesting SMART errors around being unable to read the smart errors and only during heavy read/write sessions? A scrub would cause them to crop up too which you can kick off with zpool scrub poolname and then when ready stop it again with zpool scrub -s poolname (and see how fast it's going with zpool status).

I have since received my raid card and that has been installed so i'm not running anything off the onboard ports at present. I've got the drives from my previous FreeNAS install that I might hook up in the weekend onto those ports just for kicks and if they say I am unable to get a forward replacement try and nut out with them exactly what's going on and why.

 

[Matdif]

Well I am still getting the errors. Thought I was free but I was wrong.

I wasn't getting the alerts on my phone because I made a folder for freenas in my inbox and for some reason my phone is now ignoring these specific emails.

These were from a week and a half ago or so on my first badblocks test.

Device: /dev/ada4, Read SMART Self-Test Log Failed
Device: /dev/ada1, Read SMART Error Log Failed
Device: /dev/ada2, Read SMART Self-Test Log Failed
Device: /dev/ada4, Read SMART Error Log Failed
Device: /dev/ada3, Read SMART Self-Test Log Failed
Device: /dev/ada3, Read SMART Error Log Failed
Device: /dev/ada4, unable to open device
Device: /dev/ada3, failed to read SMART Attribute Data
Device: /dev/ada1, Read SMART Self-Test Log Failed
Device: /dev/ada3, Read SMART Self-Test Log Failed
Device: /dev/ada1, failed to read SMART Attribute Data
Device: /dev/ada1, failed to read SMART Attribute Data
Device: /dev/ada1, failed to read SMART Attribute Data
Device: /dev/ada1, unable to open device
Device: /dev/ada3, ATA error count increased from 0 to 1

I started again last night and this is what I have so far. In between these two I redid cable management and switched cables and sata slots.

Device: /dev/ada3, Read SMART Error Log Failed
Device: /dev/ada2, failed to read SMART Attribute Data
Device: /dev/ada2, Read SMART Self-Test Log Failed
Device: /dev/ada3, Read SMART Self-Test Log Failed
Device: /dev/ada3, failed to read SMART Attribute Data
Device: /dev/ada3, failed to read SMART Attribute Data
Device: /dev/ada2, Read SMART Self-Test Log Failed
Device: /dev/ada3, Read SMART Self-Test Log Failed
Device: /dev/ada2, Read SMART Error Log Failed
Device: /dev/ada2, failed to read SMART Attribute Data
Device: /dev/ada3, failed to read SMART Attribute Data
Device: /dev/ada2, failed to read SMART Attribute Data
Device: /dev/ada3, Read SMART Self-Test Log Failed
Device: /dev/ada3, Read SMART Self-Test Log Failed
Device: /dev/ada2, failed to read SMART Attribute Data
Device: /dev/ada2, Read SMART Self-Test Log Failed

Looks like its going to be the maxwell card but I havent traced the sata slots yet.

 

[phonoflux]

Yeah trace the slots and also take note of the drives serial numbers to see if they were the same drives linked to the ada1-4 before and after the cable switch. How many drives have you got hooked up?

I'm not sure what badblocks is but if you setup your root user with an email, setup smtp and then also enable smart alerts you should be told each time one of these goes off.

 

[Matdif]

badblocks just writes to the drive and reads it. The drive itself reports no errors its only that it is having trouble reaching them sometimes. Its not the same drives and it is a different number. It was 3/9 before and now its just 2. I am about to stop it and look at which sata ports it is.

Edit: Whelp those 2 were indeed on the Marvell. However there were other drives in the Marvell as well that did not throw errors. I did notice however that they were on sata cables that were a bit thick and ended up pushing each other a bit. I redid the connections with thin cables and I now seem to be down to one drive giving errors.

I am going to wait another 30 mins stop the test reseat this one cable in another sata slot and then restart badblocks to run all night. Hopefully I can kill the last of the errors.

If it was the Marvel for me I would think all the drives on it would error.

 

[Matdif]

A second drive error-ed and then nothing all night. It went seven hours without anything. In the morning I reset those 2 cables in different sata ports and then went again. Its now been 2 hours with 0 errors. I am going to let this badblocks run its full course this doesnt make sense to me. I really dont want a motherboard that cant be trusted and might throw errors at any time. Its easy to run badblocks and tests now I dont want to have to deal with this after I actually set up pools and put data on this. Currently waiting for a satadom anyways that is taking forever I am going to look for ways to stress this marvell more.

 

[phonoflux]

Another update on my case. They're sending me a forward replacement for my board which means for a brief time i'll have two that I can test with. At this point it's unlikely i'll ever actually use those ports at any time in this board's future but just to try and get to the bottom of the issue for my own piece of mine, theirs, and anyone else that comes across this it will hopefully be semi-documented

I have also provided ASRock a guide on how to setup a Z2 raid making sure they have at minimum an array with 4 drives plugged in over the marvel controller and they are going to do some in house tests too, along with me do some further tests with some spare drives tomorrow (hopefully).

 

[Matdif]

Another update on my case. They're sending me a forward replacement for my board which means for a brief time i'll have two that I can test with. At this point it's unlikely i'll ever actually use those ports at any time in this board's future but just to try and get to the bottom of the issue for my own piece of mine, theirs, and anyone else that comes across this it will hopefully be semi-documented

I have also provided ASRock a guide on how to setup a Z2 raid making sure they have at minimum an array with 4 drives plugged in over the marvel controller and they are going to do some in house tests too, along with me do some further tests with some spare drives tomorrow (hopefully).

Ive gone 51 hours now of badblocks without an error. I wonder if its the sata slots I left empty. Going to wait for badblocks to finish and then load up every Marvell sata slot to try again.

 

[Matdif]

Ya this Marvell controller starts causing problem when it is put to full use. When I was on half its sata ports there were no errors. I load it up and let it badblocks and now I am seeing 3 sata drives erroring every 10-15 mins. Going to try to update firmware if possible.

 

[Matdif]

Emailed Asrock meanwhile I went through the BIOS and noticed my Marvell slots had boot drive enabled and turned it off. Someone mentioned this was recommended so I am testing again well see if it helps.

Edit: Turning off boot enabled didnt help. Might be less errors now but still errors. Been like 118 hours without an email back from Asrock.

I wonder how the freenas mini gets it working? Unless they just dont use many Marvell slots with only 6 drives?

 

[Matdif]

My bios was up to date they sent me a RMA.

 

[phonoflux]

Heya Matdiff, have come back to update you on my progress and it sounds like you're at exactly the same stage as me. Have been a bit busy which is why i've been quiet :) I also have been through the RMA process and currently have two of these boards in my possession and was able to easily replicate the issue when loading the marvel ports on both boards.

Their current suggestion to me is to test it with Windows but I pointed out that even though that may work this issue is almost certainly NOT hardware related now but more a compatibility issue between FreeBSD (or FreeNAS) and that marvel controller. I further pointed out that they list FreeBSD as being supported for use with this motherboard.

In Cyberjock's own review of the FreeNAS mini he (possibly by luck, only having to choose 4 of the 12 ports) managed to avoid the 4 problem ports (from pictures) but from what I have read from posts he has made here he would have seen these issues if he used those ports https://cyberj0ck.wordpress.com/2014/05/05/my-review-of-the-freenas-mini-part-1/

I missed your last 3 updates as it seems I need to log back in for it to send me future updates to this thread so send me a PM if you need to get my attention :) Happy to try and work through some stuff with you and to try and provide ASRock more complete information. I've got about 7 people currently on my CC list whenever they send me emails regarding technical stuff and even more when it was being sorted for RMA so hopefully something may come of this.

Are you actually needing to use 9+ ports on the motherboard or did you choose those when setting up, saw the errors and have been worrying like me that there may be additional hardware errors so went down the RMA path?

 

[Matdif]

I need 11 ports in the end. 8 hard drive 2 ssd's and a satadom. Its too bad this motherboard really had it all. I might just get some extra sata slots out of the PCI slot to get it to work. I was using it for usb 3.0 but that was just for convenience to hook up some usb 3.0 the case had.

Either way I sent my motherboard back yesterday and will have to wait for them to send me another.

 

[phonoflux]

Ahh. I ended up getting a 8port pci-e card prior to figuring out that there were issues so thankfully i'm not in that bad of a position.

For future reference always ask if they can send you a forward replacement when it comes to RMA's, they did that for me so i've got two right now until we come to a conclusion around what to do.

 

[Cpuroast]

I read a comment on some other forum that mentions that the Marvell SE9230 which is driving 4 of the SATA3 6.0 Gb/s ports is to blame.
And that the Marvell SE9172 which is driving the other 2 SATA3 6.0 Gb/s ports works perfectly.

From the manual:
Intel SATA3: SATA_0 and SATA_1
Intel SATA2: SATA_2, SATA_3, SATA_4, SATA_5
Marvell SE9172 SATA3: SATA3_M0 and SATA3_M1
Marvell SE9230 SATA3: SATAIII_M0, SATAIII_M1, SATAIII_M2, SATAIII_M3

So it would be worth testing a pool comprised of SATA_0 through SATA_5 on the Intel and SATA3_M0 and SATA3_M1 on the Marvell SE9172.
If this is stable, this would mean that there is still 8 instead of 6 SATA ports out of 12 which function correctly with FreeNAS.

 

[phonoflux]

Can confirm that an array of 8 disks over the exact setup you mentioned (Intel + 2port marvel) works flawlessly. I had 4 of my 8 running over the marvel initially and when moved to the intel ports I finished copying the remainder 8tb of data without a single error :)

I think ASRock have either gone into holiday mode or given up as I haven't heard back from them since my last post. Oh well. Guess the recommendation is indeed use the 8 other ports + a pcie raid card if you need more.