sdhci_pci0 Controller Timeout on boot

[Fitz]

Dear Community

During boot (from a SDHC card )the console informs me on a controller timeout on sdhci_pci0-slot0. Is there a possibility to overcome this ? It tries 3 times then continues.

Nov 28 13:26:19 truenas sdhci_pci0-slot0: Controller timeout
Nov 28 13:26:19 truenas sdhci_pci0-slot0: ============== REGISTER DUMP =========
Nov 28 13:26:19 truenas sdhci_pci0-slot0: Sys addr: 0x00000000 | Version: 0x000
Nov 28 13:26:19 truenas sdhci_pci0-slot0: Blk size: 0x00000000 | Blk cnt: 0x000
Nov 28 13:26:19 truenas sdhci_pci0-slot0: Argument: 0x00000000 | Trn mode: 0x000
Nov 28 13:26:19 truenas sdhci_pci0-slot0: Present: 0x21ff0001 | Host ctl: 0x000
Nov 28 13:26:19 truenas sdhci_pci0-slot0: Power: 0x0000000f | Blk gap: 0x000
Nov 28 13:26:19 truenas sdhci_pci0-slot0: Wake-up: 0x00000000 | Clock: 0x000
Nov 28 13:26:19 truenas sdhci_pci0-slot0: Timeout: 0x00000000 | Int stat: 0x000

Thanks a lot for guidance
brdgs Fitz

 

[Ericloewe]

What machine is this? Is the SD card a serious necessity here? I'm not super optimistic about your chances, but let's hear more about your setup and figure out what to do.

 

[Fitz]

Hi Eric

Its a little AsRock DeskMini 310 with a i5-9500, 32 GB DDR4, 512 GB Samsung 970, 2x2 TB Toshiba PCL200. It has a slot for SD cards on the mainboard, which I use for booting TrueNAS. I have this config since 2.5 years and it does a reliable job since then (minus the error message). I have re-installed TrueNAS 13 from scratch after CPU was changed (i3-8100 to the i5-9500) and there is a plan to replace the Toshiba HDD with 2x4GB Crucial MX500 SSD as space on the 2TB runs out.

The DeskMini is used mainly as file server, where I have data which I do not need daily when working with my laptop. Data resides on the Toshs and ZFS Mirror, the 970 holds the jails.

The intial FreeNAS installation already throw that error, but I only came back this week on the topic.

Booting and operations works supersmooth rocksolid since ever, it is just this naughty timeout, where I do not know where it comes from. No one was harmed ever since, so just curious if this timeout can be eliminated.

Thanks & brgds Fitz

 

[Ericloewe]

there is a plan to replace the Toshiba HDD with 2x4GB Crucial MX500 SSD

Sidenote: You probably don't want to that. The MX500 has a longstanding firmware bug that causes absurd write amplification (>10 times) when using TRIM. Early (at least...) Samsung 870 Evos also have either a firmware bug or a hardware defect that causes them to wear out way too early (after a handful of drive writes). So I guess that leaves WD Blue...

It has a slot for SD cards on the mainboard, which I use for booting TrueNAS.

Could you use the slot dedicated for the Wi-Fi card? You might need something smaller in the M.2 2230 orm factor, but it should work, assuming it's wired for PCIe and USB, as is typical. At that point, you might be able to disable the card reader in the system firmware and get rid of the annyoing message. Even if you can't, there's a decent chance the message will disappear if the reader is empty.

 

[Fitz]

Sidenote: You probably don't want to that. The MX500 has a longstanding firmware bug that causes absurd write amplification (>10 times) when using TRIM. Early (at least...) Samsung 870 Evos also have either a firmware bug or a hardware defect that causes them to wear out way too early (after a handful of drive writes). So I guess that leaves WD Blue...

Actually I heard of the Write Amplification bug on the MX500s, and I really hope that this issue is obsolete for the 2021's 4TB versions. The 4TB version was released Q4.2021 and comes with a revised controller SM2259H and has a new stock-firmware M3CR045. After your comment I digged the internet for this topic on the 4TB versions, but did not found anything, most messages were 2020-early 2021 and for the 250-1000 GB versions only. If you have anything found on the 4TB, please kindly share.

For the time being I plan to re-load all data to the drives (the MX500s are already sitting in the bay) and will check what SMART results deliver back. Fingers crossed.

Could you use the slot dedicated for the Wi-Fi card? You might need something smaller in the M.2 2230 orm factor, but it should work, assuming it's wired for PCIe and USB, as is typical. At that point, you might be able to disable the card reader in the system firmware and get rid of the annyoing message. Even if you can't, there's a decent chance the message will disappear if the reader is empty.

The M2.2230 could carry a WiFi module (and unfortunately only that, not suitable for storage), however I do not have a module here and therefore it is disabled i the BIOS. I would indeed prefer to disable the SD card slot (but then I would need to install TrueNAS on the M.2 SSD drive, limiting myself to not having Jails. I still understand, that you cannot have the system/swap and jails on one physical drive - and I did not want to attached USB-Sticks to the case [like UnRAID wants to have it]).

Thanks for the suggestions so far. Brgds Fitz

 

[Ericloewe]

Actually I heard of the Write Amplification bug on the MX500s, and I really hope that this issue is obsolete for the 2021's 4TB versions. The 4TB version was released Q4.2021 and comes with a revised controller SM2259H and has a new stock-firmware M3CR045. After your comment I digged the internet for this topic on the 4TB versions, but did not found anything, most messages were 2020-early 2021 and for the 250-1000 GB versions only. If you have anything found on the 4TB, please kindly share.

Interesting, I'll have to look into it. I have some 1TB units that are definitely earlier than Q4 2021 and exhibit the issue.

(and unfortunately only that, not suitable for storage)

Why? It's wired for PCIe, so it should work.

 

[Fitz]

Interesting, I'll have to look into it. I have some 1TB units that are definitely earlier than Q4 2021 and exhibit the issue.

I found here an English review, which looks pretty detailed: Link

Why? It's wired for PCIe, so it should work.

I think it needs a M.2 E-Key drive and I am not even sure if such exists. At least nothing what I am able to find. Maybe adapter...

 

[Ericloewe]

Ah crap, I'd forgotten just how silly things get with M.2 outside the typical SSD sockets.

and I did not want to attached USB-Sticks to the case

What about internal USB headers? The specs aren't very clear, but there should be some. With an adapter, plus a USB to NVMe bridge, the results might not be half bad if you can get it to work mechanically.

 

[Fitz]

I'll go for the USB adapter, ordered a small pin connector to USB port and will plug in a USB Flash drive. Should do the job, and then the SD card could potentially be removed and slot deactivated in BIOS. I will report back, also what the drive data will tell after copying all the archives back to the Crucials. Thanks!

 

[Fitz]

As a final update: I was not able to get the internal USB connectors to work as expected. Any thumbdrive took forever to boot (USB 2.0 only) and then it throwed a lot of errors.

So the solution is now to utilize the internal NVMe with a "cheap" SSD, going away from SD Card I can remember, that earlier versions of FreeNAS were explicit recommended to be startet from SD Card or Stick, that's why I did it the way until now. However, it looks like that it is now possible to put a pool on the same drive(s) where the data storage is placed so no longer need for anything else then NVMe SSD and the 2 data drives I use. Only thing open now is to see, how long the Crucial MX500 will last.

I turned also on the SED functionality of both drives and up to now it seems working nice with build in encryption. I only observed one issue when turning on "AutoTrim" on the Pool / Pool Option / Auto Trim. It gives back error messages, that drive is not ready and it only shows up, then the Auto Trim is active. It does not report when Auto Trim is inactive.

Dec 25 12:13:38 OSIR-NAS (ada0:ahcich0:0:0:0): DSM TRIM. ACB: 06 01 00 00 00 40 00 00 00 00 01 00
Dec 25 12:13:38 OSIR-NAS (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Dec 25 12:13:38 OSIR-NAS (ada0:ahcich0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
Dec 25 12:13:38 OSIR-NAS (ada0:ahcich0:0:0:0): RES: 51 04 00 00 00 40 00 00 00 01 00

Not sure, if this is connected to the SED encryption or a general issue with SSD (here: Crucial SSC 4TB - CT4000MX500SSD1).

 

[Ericloewe]

However, it looks like that it is now possible to put a pool on the same drive(s) where the data storage is placed

Not officially, so it's one of those thing's you'd have to manage manually and hope doesn't get broken during an update (not super likely, but not unheard of).

I turned also on the SED functionality of both drives and up to now it seems working nice with build in encryption. I only observed one issue when turning on "AutoTrim" on the Pool / Pool Option / Auto Trim. It gives back error messages, that drive is not ready and it only shows up, then the Auto Trim is active. It does not report when Auto Trim is inactive.

Dec 25 12:13:38 OSIR-NAS (ada0:ahcich0:0:0:0): DSM TRIM. ACB: 06 01 00 00 00 40 00 00 00 00 01 00
Dec 25 12:13:38 OSIR-NAS (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Dec 25 12:13:38 OSIR-NAS (ada0:ahcich0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
Dec 25 12:13:38 OSIR-NAS (ada0:ahcich0:0:0:0): RES: 51 04 00 00 00 40 00 00 00 01 00

Not sure, if this is connected to the SED encryption or a general issue with SSD (here: Crucial SSC 4TB - CT4000MX500SSD1).

"TRIM issues" sounds a lot like the longstanding MX500 bug...

 

[Fitz]

I am only aware on MX500 issues on the 4 TB with locking up in RAIDs, for this Crucial delivered a firmware fix recently. As I run a "Mirrored Data Dump" only, I wonder if this really is related to it. Please also note, this is the newer 4TB device with updated controller hardware / firmware. Do you have more details on the MX500 bug you are referring to and how would it be possible to nail it down, that this is the bug I encounter here ?

This is that I consider the known "longstanding MX500 bug": "...the first iterations of the mx500 have a firmware/controller flaw that leads to excessive internal operations performed eroding the health of the drive" [Link]. This should be somehow fixed now.

BTW, both drives a brandnew and all SMART results, plus the own Crucial tool reports that the drive is in best condition.

 

[Ericloewe]

This should be somehow fixed now.

I agree that it should be fixed, but I've also seen enough firmware horrors to always suspect otherwise. IIRC, the actual bug was triggered by TRIM operations and could be worked around by disabling TRIM on MX500s.

 

[ChrisRJ]

I am only aware on MX500 issues on the 4 TB with locking up in RAIDs, for this Crucial delivered a firmware fix recently. As I run a "Mirrored Data Dump" only, I wonder if this really is related to it.

No idea whether this has any practical relevance. But a mirror is just a particular RAID-level. So it would be interesting to know the exact aspects that "RAID" means in terms of lock-up.

 

[Fitz]

Here is some very basic description to that firmware update.

For the above TRIM issue, it looks like that the SED Encryption is involved in this issue. I have removed the hardware encryption and setup the NAS just working without the SED active and recreated some pools, putting data to it, deleting again etc. When turning on Auto Trim on the Pool, the error message are away. When re-actviating SED, they come back again. Seems to be an issue with the OS. Potentially I will raise in Jira a ticket.

Assume there is not so many people here, running SSD with SED active ? (Our business case here is that all data [pics of the family, contracts, financial records and family document archiv] is kept on the NAS, accessible from everywhere and for protection of theft / burglary I like to keep this data encrypted. That's also the reason why I am fine with 2x4 TB not needing plugins or VM.)