Root mount waiting for: CAM

[Hell Bomb]

I am trying to build an ESXI vSPhere 7virtual machine with 2 HBA 71605 but I am getting stuck with the error "Root mount waiting for: CAM". If I remove the HBA cards I can install/boot fine, however as soon as I add them the system get stuck on this error. Not really sure what I can do to troubleshoot as I can't get it to boot.


I have tried:
Testing each HBA individually
Updating HBA firmware
FreeNAS Latest, and 11.3
EFI & BIOS installations

The only similar error I have found is this.

https://jira.ixsystems.com/browse/NAS-106460?attachmentViewMode=list

 

[jgreco]

By "2 HBA 71605", do you mean Adaptec 71605's? Did you pass them through to FreeNAS with VT-d?

Because if you did, please start over. Read the documentation. There's significant documentation that you should read, including here on the forums:

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

Confused about that LSI card? Join the crowd ...

A recent post prompted me to think, there's a lot of confusion regarding LSI HBA/RAID cards. If you're confused, it is probably understandable... it can be a bit complicated! ZFS users on FreeNAS should avoid using hardware RAID cards. ZFS and FreeNAS work best when the drive is managed...

www.truenas.com

Don't be fooled just because some of them are old. They are just as relevant as ever.

 

[Hell Bomb]

I'm confused, you specifically say "DO use PCI-Passthrough for a decent SATA controller or HBA." which I have done. The devices are getting picked up in FreeNAS and the drivers are being applied. Hardware has been researched and everything is updated to latest and greatest. Read 2 of those articles in the past and not sure they are relevant to my issue at all.

Hardware:

SuperMicro	X9DRi-LN4F+
Intel	Xeon E5-2650 v2

Adaptec

ASR 71605

Backplane

BPN-SAS-846A

 

[jgreco]

Yes, but the list of things I tell you to do is NOT a menu. The intention for things listed in my stickies is not that they be used as a menu; it is not "DO use PCI-Passthru" _OR_ "Use an LSI HBA Crossflashed to IT". This is not pick one-or-the-other. You need to do ALL the things. Some of the things I say to do are survivable if you do not do them, others are pretty much required. I post these things here to try to help newcomers understand what is likely to work out.

The sentences after "DO use PCI-Passthrough for a decent SATA controller or HBA" describe using onboard PCH or SCU controllers or an LSI crossflashed to IT mode as examples of what to use. Says nothing about a high end Adaptec RAID controller.

Please go and re-read points 2), 3), and 4) of

https://www.truenas.com/community/t...s-and-why-cant-i-use-a-raid-controller.81931/

this article in particular. It's absolutely relevant to your issue because the Adaptec simply will not work out for you for FreeNAS. The driver support either sucks or is nonexistent, I don't recall which.

If you are going to use this thing as a hypervisor, there is a viable option to use the 71605's for ESXi datastores. I prefer the LSI high end RAID stuff myself "just because" (see note 1), but the Adaptec stuff appears to be an acceptable alternative. The only way you are likely to be able to get this to work with FreeNAS, ESXi, and the Adaptec is to do something along the lines of what I describe in

https://www.truenas.com/community/t...ative-for-those-seeking-virtualization.26095/

but you really need to pay attention to the point I mention in Key Concept 1. If you build such a system without hypervisor-level storage redundancy, ESXi may happily "pause" your VM when it experiences an I/O error on your datastores, which is probably not what you are expecting to happen. Therefore this is not a good solution for more than maybe two disks.

(note 1) "Just because" is actually because Supermicro and Dell both make heavy OEM use of the LSI RAID systems, so stuff built here in the shop tends to favor LSI as a vendor to reduce the amount of "things to know." But we do have two 71605's in stock.

 

[Hell Bomb]

I chose the 71605s because several people said they were working pretty well on this forum and they showed as working well with several other BSD flavors. From what I can tell the 71605 is a true HBA when configured properly as it hands the HDDs directly to the operating system giving all the drive info, SMART, etc. Tested it on windows and they presented themselves as individual drives.

 

[jgreco]

I don't think anyone has ever said that on this forum. The total hits for "71605" is a weak page of results, several of which are from this thread, and a random spot check of them has a bunch of "don't do that" from forum posters including me.

The 71605 is not a true HBA, and even if it were a true HBA, it is not an LSI HBA, which is the HBA brand that has a rock solid FreeNAS driver.

So basically I wrote everything you need to know in the stickies linked above. In particular, and limited to points from a single sticky because I really do have better things to do,

2) FreeBSD has incredibly robust support for the LSI HBA's.

FreeBSD's LSI HBA (mps/mpr) drivers are authored by LSI and carefully designed to work with their HBA firmware. The FreeNAS userbase has installed many thousands of these cards which have, in aggregate, BILLIONS of problem-free run-hours. Not only are they known to work very well during normal operations, but they're also known to work correctly during ABNORMAL operations, such as when a disk times out or throws an error. SMART is properly supported. Forum members are incredibly familiar with all the variations on these and can provide useful assistance. Cards such as the LSI 9240-8i, IBM ServeRAID M1015, Dell PERC H200 and H310, and others are readily available on the used market and can be converted to LSI 9211-8i equivalents.

Your RAID controller was manufactured by Adaptec with an Adaptec chipset.

3) You must crossflash to IT/IR firmware

If you don't crossflash, then a lot of the remainder of this ALSO applies to LSI non-IT-20.00.07.00 HBA's!! The IR firmware is also fine but is a few percent slower. It is not clear there is any value to doing this as you would never want to use an IR virtual device with FreeNAS. We used to do this in the old days for boot devices, but with ZFS boot this is probably no longer relevant.

The LSI 9211-8i (PCIe 2.0 based on LSI 6Gbps SAS2008) and LSI 9207-8i (PCIe 3.0 based on LSI 6Gbps SAS2308) both require firmware 20.00.07.00.

The LSI 9300-8i (PCIe 3.0 based on LSI 12Gbps SAS3000) requires firmware ... 13? I don't have one of these online right now so my appreciation to whoever follows up with the current correct answer.

You haven't crossflashed your Adaptec RAID controller to LSI HBA IT firmware.

4) FreeBSD may or may not have good support for other HBA's/RAID controllers.

FreeBSD does support a wide range of HBA's and RAID cards, but in many cases the drivers have been reverse-engineered, or may cause various problems when there is an underlying issue with a component disk. ZFS has the capability to place immense I/O workloads on your HBA, and you don't really want the controller chip crashing and taking a bunch of your disks offline with it (known to happen). ZFS expects that its ability to talk with correctly functioning HDD's is 100.000% and bad things start to happen when controllers or drivers freak out.

It might seem very tempting to use some alternative brand of HBA or RAID controller that you happen to "have handy", but the problem is that you will be on your own private little adventure. Things might seem to work fine, until one day something bad happens. And then you might be outta luck. Some RAID cards do things like encapsulating their JBOD VD's in a partition, making it effectively impossible to plug the drives into a SATA port or LSI HBA.

If you have come to FreeNAS with the intention of making your NAS into a guinea pig for testing of an unknown and untested controller, then, by all means, go ahead. Just please be aware that the measure of success isn't "I got it to make a pool." It is possible for things to work for weeks, months, even years before something adverse happens. We promote the LSI controllers because they are proven to work, with an aggregate of billions of run-hours. You will not be able to duplicate that sort of rigorous testing.

You appear to be here in the forum because you're having problem with your "alternative brand." I could never have predicted that. :-(

5) A RAID controller that supports "JBOD" or "HBA mode" isn't the same.

In these devices, you are relying on the RAID card driver to communicate from the host to the controller. As previously noted, the LSI HBA drivers have billions of proven run-hours, but in many cases, RAID drivers aren't as solid. Some of FreeBSD's RAID drivers have been tweaked to cope better with device error handling on the theory that you have redundancy (JBOD isn't), many do not allow you to poll the drive's SMART status, and in many cases you can inadvertently set up bad situations with write caching, etc. ZFS has the ability to generate immense amounts of I/O traffic that can be a crushing workload for the weedy little CPU's on a RAID controller, can totally flood the cache on a RAID card, etc. As mentioned in the previous section, many RAID cards also do things such as encapsulation of JBOD within a partition, which effectively locks you into having to use that RAID card. This is super-bad for error recovery. With SATA ports or LSI HBA ports, SATA drives are completely interchangeable.

So that would seem to apply to your situation too. Your card is clearly a RAID card with delusions of HBA.

6) A RAID controller with write cache is particularly bad.

A RAID controller with a write cache is likely to get swamped by the massive I/O ZFS is pushing. These devices are typically sized to cope with the sorts of I/O a standard server would push around, update a file here, read an executable from there, do some database updates... but ZFS will perform operations such as scrubs and resilvers that will maintain massive I/O pressure for hours or days. Even the normal I/O is demanding, as a ZFS transaction group can easily be a gigabyte, every five seconds. Hiding the actual performance of devices behind a tiny RAID card cache is not a good idea as it leads to less-predictable performance.

And that seems to apply to your situation too; the 71605 has a write cache.

So.

 

[HoneyBadger]

I don't think anyone has ever said that on this forum. The total hits for "71605" is a weak page of results, several of which are from this thread, and a random spot check of them has a bunch of "don't do that" from forum posters including me.

The 71605 is not a true HBA, and even if it were a true HBA, it is not an LSI HBA, which is the HBA brand that has a rock solid FreeNAS driver.

I searched "Adaptec" and all I can find from anything as far back as 9.10 were recommendations in varying words of "Sell it and buy an LSI."

The FreeBSD LSI drivers have literal billions of run-hours and exabytes of data processed. The situation on other OSes (including SCALE) may be different, but for a vanilla TrueNAS build, don't even bother with anything else. Just get an LSI.