sas expander keeps going offline

[RLK]

Hi I have an HP sas expander with 10 SAS 2tb drives that are in mirror with freenas 11.2 running as a vm. Periodically (once a week or two ) the freenas vm will hang and the NFS and iscsi it is serving to my network just die. When you go to freenas vm and send ctrl-alt-del the freenas vm responds but doesn't ever recover. when you reboot freenas all zpools are fine and scrubs find no errors. how do I findout next steps to troubleshoot if no drive in array is failing and there are no zpool errors. BTW the freenas vm has a lsi9200 HBA card on passthrough for drive management. Thanks in advance.

 

[Chris Moore]

What media is the FreeNAS booting from? Why do you think this has anything to do with the SAS expander?

 

[RLK]

 

[RLK]

 

[RLK]

hi chris thanks for the reply, I posted some screen shots from today when the system went offline again. tank is using all drives in expander. the boot disk is my synology nfs on which i created a 80 gig boot drive through vmware to house the freenas vm. when I try to go to the gui page to get more information it doesn't respond though as you can see above the shell does. thanks

 

[RLK]

 

[Chris Moore]

View attachment 30038

This is your boot drive.

I posted some screen shots from today when the system went offline again. tank is using all drives in expander. the boot disk is my synology nfs on which i created a 80 gig boot drive through vmware to house the freenas vm. when I try to go to the gui page to get more information it doesn't respond though as you can see above the shell does. thanks

I think we need a lot more details about your hardware configuration here because you are saying that the boot drive is on another server (Synolgy no less) and being passed to FreeNAS by vmware? How is that ??
The problem you are having is that the boot drive is effectively failing in a similar manner to a USB drive that is going bad. The FreeNAS (which you have virtualized) needs a local datastore to use for the boot drive so it can have uninterrupted access to the device.
Did you read the guidance on how to virtualize FreeNAS before you did this?

 

[RLK]

Yes I read the freenas guide and many other websites before trying this. The guide does not specify that the boot drive cannot be on an NFS datastore for the VM--(the supermicro miniserver FreeNAS vm is running does have a local ssd but I need to keep it reserved for cache tier when I begin testing vsan down the road and there is very limited space in the case). The synology is the only reliable NAS in my lab at the moment. It is simply serving up the NFS datastore that I was creating the VMs on as a trial until I get a stable system that I can test -- I'd like to see if the multitude of comments (pro and con) for virtualizing freenas apply to my setup. One screen is the boot device errors but the other shows the boot device with no error and the "tank"(sas expander) unavailable. So to the original question what is the best mechanism for tracking down where it went off. syslog didn't show anything obvious but I am not very familiar with its output and would like to know the best location to search for the errors to save troubleshooting time-this is only occuring about once a week. Of course I could just scrap the idea and put FreeNAS on a dedicated machine but I was trying to find out what the specific issue is here. If it is that the boot drive cannot be on NFS datastore then TIA for solving the problem for me.

 

[Chris Moore]

Yes I read the freenas guide and many other websites before trying this. The guide does not specify that the boot drive cannot be on an NFS datastore for the VM--(the supermicro miniserver FreeNAS vm is running does have a local ssd but I need to keep it reserved for cache tier when I begin testing vsan down the road and there is very limited space in the case).

I suggest that you find room for a small SSD to make a boot pool for your FreeNAS VM because having it mounted remotely is not working.

You might also want to read these forum posts regarding virtual FreeNAS configuration:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.
https://forums.freenas.org/index.ph...ide-to-not-completely-losing-your-data.12714/

Virtually FreeNAS ... an alternative for those seeking virtualization
https://forums.freenas.org/index.ph...ative-for-those-seeking-virtualization.26095/

FreeNAS 9.10 on VMware ESXi 6.0 Guide
https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

and these informative threads:

The ZFS ZIL and SLOG Demystified
http://www.freenas.org/blog/zfs-zil-and-slog-demystified/

Some insights into SLOG/ZIL with ZFS on FreeNAS
https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

Testing the benefits of SLOG using a RAM disk!
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561/

SLOG benchmarking and finding the best SLOG
https://forums.freenas.org/index.php?threads/slog-benchmarking-and-finding-the-best-slog.63521/

How to relocate swap to an SSD or other partition - [not needed in v 11.1 and newer systems]
https://forums.freenas.org/index.php?resources/how-to-relocate-swap-to-an-ssd-or-other-partition.68/

 

[Chris Moore]

One screen is the boot device errors but the other shows the boot device with no error and the "tank"(sas expander) unavailable. So to the original question what is the best mechanism for tracking down where it went off.

You probably need to post the full details of how you configured this, because if it isn't working, you did something wrong and to find the problem, we need to know what you did. It is incredibly unlikely that the SAS expander has anything to do with the problem, but depending on how you pass you SAS interface through to the VM, that could be all wrong.
Details make all the difference.

 

[Chris Moore]

I was trying to find out what the specific issue is here. If it is that the boot drive cannot be on NFS datastore then TIA for solving the problem for me.

That is the first problem, not guaranteed to be the only problem. Please look at this guidance for the beginnings of what we need to know for troubleshooting but keep in mind that we need to understand how you configured the VM and passed hardware into the VM.

Forum Guidelines
https://www.ixsystems.com/community/threads/forum-guidelines.45124/

 

[RLK]

Sorry for delay in response ... took some time to get new parts and reconfigure system
the system now has ssd local datastore with the freenas vm residing on it/using that for boot
was working for few days then failed again now with da6--this is an HP d2600 with HP sas 2tb 3.5 inch drives
the web interface is again unresponsive and the vm console/shell are as well, took some screenshots to provide extra info requested

 

[RLK]

 

[RLK]

 

[RLK]

 

[RLK]

 

[RLK]

 

[RLK]

HTH
LSI card is 9200-8e pci card with direct passthrough to the freenas vm (with sff8088 direct cable connection do the hp drive array with 10 2tb sas drives all showing healthy on status dashboard at every reboot) -Looks like a drive is failing from above messages but with console frozen is there a way to get more data about the issue? FWIW when I have had access to the dashboard it has always reported the drives are healthy. TIA

 

[RLK]

Rebooted freenas vm and have smartctl running a long test on all drives ... estimate is 5-6 hours for completion

 

[Spearfoot]

@RLK : I run three virtualized FreeNAS All-In-One servers and I've always reserved all of the the FreeNAS VM memory so that it is 'locked'. I use ESXi 6.0, but later versions support this feature as well, probably under the 'Resources' settings for the VM.

Not sure that this will fix your problem, but it might help...