Unable to boot FreeNAS - Error Reading SATA Passthru

[Keith Pratola]

Earlier this week our FreeNAS server crashed. When I checked on IPMI, it appeared to be a memory issue.



We removed the bad memory, but the issue appears to be much worse.







This is a Supermicro server with a LSI 9207-4i4e and about 60 hard drives. The HBA was running firmware 16 in IT mode. After some research it sounded like version 17 might solve the problem, so I upgraded. Still have the same issue. What I am not sure of is if this is a hardware issue or software issue. We are running the latest version of FreeNAS. I tried booting from the last release that was installed and also from the default install.



They all have the same issue. Anyway I can fix this without losing data?

 

[jgreco]

Firmware phase 20 is the proper firmware for the latest version of FreeNAS. Suggest trying to flash the firmware to that and then see what's up.

 

[Keith Pratola]

Is there release notes on that? I had version 20 when I first installed FreeNAS 9.3 back in April and was getting "WARNING: Firmware version 20 does not match driver version 16 for /dev/mps0" which is why I downgraded to firmware 16. When was the FreeNAS driver version upgraded to 20?

 

[Bidule0hm]

Since the update of september, the ones with version 16 are from june and before.

 

[Keith Pratola]

I will give it a try. Unfortunately the virtual media for the IPMI doesn't work, so I am stuck driving to the datacenter to upgrade the firmware and I was there just a couple of nights ago, but I didn't realize FreeNAS was up to version 20.

 

[Ericloewe]

Also, update the expander. The messages sound like the expander is involved in the problem.

 

[Keith Pratola]

So two things. From the boot menu I tried booting from the FreeNAS (default) which is the install from April. That had the same issue. If it was firmware related, wouldn't the original install that uses firmware version 16 still boot? As for the SAS expander, I disconnected the additional disk shelf which is where the expander is located and still got the same error messages. Plus updating the expander can be a bit of a challenge. The Supermicro motherboard in this server doesn't boot from USB or a USB CD ROM. Not sure why or if it is expected to use floppy disks just to update the BIOS. The only way I was able to update the firmware on the HBA was to put it in a different server. This is a Supermicro X8DTI-F motherboard.

 

[jgreco]

Yeah, firmware updates suck. In theory you may be able to do it directly from FreeNAS but that strategy seems a little fraught with peril.

It kind of feels like you have a hardware fault developing somewhere. Is it any better after having reseated the HBA (which you had to do to update the firmware)?

 

[Keith Pratola]

I've tried doing a firmware update in the past from FreeNAS and it wouldn't work. I believe because the HBA is in use at the time. I also feel it is probably hardware. Not only was the HBA reseated, I tried different PCI slots, and even swapped the motherboard with an identical model and it is the same issue.

Unfortunately it is a big SAN that is a bit older and no Supermicro support on it. The motherboard is in a 4U chassis and there is an additional 4U chassis with the SAS expander. So I have the possibility of backplane issues, expander, HBA... The only thing that occurred was the server crashed due to memory. Prior to that it has been running decent, but there has been a few unknown reboots. Just sucks because it has about 80 TB RAW capacity.

 

[jgreco]

Yeah, to be able to do the firmware upgrade from FreeNAS I believe you'd need to be in singleuser mode, and maybe some other bodgery. I intend to see if I can figure that out one of these days because we've got these in ESXi boxes that I don't really want to down just to do firmware upgrades on.

 

[cyberjock]

Can you post the specs for the Supermicro server?

I'm wondering if the problem isn't backwards. The RAM error is caused by a system crash (system crashes can leave the memory controller in an inconsistent state) and the crash occurred because of your mps issues.

Firmware v17 should never be used on any version of FreeNAS. There are no issues with p16 that FreeNAS is directly involved in that have affected users.

 

[Keith Pratola]

Can you post the specs for the Supermicro server?

I'm wondering if the problem isn't backwards. The RAM error is caused by a system crash (system crashes can leave the memory controller in an inconsistent state) and the crash occurred because of your mps issues.

Firmware v17 should never be used on any version of FreeNAS. There are no issues with p16 that FreeNAS is directly involved in that have affected users.

Well, I can either go back to firmware 16 or upgrade to 20. If version 16 is still good, then I doubt my problem is firmware related. It is a Supermicro X8DTI-F motherboard with Intel Xeon E5503 and 48 GB of memory. After the memory error, I replaced all the memory. It is using a LSI 9207-4i4e HBA card. I don't know the chassis model numbers, but essentially a 4U chassis with 24 hard drives with another 4U expansion shelf that has probably 36 or more hard drives in it. FreeNAS is installed on a single SSD. This was originally an Openfiler SAN, but I re-purposed it with FreeNAS. It has been mostly good up until it crashed last week and no longer boots.

 

[Keith Pratola]

So what are my chances of getting this working without any data loss? I am considering just installing from scratch to see if at least all the disks are being detected.

 

[jgreco]

It isn't clear where the problem is. It seems likely that there's something wrong.

I'd follow the burn-in testing steps (disks: read tests only, no writing) suggested in my burn-in sticky.

https://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/