Dell R630 FreeNAS v9-10- Crashing on many fronts

[Bryon Brinkmann]

Anyone,
I have a Dell R630 2 Intel xeon E5-2697 v3 (16 cores per) with 384gb of ram, 4 1.98tb SSD's and 6 980gb SSD's w/ LSI controller card with 2 120gb SATADOM's where the OS was installed. I've been having some issue with this crashing on various fronts. First, I had a 8GB cache card that I think was questionable so I removed it from the equation thinking this might have been causing the issue (meaning tried to copy 80 gb data file and would crash @ 100mb to 50GB didn't matter). So when I remove it I was able to copy well over 200gb without issue. I still haven't done enough testing to determine. So I started to try to build a jail with advance setting and BOOM it puked again. I've run a couple of extensive hardware tests and everything passes without issue. This is a very beefy server and was wondering if there are any known issues with this type of setup. I'm new to FreeNas so I'm not sure of its limitations if any. So, my other question is, should I just build a ESXi and use Freenas under that. This server was given to me because of an issue that occurred in the server room and some water. I don't have a lot of trust in it but its passed every hardware test I've done, soooo....... I'm also not sure where the crashdump lives so I haven't attached one. Help? Any thoughts would be helpful.

 

[MrToddsFriends]

[...] LSI controller card with 2 120gb SATADOM's where the OS was installed [...] I had a 8GB cache card that I think was questionable so I removed it from the equation [...]

Sounds like HW Raid is being used, which would be a bad thing for FreeNAS. Which LSI HBA/RAID controller is installed in this system? The SATA DOMs (mirrored pair in FreeNAS?) are also connected to this LSI controller?

I don't have a lot of trust in it but its passed every hardware test I've done, soooo.......

Which tests did you perform? Memtest? CPU stress tests? Mass storage R/W tests? OS used for stress testing?

 

[Bryon Brinkmann]

Sounds like HW Raid is being used, which would be a bad thing for FreeNAS. Which LSI HBA/RAID controller is installed in this system? The SATA DOMs (mirrored pair in FreeNAS?) are also connected to this LSI controller?



Which tests did you perform? Memtest? CPU stress tests? Mass storage R/W tests? OS used for stress testing?

No Raid's - SATADOM's attached via Onboard SATA ports. FreeNas see's both during install process. They are installed on port I (other one is port J) Not mirrored. LSI Sas9211-8i no raid or mirrors all drives attached via the backplane > LSI - Used the extended Dell hardware testing (took about 6+ hours to complete). I'm testing to see if one of the SATADOM's is F'd. Do you know where the crash dump is stored?

 

[Sakuru]

/data/crash

 

[Bryon Brinkmann]

So, I was able to copy data over with no crash (mind you, not a whole lot just a couple 100+ gigs). Once I started creating a Jail "BOOM WENT THE DYNAMITE"! Server crashed like a hooker on a coming off a meth high. It now keeps rebooting PANIC's, Dumps crash in a loop, thoughts? FYI, The Jail was place on the /mnt/share/jails.

How can I boot to a single user mode to grab the crash dump?

 

[Sakuru]

Do you have a backup of your config? You could try reinstalling FreeNAS and importing your config.

Yes, I believe you can get into single user mode from the GRUB menu.

 

[Bryon Brinkmann]

Do you have a backup of your config? You could try reinstalling FreeNAS and importing your config.

Yes, I believe you can get into single user mode from the GRUB menu.

Any idea why it would crash building a Jail? I mean crash, it would just start to come back up and crash on boot, restart again and again and again. I've installed ESXi and a client Freenas with 32GB mem and 16 CPU's 5 1TB virtual disks and it works with no issue. I don't think theres a hardware issue since I've been able to install a bunch of other OS's without issue or crashes. Is there a memory limitation with FreeNas 386GB not supported? Maybe the CPU is not supported? Anywhere I can find this info?

 

[Sakuru]

Your hardware should be fully supported. I would have to see the crashes to know what's going on. It certainly should not crash just because you built a jail.

Do you know if your 9211-8i is flashed to IT mode?

 

[Bryon Brinkmann]

Your hardware should be fully supported. I would have to see the crashes to know what's going on. It certainly should not crash just because you built a jail.

Do you know if your 9211-8i is flashed to IT mode?

I think you're on to something there! I flashed the bios to another version when I brought it home. I just went to LSI's site a found that version on the very bottom of the firmware section. Let me flash it and I'll rebuild and let you know!

 

[Sakuru]

The current firmware version for 9211-8i is 20.00.07.00. You do not need the BIOS, but it shouldn't hurt anything. Make sure you use the IT version, not IR.

 

[Bryon Brinkmann]

So I tried to flash to the IT version but get an error. It's currently running the IR version.

 

[dlavigne]

And what was the error?

 

[Bryon Brinkmann]

The error I get is "The firmware flash image product ID is invalid".

 

[Bryon Brinkmann]

The error I get is "The firmware flash image product ID is invalid".

***Update***
I tried flashing for DOS and I got this error "Can not flash IT Firmware over IR firmware"! I've also tried erase the firmware and I get "erase command not supported on this platform"!

2 steps forward 3 steps back. WTF how the heck can I erase the IR firmware and install the IT?

 

[Ericloewe]

• Make sure you have the correct firmware (don't try LSI SAS 9211 firmware with an SAS2308 card, for instance)
• Use the EFI version of sas2flash from the EFI shell
• If it still bugs you, erase the current firmware and flash IT P20.00.07

 

[Bryon Brinkmann]

• Make sure you have the correct firmware (don't try LSI SAS 9211 firmware with an SAS2308 card, for instance)
• Use the EFI version of sas2flash from the EFI shell
• If it still bugs you, erase the current firmware and flash IT P20.00.07

Sorry to ask, but how do I do that? new to this

 

[Ericloewe]

If the motherboard BIOS does not include the EFI shell executable, get one from the internet, put it on a FAT32 USB drive and boot to it. Obviously, include the sas2flash executable and any other files you need.

 

[Bryon Brinkmann]

OK, so I removed the LSI and used the PERC H330 (configured as just an HBA non-raid). Everything was working great I.E. copying data, Volumes ETC. So I wanted to see what happened if I tried to install a plug-in. Selected Plex and it started the download blah blah and as soon as it started creating the jail, it shat it's self. Really would the controller cause that? I was copying data just fine 90mb/s +

 

[Bryon Brinkmann]

Crash dumps -

 

[Bryon Brinkmann]

Crash dumps -

If I read this right its crashing right when it starts creating the bridge/ifconfig on the lagg interface. Right Wrong or complete air ball?

I need to clarify a bit. I've been able to create a jail manually but not if a plugin is used IE Plex to create a jail. I haven't tried any other plugin's. Once it crashes it NEVER comes back and I have to rebuild the whole thing.