FreeNAS 11.2Beta2 Crash and now Locks up on Pool Import

[Jerzy Sobski]

I am currently running Freenas 11.2 Beta 2. Overnight Freenas reboot possibly from a crash and it attempted to reboot and got hung up during a Pool import. I have tried several times to turn of Server and reboot it. It gets to same point before it locks up (At least it appears to be locked up). Below is a screean capture of where it locks up.

Is there any work around to find out what is causing this and have it boot up completely.

 

[Ericloewe]

Your image is broken. Please always post images directly to the forum by copy/pasting them into your messages.

 

[Chris Moore]

Did you upgrade the pool to the new feature flags?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Jerzy Sobski]

Did you upgrade the pool to the new feature flags?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

Yes.

 

[Chris Moore]

No turning back then. Can you fix the photo?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Jerzy Sobski]

Your image is broken. Please always post images directly to the forum by copy/pasting them into your messages.

On my end the image seems to show up when I look at message. I went and pasted image below which was attached using a link in original message.

 

[Chris Moore]

On my end the image seems to show up when I look at message. I went and pasted image below which was attached using a link in original message.

That's it? It just hangs there at that point.

Overnight Freenas reboot possibly from a crash

Did you get any indicator of why it rebooted itself initially?

 

[Ericloewe]

No turning back then.

Sure there is: https://forums.freenas.org/index.php?resources/zfs-feature-flags-in-freenas.95/

Feature flags are not a boogeyman that chains you to the arrow of time. Unless you removed a device from the pool, it will import just fine all the way back to 9.10 (assuming you don't have an active checkpoint and aren't using skein or sha512, all three of which are reversible).

 

[Chris Moore]

Feature flags are not a boogeyman that chains you to the arrow of time.

Good to know.

I am currently running Freenas 11.2 Beta 2.

Have you tried booting into your previous boot environment? Based on what @Ericloewe said, that might work to get you back to your data.

 

[Jerzy Sobski]

That's it? It just hangs there at that point. Did you get any indicator of why it rebooted itself initially?

No. When I woke up, I noticed I had no access to server. Went and check and saw what the screen capture above had. Everytime I reboot, it ends up at same point.

I have not tried to do anything other than try and reboot at this time. Since I am not that experienced with Freenas, I thought Id ask before I tried anything else.

The Drives for the pool are on two 10-bay towers connected to the server. Can I reboot the server with the two towers turned off and once booted turn turn on towers and add pool after freenas has booted? Currently only 15 drives are in use and the remaining fives slots are not in use.

 

[Jerzy Sobski]

Good to know.

Have you tried booting into your previous boot environment? Based on what @Ericloewe said, that might work to get you back to your data.

I have not tried that yet. Looking for any input before I try to do that.

I believe I have Freenas Settings backed from Freenas 11.2Beta1 before I upgraded to Beta2 recently.

Seemed to be running ok till I found it down this morning.

 

[Chris Moore]

It would be really nice if you would give us a rundown on the hardware you are using.

Seemed to be running ok till I found it down this morning.

We still don't know what caused the reboot that started this. It would be nice to know that. If you can turn the power to the disks off and reboot the FreeNAS, it may allow the system to boot to the point you can access the GUI. Once you are in the GUI, you should be able to power the disks and FreeNAS should detect them if there is no hardware fault. The pool should not automatically import. You can import the pools from a command prompt using zpool import -a
It is possible that the import may not put your pool back in the proper path. However, if it works, it will allow us to run a scrub to ensure the health of the pool.

 

[Jerzy Sobski]

It would be really nice if you would give us a rundown on the hardware you are using.

We still don't know what caused the reboot that started this. It would be nice to know that. If you can turn the power to the disks off and reboot the FreeNAS, it may allow the system to boot to the point you can access the GUI. Once you are in the GUI, you should be able to power the disks and FreeNAS should detect them if there is no hardware fault. The pool should not automatically import. You can import the pools from a command prompt using
zpool import -a

It is possible that the import may not put your pool back in the proper path. However, if it works, it will allow us to run a scrub to ensure the health of the pool.

Thanks, Ill give that a try tonight and send feedback.

 

[Jerzy Sobski]

It would be really nice if you would give us a rundown on the hardware you are using.

We still don't know what caused the reboot that started this. It would be nice to know that. If you can turn the power to the disks off and reboot the FreeNAS, it may allow the system to boot to the point you can access the GUI. Once you are in the GUI, you should be able to power the disks and FreeNAS should detect them if there is no hardware fault. The pool should not automatically import. You can import the pools from a command prompt using zpool import -a
It is possible that the import may not put your pool back in the proper path. However, if it works, it will allow us to run a scrub to ensure the health of the pool.

Here is an update. I went through and booted Freenas without drives online. Once booted, I went and turned each Set of drives on, From this I was able to discover that a drive went bad. I was able to replace it and its currently resilvering drive.

New problem is that the plugins were under the old warden system. Now that I have booted and abeen able to get volume back online, I now longer can see the Jails or Plugins under old GUI.

After looking into it further it appears that Volume has been mounted outside of its original location.

Below is a Screen capture of the structure right now. The Vol0 inside the "/mnt" was something created after silvering occurred once drive was replaced and SMB shares were turned on. The Vol0 is a newly created when SMB shares was turned on. The Vol0 which is outside located at bottom of tree is what should have been mounted in the "/mnt". Same with iocage which was also located in /mnt and now is outside that mount point.

I think this may be why the plugins did not start. I do remember during boot after replacing the drive a message came up that Vol0 already existed and was provided a warning.

Not sure how to resolve this especially while resilver is going on which from past experiences takes several days due to the size.

Any suggestions,
- Will going back to older snapshot fix this?
- Will uploading backed up config file I had made pre-upgrading from 11.2Beta1 to 11.2Beta2 fix it or cause more problems?
- Can rebooting cause problems with resilvering or will resilvering continue from it was prior to reboot?

Only good thing so far is that data although located in wrong location is still present.

Any help would be appreciated.

 

[Chris Moore]

You should have waited for the resilver to complete, then reboot. The system is not configured properly because the drives were not online when the system booted. We were supposed to be doing troubleshooting, not trying to get the shares and services back online.

After looking into it further it appears that Volume has been mounted outside of its original location.

I told you that would happen. You tried to get services running though and that may have broken things.

New problem is that the plugins were under the old warden system. Now that I have booted and abeen able to get volume back online, I now longer can see the Jails or Plugins under old GUI.

They are not working because the path is wrong. After the resilver you should be able to reboot the system clean and the vol0 will be mounted where it is supposed to be. You need to turn off SMB and delete that new folder that was created.

Not sure how to resolve this especially while resilver is going on which from past experiences takes several days due to the size.

I might have some idea why your resilver takes so long if you would have shared your hardware configuration, but you didn't. My system resilvers a drive in the main pool in about 3 hours but they are 4TB drives.

Can rebooting cause problems with resilvering or will resilvering continue from it was prior to reboot?

The resilver should pickup where it left off, but I don't like to reboot during a resilver.

 

[Jerzy Sobski]

You should have waited for the resliver to complete, then reboot. The system is not configured properly because the drives were not online when the system booted. We were supposed to be doing troubleshooting, not trying to get the shares and services back online.

I told you that would happen. You tried to get services running though and that may have broken things.

They are not working because the path is wrong. After the resilver you should be able to reboot the system clean and the vol0 will be mounted where it is supposed to be. You need to turn off SMB and delete that new folder that was created.

I might have some idea why your resilver takes so long if you would have shared your hardware configuration, but you didn't. My system resilvers a drive in the main pool in about 3 hours but they are 4TB drives.

The resilver should pickup where it left off, but I don't like to reboot during a resilver.

Thanks for the help.

Looks like Resilvering of drive is moving much faster under 11.2. Last time I had to resilver was under freenas 11.1. In 10 hours its already around 65%. Looks like it will be done this evening at this pace. Much better then its had been before.

The system I am using the following:
- HP ProLiant ML10 v2 Tower Server System Intel Dual-core i3-4150 3.5
- 32 GB (4x8 UDIMM) DDR3 Unbuffered Ram
- 15 - Seagate 4TB Enterprise Capacity HDD 7200RPM SATA 6Gbps 128 MB Cache Internal Bare Drive (ST4000NM0033)
- 2 ea 10 bay Hot Swap SATA External Enclosures (with Port multipliers)
- 2 Highpoint RocketRAID 622 PCIe 2.0 SATA (Set on JBOD) each card controls a 10 bay tower.

The entire volume is composed of the 15 drives combined. (There are 5 slots still free on one of the 10 bay towers.

Not sure if that helps any.

 

[Chris Moore]

So it is all one big vdev? Could you please share your zpool status? If that is what you did, and used SATA port multiplyer hardware, that is why it is so slow to resilver. I used port multiplyer hardware myself, as recently as a year ago, but it was not reliable for me. It was probably the drive failure that caused the reboot in your system, but it should not have. That is something I have seen a couple times with the BETA release. Please report that as a bug, so that the development team can see the crash data and try to get that fixed. The system not booting was probably the result of the port multiplyer hardware. I had that happen many times on my NAS and it is part of the reason I stopped using the port multiplyer. When did you upgrade to the BETA version? The driver for the port multiplyer hardware wasn't working properly for me since getting 9.10, but I was still getting it to do the work until 11.0. What version of FreeNAS were you using before you upgraded to the BETA?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Jerzy Sobski]

So it is all one big vdev?

Yes I believe so. (Note sure what vdev means)

Could you please share your zpool status?

If that is what you did, and used SATA port multiplyer hardware, that is why it is so slow to resilver. I used port multiplyer hardware myself, as recently as a year ago, but it was not reliable for me. It was probably the drive failure that caused the reboot in your system, but it should not have. That is something I have seen a couple times with the BETA release. Please report that as a bug, so that the development team can see the crash data and try to get that fixed.

I sure will. Where and which logs should I send to development team when I report this.
Just an FYI, the problem with port multipliers has also been an issue in the past. I was not aware that was causing the problem. As for the screen shot I posted. This was something new which I had never seen in the past and why it didnt dawn on me that it was a HD failure.

The system not booting was probably the result of the port multiplyer hardware.

This is good to know. I never realized the port multiplier was part of the problem why system was not booting after a HD went bad. explains why I had this same issue under older version.

I had that happen many times on my NAS and it is part of the reason I stopped using the port multiplyer.

What are you now using instead of the port multiplier now?

When did you upgrade to the BETA version?

I updgraded the same day that they were release on both 11.2Beta1 and 11.2Beta2;

The driver for the port multiplyer hardware wasn't working properly for me since getting 9.10, but I was still getting it to do the work until 11.0. What version of FreeNAS were you using before you upgraded to the BETA?

I was using 11.1 just before the upgrade to 11.2Beta1 and then to 11.2Beta2. Upgraded to each on the day they were released.


Again thanks for the help.

 

[Chris Moore]

Yes I believe so. (Note sure what vdev means)

No, not one big vdev, but I think I should get you to look at some material to help you with your terminology. Please look at these resources when you can:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

I sure will. Where and which logs should I send to development team when I report this.

You first need to create an account here: https://redmine.ixsystems.com/projects/freenas
Then, in the FreeNAS user interface (UI) you can submit a support request.
Here is a link to the documentation: http://doc.freenas.org/11/system.html#support
It will attach log files from your server automatically.

Just an FYI, the problem with port multipliers has also been an issue in the past. I was not aware that was causing the problem.

SATA Port Multipliers have been troublesome for many FreeNAS users over the years and pose a risk of data loss. As such, most forum regulars like myself suggest moving away from them. I know that they represent a fairly significant hardware investment; especially with two, ten drive, towers. The thing is, in the right set of circumstances, it could cost you your data and I guess you wouldn't want that. What I did was move my drives into a SAS expander chassis. You can often pick one of those up for a fairly reasonable price, where they have been retired from a data-center. I can make some hardware suggestions if you are interested in that.

I updgraded the same day that they were release on both 11.2Beta1 and 11.2Beta2;

After you file the bug report, and when you are ready to reboot, you might want to select your BETA1 boot environment, there have been several users that reported problems with deleting files under the BETA2 (it is a BETA after all) and when they tried to delete a directory of files, it would cause the system to lockup with the only way to get it to respond again was to reboot.

 

[Chris Moore]

Since you are using the BETA release, this would be the correct link to the documentation:
http://doc.freenas.org/11.2/system.html#support