boot up stuck at importing pool

[cyberjock]

Actually, I think I know what your problem is... you shouldn't have 18 drives in one vdev that is RAIDZ1. I'd bet that because of that super-wide vdev you are having problems if something even sneezes on your zpool.

 

[nkrapf]

is there any know issues with 2012 disk optimization against an freenas iscsi target?
we didnt have these issues till we upgraded to 2012r2 and ran disk optimization to reclaim space on freenas vol

we started to run it again today and the system is non responsive again after about 30-60 min of windows2012r2 running disk optimization.

 

[nkrapf]

we have 16 3tb drives
then the log ssd and cache ssd
and 2 usb boot

 

[nkrapf]

ahh...
this might be cause.
will run extensive ecc memtest next

 

[cyberjock]

When the box crashes and reboots the memory controller can be left in an inconsistent state. For this reason, any ECC errors that occur after a crash/random reboot but before you do a cold shutdown and power-on should be considered erroneous themselves (but it should definitely be monitored to see if it was a one-time thing or recurring since a recurring problem may be because of bad RAM). A cold boot is required to ensure the memory controller and memory is in a consistent state, otherwise you run the risk of getting random ECC errors when there actually is no hardware problem.

From my experience, if your RAM is truly bad you'd be having errors regularly instead of just a one-off.

Unfortunately this creates it's own vicious circle. If the RAM is truly bad, it can cause the box to reboot randomly. So then if its rebooting randomly you can't tell if its rebooting because of a RAM problem or because of "something else". So you are forced to prove that it is or isn't the RAM at fault.

 

[nkrapf]

we will let the memory test run it course just to be sure.
all previous crashes did not post a memory error. this is the first time IPMI reported a memory issue.

the crash happens as follows.
we notice web, ssh, console is non responsive. after a few minutes the system reboots itself.

the only change we have made was upgrading to latest releases and upgrading the server that accesses it to win2012r2
we upgraded the veeam box to 2012 because we previously ran into issues where windows was reporting plenty of free space but the pool was reporting over 80% full.
Prior to upgrading i tried using sdelete and some other scripts I found on nimble support site that zeros out the free space, this was working but was extremely slow.
We then decided to upgrade to 2012 since it had the advanced disk management that was required and started to run disk optimization.
at least 2 of the crashes I know this task was running when it crashed. the other time i am not sure if the task auto kicked off or not, but i suspect it because the defrag task is at 3am and the crashes in middle of night were just after 3am. maybe be a possible bug??

 

[nkrapf]

memory test passed with no errors, will try running and disable the disk optimization in 2012 and see how it goes, will take about 9hr for system to boot based on last 3 times.

 

[nkrapf]

no crash last night with win2012r2 disk optimization turned off, running on original usb stick.

 

[cyberjock]

I'm not sure why you are defragmenting VMs on FreeNAS iSCSI targets. They add no value and actually can create new problems (as well as hurt performance overall).

You should be leaving the defrag off, even if it doesn't resolve the issues in the end.

 

[nkrapf]

Its not so much the defrag process, its the trim/zero out free space.
Windows 2008r2 is not releasing the free space on the freenas iscsi target. So while windows is reporting more than 50% free space, freenas was reporting less than 20% free space.
As mentioned before, we attempted to use sdelete to zero out free space, which did work but was very slow and disk io intensive. We then found some scripts that do the same thing on the nimble support site, this was working better but still very slow. We tried a product call Perfect storage that seemed to work better and was more disk IO aware and could run in background http://www.raxco.com/products/perfectstorage

We noticed that Windows2012R2 supports the iscsi trim / release of free space on the freenas device so we upgraded from 2008 to 2012. After this upgrade everything was still running ok. It was not till we ran windows disk optimizer, then freenas crashed.
we had turned optimizer on as schedule around 3am. the system crashed 2 more times just after 3am. then once everything was running again we started running optimizer again because we still have about 10TB to reclaim, freenas crashed again. Last night after we got freenas running again we turned off optimizer and freenas has not crashed.

So what does Storage Optimizer do in Windows 2012 R2?
The Storage Optimizer in Windows 8/Server 2012 , also takes care of maintenance activities like compacting data and compaction of file system allocation for enabling capacity reclamation on thinly provisioned disks. This is again platform specific, so if your storage platform supports it, Storage Optimizer will consolidate lightly used ‘slabs’ of storage and release those freed storage ‘slabs’ back to your storage pool for use by other Spaces or LUNs. This activity is done on a periodic basis i.e., without any user intervention and completes the scheduled task provided it is not interrupted by the user. I am not getting into storage spaces and storage pools as this will further lengthen this topic, you can refer TechNet regarding Storage Spaces overviewfor details.

on a side note, we have nearly exact set up at a second location but not as sever issue with unclaimed space and there are no issues on that box. So this leads me to think there might be a bug with freenas and windows2012 optimizer when it reclaims x amount of space. the second unit did not have very much free space that needed to be reclaimed.

 

[nkrapf]

also the freenas target is not part of the vmware environment. The Veeam VM directly attaches to the freenas iscsi target using windows iscsi initiator. nothing to do with vmware.

 

[mjws00]

Wouldn't surprise me if there was a bug with the 2012 optimizer and the current iscsi implementation. It is using low level disk functions that could be broken in the same way some of the vhd, vhxd, features are. Even unmap/trim are pretty new, so not unreasonable to think something might be up with the Microsoft stack, that tool, and iscsi.

I'm not saying that's what's up. But you could be hitting a known bug or bleeding edge issue from a different angle. I'd be looking in that vein as well.

 

[nkrapf]

so its been stable for a while now. only difference is the disk optimization task is de-activated

 

[nkrapf]

some additional info.
Windows2012 recognizes the ISCSI target ( Z drive) as a thin provisioned drive. based on that it runs a trim process.

 

[Darren Myers]

Wait , if its a iSCSI volume, the first screen shot shows 128% usage on vol1 yet the pool is at 54%. Is this over the golden rule of iSCSI targets or am i wrong somewhere?

 

[syion]

I was having the same problem but perhaps for different reason. I have been reading this thread the other one referenced by Cyberjock. We did a reboot of our freenas server this am and it would not load the gui. I went to the server to find it still booting up in the same state as this thread started in. Prior to the reboot I deleted 3 snapshot tasks and created a single new one. We also setup zfs replication and were testing it on small dataset and it worked perfect.

When I saw the boot error I tried a different USB flash drive and that yielded the same result. After reading the this post I put the original USB boot drive in and waited it out. All is back to normal - thanks to this tread.



BOOTING ERROR

Beginning ZFS Volume Imports

Importing 1314977310191665171

tsg 9808897 import pool version 5000; software version 5000/5; uts 9.3-RELEASE-p16 903000 amd64


SYSTEM INFO

ASUSROCK Motherboard
Intel(R) Core(TM) i5-4690K CPU @ 3.50GHz
4 x 3TB WD RED NAS DRIVES
500MB of data using just under 1TB of total disk space with RAIDZ2
16GB RAM
Single Volume, 20 +/- datasets
RAID-Z2
boots from usb stick 8GB
latest update release of 9.3 9.3-stable 201506292332

 

[nkrapf]

we have not had any issues since disabling the windows2012 disk optimization task.

 

[nkrapf]

well still no issues since turning off disk optimization in windows 2012. looks like its related to that.
thank you everyone for all the support and feed back.