Importing pool crashes OS

[yarkin]

Hi to all!

I have FreeNAS 9.10-STABLE-201605021851 installed 2 month ago to a new hardware and running well until last night. It has configured 2 ZFS pools: ssdpool1 (4 SSD RAIDZ1) and hddpool1 (5 HDD RAIDZ2). I'm a beginner at FreeNAS and ZFS.

After FreeNAS started scrubbing ssdpool1 pool last night (it sent me email) OS went to reboot and then failed to init ZFS (it was prompt like "db>").
I haven't found what to do and just reinstalled FreeNAS to last stable release (9.10-STABLE-201606270534) and then tried to import my data pools. hddpool1 was impoerted without any issue, but when I tried to import the other one there were many errors in server console like:

Code:

Tracing command kernel pid 0 tid 100088 td 0xffffff000262b000
sched_switch() at sched_switch+0x154
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sleep() at _sleep+0x357
taskqueue_thread_loop() at taskqueue_thread_loop+0xb7
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8012426d30, rbp = 0 ---

and then server went to reboot.

'zpool import' shows ssdpool1 is fine:

Code:

# zpool import
   pool: ssdpool1
     id: 5935814353070452819
  state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
config:

        ssdpool1                                        ONLINE
          raidz1-0                                      ONLINE
            gptid/a8a04159-393a-11e6-8dc6-0025905c9f10  ONLINE
            gptid/a8c9b9e9-393a-11e6-8dc6-0025905c9f10  ONLINE
            gptid/a8f63183-393a-11e6-8dc6-0025905c9f10  ONLINE
            gptid/a91fa680-393a-11e6-8dc6-0025905c9f10  ONLINE

I can import it in readonly mode:

Code:

# zpool import -F -f -o readonly=on -R /mnt/temp ssdpool1

and 'zfs list' shows everything inside the pool.

I tried to make a copy of needed data (2 zvols) to hddpool1 by using 'zfs send ... | zfs recv ...' , the first zvol was copied well (it's small, about 1 MB of data), but then the second one (it's big, about 300 GB of data) was copying some time the error occured again and system went to reboot.

I've checked my RAM (new memory with ECC) by memtest86 and it's done 2 passes without any errors.
Now I'm running 'zdb -e -bcsvL ssdpool1' but it requires many hours to done.

What should I do if zdb makes the same error and my system is rebooted or even doesn't show any errors?


Hope for your help! Thanks.

 

[rs225]

You will probably need to do a manual copy of the dataset that generated the reboot. Eventually you may determine which file or directory triggers the crash. Copy out everything else. Once everything is backed up, the pool will probably need to be destroyed and re-created.

Did this pool ever have a cache/L2ARC device added and removed from it?

 

[yarkin]

You will probably need to do a manual copy of the dataset that generated the reboot. Eventually you may determine which file or directory triggers the crash. Copy out everything else. Once everything is backed up, the pool will probably need to be destroyed and re-created.

Did this pool ever have a cache/L2ARC device added and removed from it?

Yes, I planned to copy all data and re-create, but this pool contains only 2 zvols, which exported as iSCSI block devices (and formated as NTFS). One of them I've already copied, but most important to me copy the last one. Can I copy zvol by dd utility?

No, the pool hasn't have any caches (it's pool of SSD disks).

New pitfall is that 'zdb' reads my pool well (20K IOPs, utilizes all CPUs) until their process reaches 16 GB of RAM (server has 32GB RAM), but after that it's slowing down to about 500 IOPs or less, and utilizes less that one CPU core :-(

 

[yarkin]

I found some crash info in '/data/crash/' maybe it will helpful.

 

[rs225]

Can I copy zvol by dd utility?

Yes, but if it crashes during the dd, it will be more complicated.

 

[yarkin]

I imported the pool in readonly mode, and manually appended zvol to '/etc/ctl.conf' and reloaded 'ctld' daemon. Then destination server was able to mount this block device and show file structure (Hyper-V storage). After that I started copying VM disks' files to other system. It looks like only one disk was damaged and by the fortune this disk is the most important for me (it's a small in size and rare modified, especially at night). I wrote script which reads from beginning of the file and them from the end, and it seems I lost only about 1MB of data.

I'm going to completely erase all data on disks from this pool and then re-create it. What is the best way to test stability of new pool?

 

[rs225]

I looked at the crash dump, but all I can make of it is that a code assertion was triggered.(Something that should not happen.) It looks like ZFS tried to allocate too large a block. Were you using any options that might affect that, such as recordsize, blocksize, or compression settings?

Another possibility is that this is a side effect of corruption, so you should consider whether something is wrong with the hardware.

 

[yarkin]

Were you using any options that might affect that, such as recordsize, blocksize, or compression settings?

I use default settings most of time, and if these options are not in UI or in secret/advanced sections of UI, I didn't change it.

Another possibility is that this is a side effect of corruption, so you should consider whether something is wrong with the hardware.

Yes, maybe this is it. I've tasted RAM with no errors.
Previous (the same) chasis had problem with one hot-swap SATA bay, but when a disk was working in this bay FreeNAS just showed timeout errors and didn't crash.

For now, I've destroyed bad pool and then tested underlying disks by 'fio' for read/write load and everything was good.
After that I created new pool with one dataset and one zvol inside it, and started testing its block device '/dev/zvol/ssdpool1/dataset/storage1' by the same utility. After 3 hours of different and well passed tests I stopped watching to it, but in a hour after that I realized the server didn't respond even to ping. In the console it was some errors about swap on the pool and I/O errors. (Can I disable swap on my data disks? The server has enough installed RAM, 32GB, and I'm going to add 32GB more). And after reboot the server lost one disk (but I don't have physical access to the server now).

Why did FreeNAS go down after some disk errors instead of marking pool as degraded? :-(

 

[rs225]

There should always be some swap; FreeBSD kernel algorithms are designed with the expectation that it exists. However, you can turn off the swap on the pool drives, and have a single swap on some other drive. If you can avoid a drive or bay that has problems, this would be ideal. I/O errors on swap could be causing crashes.

For testing, you could turn off all swap and see if the problem disappears. But if swap I/O errors are the cause of your crashes, then it sounds like you do need at least one swap device.