Panic on ZFS import

[Mdunlap]

I'm getting a kernel panic on import. I've tried running it in debug and then force mounting in read only. The kernel locks up and I have to reboot.

I've moved the drives to a new machine to make sure there were no hardware issues.

I was trying with ubuntu and zfs

Code:

root@ubuntu:~# modprobe zfs zfs_rescue=1
root@ubuntu:~# zpool import -o readonly=on -N Storage
cannot import 'Storage': pool may be in use from other system, it was last accessed by freenas.local (hostid: 0x10387101) on Tue May 29 23:34:32 2018
use '-f' to import anyway
root@ubuntu:~# zpool import -fo readonly=on -N Storage

in dmesg:

Code:

[140227.665512] VERIFY3(c < (1ULL << 24) >> 9) failed (36028797018963967 < 32768)
[140227.665515] PANIC at zio.c:230:zio_buf_alloc()
[140227.665516] Showing stack for process 30009
[140227.665518] CPU: 6 PID: 30009 Comm: zpool Tainted: P		   O	4.13.0-36-generic #40~16.04.1-Ubuntu
[140227.665519] Hardware name: MSI MS-7885/X99A RAIDER (MS-7885), BIOS P.40 05/25/2016
[140227.665519] Call Trace:
[140227.665525]  dump_stack+0x63/0x8b
[140227.665533]  spl_dumpstack+0x42/0x50 [spl]
[140227.665535]  spl_panic+0xbe/0x100 [spl]
[140227.665572]  ? vdev_queue_io+0x188/0x200 [zfs]
[140227.665573]  ? mutex_lock+0x12/0x40
[140227.665590]  ? zio_wait_for_children+0x82/0xa0 [zfs]
[140227.665605]  ? zio_vdev_io_done+0x2e/0x180 [zfs]
[140227.665607]  ? kmem_cache_alloc+0x15c/0x1b0
[140227.665610]  ? spl_kmem_cache_alloc+0x72/0x7c0 [spl]
[140227.665621]  ? arc_space_consume+0x55/0x70 [zfs]
[140227.665631]  ? buf_cons+0x6a/0x70 [zfs]
[140227.665633]  ? spl_kmem_cache_alloc+0x126/0x7c0 [spl]
[140227.665648]  zio_buf_alloc+0x43/0x60 [zfs]
[140227.665658]  arc_get_data_buf.isra.22+0x26c/0x390 [zfs]
[140227.665668]  arc_buf_alloc+0x10b/0x170 [zfs]
[140227.665677]  arc_read+0x1b2/0xa10 [zfs]
[140227.665690]  traverse_prefetch_metadata+0xbc/0xd0 [zfs]
[140227.665701]  traverse_visitbp+0x566/0x980 [zfs]
[140227.665713]  traverse_visitbp+0x5cf/0x980 [zfs]
[140227.665724]  traverse_visitbp+0x5cf/0x980 [zfs]
[140227.665735]  traverse_dnode+0x7f/0xe0 [zfs]
[140227.665745]  traverse_visitbp+0x712/0x980 [zfs]
[140227.665756]  traverse_visitbp+0x5cf/0x980 [zfs]
[140227.665767]  traverse_visitbp+0x5cf/0x980 [zfs]
[140227.665778]  traverse_visitbp+0x5cf/0x980 [zfs]
[140227.665789]  traverse_visitbp+0x5cf/0x980 [zfs]
[140227.665799]  traverse_visitbp+0x5cf/0x980 [zfs]
[140227.665810]  traverse_dnode+0x7f/0xe0 [zfs]
[140227.665821]  traverse_visitbp+0x888/0x980 [zfs]
[140227.665831]  traverse_impl+0x1ab/0x410 [zfs]
[140227.665847]  ? spa_async_suspend+0x90/0x90 [zfs]
[140227.665857]  traverse_dataset+0x45/0x50 [zfs]
[140227.665872]  ? spa_async_suspend+0x90/0x90 [zfs]
[140227.665883]  traverse_pool+0x178/0x1b0 [zfs]
[140227.665897]  ? spa_async_suspend+0x90/0x90 [zfs]
[140227.665912]  spa_load+0x19f9/0x1cb0 [zfs]
[140227.665914]  ? taskq_create+0x1fd/0x340 [spl]
[140227.665929]  spa_load_best+0x5d/0x280 [zfs]
[140227.665932]  ? zpool_get_rewind_policy+0x18c/0x1a0 [zcommon]
[140227.665946]  spa_import+0x1d0/0x720 [zfs]
[140227.665951]  ? nvpair_value_common.part.11+0xc7/0x120 [znvpair]
[140227.665968]  zfs_ioc_pool_import+0xfa/0x130 [zfs]
[140227.665983]  zfsdev_ioctl+0x45d/0x4f0 [zfs]
[140227.665986]  do_vfs_ioctl+0xa4/0x600
[140227.665987]  ? __do_page_fault+0x283/0x4f0
[140227.665989]  SyS_ioctl+0x79/0x90
[140227.665990]  entry_SYSCALL_64_fastpath+0x24/0xab
[140227.665992] RIP: 0033:0x7f4af13f4f47
[140227.665992] RSP: 002b:00007ffde625ca78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[140227.665993] RAX: ffffffffffffffda RBX: 00007ffde62600b0 RCX: 00007f4af13f4f47
[140227.665994] RDX: 00007ffde62600b0 RSI: 0000000000005a02 RDI: 0000000000000003
[140227.665994] RBP: 000000000096d060 R08: 000000000096d060 R09: 0000000000a791f8
[140227.665995] R10: 000000000000019d R11: 0000000000000246 R12: 000000000096fd80
[140227.665995] R13: 00000000009765f8 R14: 0000000000000000 R15: 0000000000000000

Looking online, someone has had success doing this: https://github.com/zfsonlinux/zfs/issues/6414

I'm not sure what else to try. It just stopped working one day and kernel panicked.

The freenas 11 machine is quite capable 16gb ram 4 core i7
The Ubuntu box is 32gb ram 8 core i7

Any advice would be appreciated.

 

[Mdunlap]

Should I be testing this on a blank FreeBSD box because zfs versions between BSD and Linux are different?

 

[Mdunlap]

Thanks. I’ve got 16gb.

 

[Stux]

VERIFY3(c < (1ULL << 24) >> 9)

You should see if you can find this code in the ZoL codebase...

I want to know what c is, as it’s stupid big, and the code is complaining because it’s bigger than 32KB.

My guess is it’s trying to allocate 36 Exabytes!!!

If you import in FreeNAS it very well might work.

 

[Mdunlap]

This script worked flawlessly, I made a dd copy of all of my drives. ran this script on each of the drives https://gist.github.com/jshoward/5685757 and reverted back to an old txg.

Thanks!

 

[Mdunlap]

/thread

I dont know how to close threads, but this one is solved