Our Setup
Hardware
BACKGROUND
We have a box with 60 hard drives running FreeNAS 11.1 U7 that has been working fine for a long time. Server is lightly used, and running stable with no known environmental or configuration changes occurring recently.
BEHAVIOR WE'RE EXPERIENCING
Server stopped responding. Upon reboot, it is giving a kernel panic when trying to import volumes.
STEPS TAKEN
KERNAL PANIC MESSAGE
Virtual Media Record Macro Options User List Capture Power Control Exit
Importing 17238327038330746117
txg 47837309 open pool version 5000; software version 5000/5; uts 11.1-STABLE 1
101505 amd64panic: solaris assert: offset + size <= sm->sm_start + sm->sm_size (
0x64060634802000 <= 0x230000000000), file: /freenas-releng/freenas/_BE/os/sys/cd
dl/contrib/opensolaris/uts/common/fs/zfs/space_map.c, line: 119
cpuid = 1
KDB stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe202386d7e0
upanic() at upanic+0x186/frame 0xfffffe202386d860
panic() at panic+8x43/frame 0xfffffe202386d8c0
assfai130) at assfai13+0x2c/frame 0xfffffe202386d8e0
space_map_load() at space_map_load+0x352/frame 0xfffffe202386d970
metas lab_load() at metas lab_load+0x2b/frame 0xfffffe202386d990
metas lab_preload() at metas lab_preload+0x89/frame 0xfffffe202386d9c0
taskq_run() at taskq_run+0x10/frame 0xfffffe202386d9e0
taskqueue_run_locked() at taskqueue_run_locked+0x147/frame 0xfffffe202386da40
taskqueue_thread_loop() at taskqueue_thread_loop+0xb8/frame 0xfffffe202386da70
fork_exit() at fork_exit+0x85/frame 0xfffffe202386dab0
fork_trampoline () at fork_trampoline +0xe/frame 0xfffffe202386dab0
--- trap 0, rip = 0, rsp = 0, rbp = 0
KDB: enter: panic
[ thread pid 0 tid 101998 1
Stopped at
db>
---
kdb_enter+0x3b: movq $0, kdb_why
I've ordered another HBA card just in case, but not sure where to start. What do you think?
Hardware
- 45 Drives Turbo 60 XL
- 60 SATA drives (6 10 disk vdevs raidz2)
- 4 LSI 9305-24i SAS HBAs
- 8 SSD cache drives
- 2 SSDs for config
- 1 flash drive with backup config and cache
- FreeNAS 11.1 U7
BACKGROUND
We have a box with 60 hard drives running FreeNAS 11.1 U7 that has been working fine for a long time. Server is lightly used, and running stable with no known environmental or configuration changes occurring recently.
BEHAVIOR WE'RE EXPERIENCING
Server stopped responding. Upon reboot, it is giving a kernel panic when trying to import volumes.
STEPS TAKEN
- After trying to reboot, same symptoms persist
- Ran memtest86 with no errors
KERNAL PANIC MESSAGE
Virtual Media Record Macro Options User List Capture Power Control Exit
Importing 17238327038330746117
txg 47837309 open pool version 5000; software version 5000/5; uts 11.1-STABLE 1
101505 amd64panic: solaris assert: offset + size <= sm->sm_start + sm->sm_size (
0x64060634802000 <= 0x230000000000), file: /freenas-releng/freenas/_BE/os/sys/cd
dl/contrib/opensolaris/uts/common/fs/zfs/space_map.c, line: 119
cpuid = 1
KDB stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe202386d7e0
upanic() at upanic+0x186/frame 0xfffffe202386d860
panic() at panic+8x43/frame 0xfffffe202386d8c0
assfai130) at assfai13+0x2c/frame 0xfffffe202386d8e0
space_map_load() at space_map_load+0x352/frame 0xfffffe202386d970
metas lab_load() at metas lab_load+0x2b/frame 0xfffffe202386d990
metas lab_preload() at metas lab_preload+0x89/frame 0xfffffe202386d9c0
taskq_run() at taskq_run+0x10/frame 0xfffffe202386d9e0
taskqueue_run_locked() at taskqueue_run_locked+0x147/frame 0xfffffe202386da40
taskqueue_thread_loop() at taskqueue_thread_loop+0xb8/frame 0xfffffe202386da70
fork_exit() at fork_exit+0x85/frame 0xfffffe202386dab0
fork_trampoline () at fork_trampoline +0xe/frame 0xfffffe202386dab0
--- trap 0, rip = 0, rsp = 0, rbp = 0
KDB: enter: panic
[ thread pid 0 tid 101998 1
Stopped at
db>
---
kdb_enter+0x3b: movq $0, kdb_why
I've ordered another HBA card just in case, but not sure where to start. What do you think?