System hungs (kernel bug ?)

[dea75]

Hi to all !

I have a serious problem with a production server on Freenas 8.0.3-4-(p1), seems identical to Ticket #1038.

Hardware:
HP Proliant 350 series, ECC RAM, redundant power supply, HP Smartarray RAID controller, USB pendrive for OS.
No hardware problems detected.

On Freenas 8.03-4-p1, after 12-15 uptime days system crash: "Fatal double fault" message.

Storage seems OK:
zpool status -x: "all pools are healthy"

Only one note: when the system crash, occurs approximately in the night (3 am).

Any idea ?

Very thanks

 

[peterh]

run memtest86 as a starter.

( you have a hardware problem )

 

[dea75]

OK, thanks for the answer.

I've already programmed a full hardware test on Wednesday night.
I'm absolutely perplexed, however, advanced ECC RAM controller report all OK.

When I have news report it immediately.

Very thanks

Luca

 

[William Grzybowski]

This is probably because the system is try to run a scrub operation, and fails badly for the lack of memory...

To confirm, paste: zpool history

 

[dea75]

Unfortunately this problem seems not linked to scrub operation.. (manual scrub pass without errors)

However I will test the memory on Wednesday (ten hours loop is good ?)

********************

zpool history

2012-04-07.02:00:02 zfs snapshot -r dati@backup
2012-04-07.02:00:03 zfs clone dati@backup dati/daily-backup
2012-04-07.07:05:50 zfs destroy dati/daily-backup
2012-04-07.07:05:51 zfs destroy dati@backup
2012-04-07.08:00:04 zfs snapshot -r dati@auto-20120407.0800-1w
2012-04-07.08:00:05 zfs destroy dati@auto-20120331.0800-1w
2012-04-07.09:00:03 zfs snapshot -r dati@auto-20120407.0900-1w
2012-04-07.09:00:04 zfs destroy dati@auto-20120331.0900-1w
2012-04-07.10:00:04 zfs snapshot -r dati@auto-20120407.1000-1w
2012-04-07.10:00:05 zfs destroy dati@auto-20120331.1000-1w
2012-04-07.11:00:04 zfs snapshot -r dati@auto-20120407.1100-1w
2012-04-07.11:00:05 zfs destroy dati@auto-20120331.1100-1w
2012-04-07.12:00:04 zfs snapshot -r dati@auto-20120407.1200-1w
2012-04-07.12:00:06 zfs destroy dati@auto-20120331.1200-1w
2012-04-07.13:00:04 zfs snapshot -r dati@auto-20120407.1300-1w
2012-04-07.13:00:06 zfs destroy dati@auto-20120331.1300-1w
2012-04-07.14:00:03 zfs snapshot -r dati@auto-20120407.1400-1w
2012-04-07.14:00:05 zfs destroy dati@auto-20120331.1400-1w
2012-04-07.15:00:04 zfs snapshot -r dati@auto-20120407.1500-1w
2012-04-07.15:00:05 zfs destroy dati@auto-20120331.1500-1w
2012-04-07.16:00:04 zfs snapshot -r dati@auto-20120407.1600-1w
2012-04-07.16:00:05 zfs destroy dati@auto-20120331.1600-1w
2012-04-07.17:00:03 zfs snapshot -r dati@auto-20120407.1700-1w
2012-04-07.17:00:04 zfs destroy dati@auto-20120331.1700-1w
2012-04-07.18:00:03 zfs snapshot -r dati@auto-20120407.1800-1w
2012-04-07.18:00:04 zfs destroy dati@auto-20120331.1800-1w
2012-04-07.19:00:04 zfs snapshot -r dati@auto-20120407.1900-1w
2012-04-07.19:00:05 zfs destroy dati@auto-20120331.1900-1w
2012-04-08.02:00:01 zfs snapshot -r dati@backup
2012-04-08.02:00:02 zfs clone dati@backup dati/daily-backup
2012-04-08.07:06:00 zfs destroy dati/daily-backup
2012-04-08.07:06:01 zfs destroy dati@backup
2012-04-09.08:00:04 zfs snapshot -r dati@auto-20120409.0800-1w
2012-04-09.08:00:05 zfs destroy dati@auto-20120402.0800-1w
2012-04-09.09:00:04 zfs snapshot -r dati@auto-20120409.0900-1w
2012-04-09.09:00:06 zfs destroy dati@auto-20120402.0900-1w
2012-04-09.10:00:04 zfs snapshot -r dati@auto-20120409.1000-1w
2012-04-09.10:00:05 zfs destroy dati@auto-20120402.1000-1w
2012-04-09.11:00:04 zfs snapshot -r dati@auto-20120409.1100-1w
2012-04-09.11:00:05 zfs destroy dati@auto-20120402.1100-1w
2012-04-09.12:00:03 zfs snapshot -r dati@auto-20120409.1200-1w
2012-04-09.12:00:05 zfs destroy dati@auto-20120402.1200-1w
2012-04-09.13:00:04 zfs snapshot -r dati@auto-20120409.1300-1w
2012-04-09.13:00:05 zfs destroy dati@auto-20120402.1300-1w
2012-04-09.14:00:04 zfs snapshot -r dati@auto-20120409.1400-1w
2012-04-09.14:00:05 zfs destroy dati@auto-20120402.1400-1w
2012-04-09.15:00:04 zfs snapshot -r dati@auto-20120409.1500-1w
2012-04-09.15:00:05 zfs destroy dati@auto-20120402.1500-1w
2012-04-09.16:00:03 zfs snapshot -r dati@auto-20120409.1600-1w
2012-04-09.16:00:05 zfs destroy dati@auto-20120402.1600-1w
2012-04-09.17:00:05 zfs snapshot -r dati@auto-20120409.1700-1w
2012-04-09.17:00:06 zfs destroy dati@auto-20120402.1700-1w
2012-04-09.18:00:04 zfs snapshot -r dati@auto-20120409.1800-1w
2012-04-09.18:00:05 zfs destroy dati@auto-20120402.1800-1w
2012-04-09.19:00:03 zfs snapshot -r dati@auto-20120409.1900-1w
2012-04-09.19:00:04 zfs destroy dati@auto-20120402.1900-1w
2012-04-10.02:00:02 zfs snapshot -r dati@backup
2012-04-10.02:00:02 zfs clone dati@backup dati/daily-backup
2012-04-10.07:05:54 zfs destroy dati/daily-backup
2012-04-10.07:05:55 zfs destroy dati@backup
2012-04-10.08:00:03 zfs snapshot -r dati@auto-20120410.0800-1w
2012-04-10.08:00:05 zfs destroy dati@auto-20120403.0800-1w
2012-04-10.09:00:03 zfs snapshot -r dati@auto-20120410.0900-1w
2012-04-10.09:00:05 zfs destroy dati@auto-20120403.0900-1w
2012-04-10.10:00:04 zfs snapshot -r dati@auto-20120410.1000-1w
2012-04-10.10:00:05 zfs destroy dati@auto-20120403.1000-1w
2012-04-10.11:00:05 zfs snapshot -r dati@auto-20120410.1100-1w
2012-04-10.11:00:06 zfs destroy dati@auto-20120403.1100-1w
2012-04-10.12:00:05 zfs snapshot -r dati@auto-20120410.1200-1w
2012-04-10.12:00:08 zfs destroy dati@auto-20120403.1200-1w
2012-04-10.13:00:05 zfs snapshot -r dati@auto-20120410.1300-1w
2012-04-10.13:00:07 zfs destroy dati@auto-20120403.1300-1w
2012-04-10.14:00:04 zfs snapshot -r dati@auto-20120410.1400-1w
2012-04-10.14:00:05 zfs destroy dati@auto-20120403.1400-1w
2012-04-10.15:00:04 zfs snapshot -r dati@auto-20120410.1500-1w
2012-04-10.15:00:06 zfs destroy dati@auto-20120403.1500-1w
2012-04-10.16:00:04 zfs snapshot -r dati@auto-20120410.1600-1w
2012-04-10.16:00:06 zfs destroy dati@auto-20120403.1600-1w
2012-04-10.17:00:04 zfs snapshot -r dati@auto-20120410.1700-1w
2012-04-10.17:00:05 zfs destroy dati@auto-20120403.1700-1w
2012-04-10.18:00:04 zfs snapshot -r dati@auto-20120410.1800-1w
2012-04-10.18:00:05 zfs destroy dati@auto-20120403.1800-1w
2012-04-10.19:00:04 zfs snapshot -r dati@auto-20120410.1900-1w
2012-04-10.19:00:05 zfs destroy dati@auto-20120403.1900-1w
2012-04-11.02:00:01 zfs snapshot -r dati@backup
2012-04-11.02:00:02 zfs clone dati@backup dati/daily-backup
2012-04-11.07:06:14 zfs destroy dati/daily-backup
2012-04-11.07:06:16 zfs destroy dati@backup
2012-04-11.08:00:04 zfs snapshot -r dati@auto-20120411.0800-1w
2012-04-11.08:00:06 zfs destroy dati@auto-20120404.0800-1w
2012-04-11.09:00:05 zfs snapshot -r dati@auto-20120411.0900-1w
2012-04-11.09:00:06 zfs destroy dati@auto-20120404.0900-1w
2012-04-11.10:00:04 zfs snapshot -r dati@auto-20120411.1000-1w
2012-04-11.10:00:05 zfs destroy dati@auto-20120404.1000-1w
2012-04-11.11:00:06 zfs snapshot -r dati@auto-20120411.1100-1w
2012-04-11.11:00:08 zfs destroy dati@auto-20120404.1100-1w
2012-04-11.12:00:06 zfs snapshot -r dati@auto-20120411.1200-1w
2012-04-11.12:00:08 zfs destroy dati@auto-20120404.1200-1w
2012-04-11.13:00:04 zfs snapshot -r dati@auto-20120411.1300-1w
2012-04-11.13:00:05 zfs destroy dati@auto-20120404.1300-1w
2012-04-11.14:00:04 zfs snapshot -r dati@auto-20120411.1400-1w
2012-04-11.14:00:06 zfs destroy dati@auto-20120404.1400-1w
2012-04-11.15:00:07 zfs snapshot -r dati@auto-20120411.1500-1w
2012-04-11.15:00:10 zfs destroy dati@auto-20120404.1500-1w
2012-04-11.16:00:03 zfs snapshot -r dati@auto-20120411.1600-1w
2012-04-11.16:00:06 zfs destroy dati@auto-20120404.1600-1w
2012-04-11.17:00:04 zfs snapshot -r dati@auto-20120411.1700-1w
2012-04-11.17:00:06 zfs destroy dati@auto-20120404.1700-1w
2012-04-11.18:00:04 zfs snapshot -r dati@auto-20120411.1800-1w
2012-04-11.18:00:07 zfs destroy dati@auto-20120404.1800-1w
2012-04-11.19:00:04 zfs snapshot -r dati@auto-20120411.1900-1w
2012-04-11.19:00:05 zfs destroy dati@auto-20120404.1900-1w
2012-04-12.02:00:02 zfs snapshot -r dati@backup
2012-04-12.02:00:03 zfs clone dati@backup dati/daily-backup
2012-04-12.07:05:55 zfs destroy dati/daily-backup
2012-04-12.07:05:57 zfs destroy dati@backup
2012-04-12.08:00:04 zfs snapshot -r dati@auto-20120412.0800-1w
2012-04-12.08:00:05 zfs destroy dati@auto-20120405.0800-1w
2012-04-12.09:00:03 zfs snapshot -r dati@auto-20120412.0900-1w
2012-04-12.09:00:05 zfs destroy dati@auto-20120405.0900-1w
2012-04-12.10:00:05 zfs snapshot -r dati@auto-20120412.1000-1w
2012-04-12.10:00:08 zfs destroy dati@auto-20120405.1000-1w
2012-04-12.11:00:08 zfs snapshot -r dati@auto-20120412.1100-1w
2012-04-12.11:00:11 zfs destroy dati@auto-20120405.1100-1w
2012-04-12.12:00:04 zfs snapshot -r dati@auto-20120412.1200-1w
2012-04-12.12:00:06 zfs destroy dati@auto-20120405.1200-1w
2012-04-12.13:00:06 zfs snapshot -r dati@auto-20120412.1300-1w
2012-04-12.13:00:08 zfs destroy dati@auto-20120405.1300-1w
2012-04-12.13:11:04 zpool import -o cachefile=none -R /mnt -f dati
2012-04-12.13:11:05 zpool set cachefile=/data/zfs/zpool.cache dati
2012-04-12.14:00:08 zfs snapshot -r dati@auto-20120412.1400-1w
2012-04-12.14:00:10 zfs destroy dati@auto-20120405.1400-1w
2012-04-12.15:00:06 zfs snapshot -r dati@auto-20120412.1500-1w
2012-04-12.15:00:09 zfs destroy dati@auto-20120405.1500-1w
2012-04-12.16:00:04 zfs snapshot -r dati@auto-20120412.1600-1w
2012-04-12.16:00:08 zfs destroy dati@auto-20120405.1600-1w
2012-04-12.17:00:05 zfs snapshot -r dati@auto-20120412.1700-1w
2012-04-12.17:00:06 zfs destroy dati@auto-20120405.1700-1w
2012-04-12.18:00:05 zfs snapshot -r dati@auto-20120412.1800-1w
2012-04-12.18:00:06 zfs destroy dati@auto-20120405.1800-1w
2012-04-12.19:00:03 zfs snapshot -r dati@auto-20120412.1900-1w
2012-04-12.19:00:04 zfs destroy dati@auto-20120405.1900-1w
2012-04-13.02:00:02 zfs snapshot -r dati@backup
2012-04-13.02:00:03 zfs clone dati@backup dati/daily-backup
2012-04-13.07:06:02 zfs destroy dati/daily-backup
2012-04-13.07:06:03 zfs destroy dati@backup
2012-04-13.08:00:03 zfs snapshot -r dati@auto-20120413.0800-1w
2012-04-13.08:00:04 zfs destroy dati@auto-20120406.0800-1w
2012-04-13.09:00:04 zfs snapshot -r dati@auto-20120413.0900-1w
2012-04-13.09:00:05 zfs destroy dati@auto-20120406.0900-1w
2012-04-13.10:00:04 zfs snapshot -r dati@auto-20120413.1000-1w
2012-04-13.10:00:06 zfs destroy dati@auto-20120406.1000-1w
2012-04-13.11:00:04 zfs snapshot -r dati@auto-20120413.1100-1w
2012-04-13.11:00:06 zfs destroy dati@auto-20120406.1100-1w
2012-04-13.12:00:15 zfs snapshot -r dati@auto-20120413.1200-1w
2012-04-13.12:00:17 zfs destroy dati@auto-20120406.1200-1w
2012-04-13.13:00:06 zfs snapshot -r dati@auto-20120413.1300-1w
2012-04-13.13:00:08 zfs destroy dati@auto-20120406.1300-1w
2012-04-13.14:00:03 zfs snapshot -r dati@auto-20120413.1400-1w
2012-04-13.14:00:05 zfs destroy dati@auto-20120406.1400-1w
2012-04-13.15:00:04 zfs snapshot -r dati@auto-20120413.1500-1w
2012-04-13.15:00:05 zfs destroy dati@auto-20120406.1500-1w
2012-04-13.16:00:04 zfs snapshot -r dati@auto-20120413.1600-1w
2012-04-13.16:00:06 zfs destroy dati@auto-20120406.1600-1w
2012-04-13.17:00:04 zfs snapshot -r dati@auto-20120413.1700-1w
2012-04-13.17:00:06 zfs destroy dati@auto-20120406.1700-1w
2012-04-13.18:00:03 zfs snapshot -r dati@auto-20120413.1800-1w
2012-04-13.18:00:04 zfs destroy dati@auto-20120406.1800-1w
2012-04-13.19:00:04 zfs snapshot -r dati@auto-20120413.1900-1w
2012-04-13.19:00:05 zfs destroy dati@auto-20120406.1900-1w
2012-04-14.02:00:01 zfs snapshot -r dati@backup
2012-04-14.02:00:02 zfs clone dati@backup dati/daily-backup
2012-04-16.08:03:07 zpool import -o cachefile=none -R /mnt -f dati
2012-04-16.08:03:08 zpool set cachefile=/data/zfs/zpool.cache dati
2012-04-16.08:04:03 zfs snapshot -r dati@auto-20120416.0804-1w
2012-04-16.08:04:04 zfs destroy dati@auto-20120407.0900-1w
2012-04-16.08:04:04 zfs destroy dati@auto-20120409.0800-1w
2012-04-16.08:04:05 zfs destroy dati@auto-20120407.1500-1w
2012-04-16.08:04:06 zfs destroy dati@auto-20120407.1000-1w
2012-04-16.08:04:07 zfs destroy dati@auto-20120407.1700-1w
2012-04-16.08:04:08 zfs destroy dati@auto-20120407.1600-1w
2012-04-16.08:04:08 zfs destroy dati@auto-20120407.1800-1w
2012-04-16.08:04:09 zfs destroy dati@auto-20120407.1100-1w
2012-04-16.08:04:10 zfs destroy dati@auto-20120407.1300-1w
2012-04-16.08:04:11 zfs destroy dati@auto-20120407.1200-1w
2012-04-16.08:04:12 zfs destroy dati@auto-20120407.1900-1w
2012-04-16.08:04:13 zfs destroy dati@auto-20120407.1400-1w
2012-04-16.08:04:14 zfs destroy dati@auto-20120407.0800-1w
2012-04-16.08:06:13 zfs destroy -r dati/daily-backup
2012-04-16.08:09:53 zfs destroy dati@backup
2012-04-16.09:00:04 zfs destroy dati@auto-20120409.0900-1w
2012-04-16.09:04:03 zfs snapshot -r dati@auto-20120416.0904-1w
2012-04-16.10:00:05 zfs destroy dati@auto-20120409.1000-1w
2012-04-16.10:04:02 zfs snapshot -r dati@auto-20120416.1004-1w
2012-04-16.11:00:05 zfs destroy dati@auto-20120409.1100-1w
2012-04-16.11:04:04 zfs snapshot -r dati@auto-20120416.1104-1w
2012-04-16.12:00:04 zfs destroy dati@auto-20120409.1200-1w
2012-04-16.12:04:05 zfs snapshot -r dati@auto-20120416.1204-1w
2012-04-16.13:00:04 zfs destroy dati@auto-20120409.1300-1w
2012-04-16.13:04:03 zfs snapshot -r dati@auto-20120416.1304-1w
2012-04-16.14:00:07 zfs destroy dati@auto-20120409.1400-1w
2012-04-16.14:04:05 zfs snapshot -r dati@auto-20120416.1404-1w


********************

 

[dea75]

.. is possible, if I create ZFS snapshot, than make a backup from this (heavy load from snapshot) generate an "out of memory" condition, than crash ?
This would explain everything, starting from the time of crash ...

 

[dea75]

UPDATE:

14 hours testing: hardware all ok.

Only one possibility: under stress and low memory condition Freenas crash, is possible ? Now I try to increase memory...
(7 Gbyte ECC RAM are now installed).

Thanks

Luca