FreeNAS 11.1 network interface crashed

[mosh]

Hi,
Yesterday night my FreeNAS server became unreachable, it didn't answer to ping.
checking the server via IPMI i found that it was up and running, and only the networking was down.
I rebooted the server and it came up fine.

Checking /var/log/messages i found the following, However, It doesn't say much to me. (bxe0 is the NIC)
does anyone met this kind of behavior? Any recommendations? fixes?

OS: FreeNAS-11.1-U4

Thanks!

Code:

Nov 12 21:17:12 strg-shared bxe0: ERROR: MC assert!
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_LIST_INDEX 0x1a
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x0 = 0x81ec67c5 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x1 = 0x81fba571 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x2 = 0x820ae338 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x3 = 0x821a20a3 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x4 = 0x82295e5d 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x5 = 0x82389c02 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x6 = 0x8247d9c0 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x7 = 0x82571726 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x8 = 0x826654bc 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x9 = 0x82759263 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xa = 0x8284d03c 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xb = 0x82940d9f 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xc = 0x00000001 0x05040000 0x00000800 0x0001004f
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xd = 0x7fb86870 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xe = 0x7fc7a600 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xf = 0x7fd6e3a9 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x10 = 0x7fe6213a 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x11 = 0x7ff55f0b 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x12 = 0x80049c82 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x13 = 0x8013da30 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x14 = 0x802317eb 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x15 = 0x8032555b 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x16 = 0x804192f7 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x17 = 0x8050d0a4 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x18 = 0x80600e62 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x19 = 0x806f4bdf 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x1a = 0x807e8980 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x1b = 0x808dc740 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x1c = 0x809d1c33 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x1d = 0x80ac59db 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x1e = 0x80bb976b 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x1f = 0x80cad501 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x20 = 0x80da12af 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x21 = 0x80e95043 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x22 = 0x80f88dd2 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x23 = 0x8107cb93 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x24 = 0x8117091d 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x25 = 0x812646bd 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x26 = 0x81358451 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x27 = 0x8144c1f9 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x28 = 0x8153ff97 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x29 = 0x81633d61 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x2a = 0x81727ace 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x2b = 0x8181b888 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x2c = 0x8190f636 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x2d = 0x81a033d8 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x2e = 0x81af714e 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x2f = 0x81beaf04 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x30 = 0x81cdecac 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x31 = 0x81dd2a4a 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_LIST_INDEX 0x18
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x0 = 0x81ec67cd 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x1 = 0x81fba577 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x2 = 0x820ae33d 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x3 = 0x821a20a8 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x4 = 0x82295e62 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x5 = 0x82389c08 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x6 = 0x8247d9c5 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x7 = 0x8257172c 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x8 = 0x826654c1 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x9 = 0x82759269 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0xa = 0x8284d042 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0xb = 0x82940da4 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0xc = 0x7fa92ac4 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0xd = 0x7fb86877 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0xe = 0x7fc7a604 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0xf = 0x7fd6e3b0 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x10 = 0x7fe6213e 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x11 = 0x7ff55f0e 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x12 = 0x80049c87 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x13 = 0x8013da35 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x14 = 0x802317ef 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x15 = 0x8032555e 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x16 = 0x804192fa 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x17 = 0x8050d0a8 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x18 = 0x80600e66 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x19 = 0x806f4be3 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x1a = 0x807e8986 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x1b = 0x808dc745 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x1c = 0x809d1c38 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x1d = 0x80ac59e1 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x1e = 0x80bb9772 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x1f = 0x80cad506 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x20 = 0x80da12b4 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x21 = 0x80e95048 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x22 = 0x80f88dd6 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x23 = 0x8107cb9a 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x24 = 0x81170923 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x25 = 0x812646c3 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x26 = 0x81358455 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x27 = 0x8144c1fd 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x28 = 0x8153ff9b 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x29 = 0x81633d66 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x2a = 0x81727ad2 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x2b = 0x8181b88e 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x2c = 0x8190f63b 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x2d = 0x81a033db 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x2e = 0x81af7154 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x2f = 0x81beaf0a 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x30 = 0x81cdecb0 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x31 = 0x81dd2a51 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_LIST_INDEX 0x18
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x0 = 0x81ec67d4 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x1 = 0x81fba57e 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x2 = 0x820ae345 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x3 = 0x821a20b0 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x4 = 0x82295e69 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x5 = 0x82389c10 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x6 = 0x8247d9cd 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x7 = 0x82571734 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x8 = 0x826654c8 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x9 = 0x82759272 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0xa = 0x8284d049 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0xb = 0x82940dab 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0xc = 0x7fa92acb 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0xd = 0x7fb8687e 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0xe = 0x7fc7a60b 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0xf = 0x7fd6e3b8 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x10 = 0x7fe62145 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x11 = 0x7ff55f15 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x12 = 0x80049c8f 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x13 = 0x8013da3c 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x14 = 0x802317f6 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x15 = 0x80325566 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x16 = 0x80419301 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x17 = 0x8050d0ae 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x18 = 0x80600e6d 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x19 = 0x806f4bea 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x1a = 0x807e898e 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x1b = 0x808dc74c 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x1c = 0x809d1c3f 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x1d = 0x80ac59e8 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x1e = 0x80bb977a 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x1f = 0x80cad50e 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x20 = 0x80da12bb 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x21 = 0x80e95050 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x22 = 0x80f88ddd 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x23 = 0x8107cba1 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x24 = 0x8117092b 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x25 = 0x812646ca 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x26 = 0x8135845c 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x27 = 0x8144c205 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x28 = 0x8153ffa3 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x29 = 0x81633d6e 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x2a = 0x81727ada 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x2b = 0x8181b895 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x2c = 0x8190f643 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x2d = 0x81a033e3 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x2e = 0x81af715b 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x2f = 0x81beaf13 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x30 = 0x81cdecb8 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: CSTORM_ASSERT_INDEX 0x31 = 0x81dd2a59 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_LIST_INDEX 0x18
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x0 = 0x81ec67d0 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x1 = 0x81fba57b 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x2 = 0x820ae341 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x3 = 0x821a20ac 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x4 = 0x82295e66 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x5 = 0x82389c0c 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x6 = 0x8247d9c9 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x7 = 0x82571730 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x8 = 0x826654c4 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x9 = 0x8275926d 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0xa = 0x8284d045 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0xb = 0x82940da8 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0xc = 0x7fa92ac7 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0xd = 0x7fb8687a 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0xe = 0x7fc7a607 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0xf = 0x7fd6e3b4 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x10 = 0x7fe62141 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x11 = 0x7ff55f12 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x12 = 0x80049c8b 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x13 = 0x8013da38 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x14 = 0x802317f3 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x15 = 0x80325562 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x16 = 0x804192fe 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x17 = 0x8050d0ab 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x18 = 0x80600e69 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x19 = 0x806f4be6 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x1a = 0x807e8989 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x1b = 0x808dc749 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x1c = 0x809d1c3b 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x1d = 0x80ac59e4 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x1e = 0x80bb9776 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x1f = 0x80cad50a 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x20 = 0x80da12b7 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x21 = 0x80e9504c 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x22 = 0x80f88dd9 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x23 = 0x8107cb9d 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x24 = 0x81170927 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x25 = 0x812646c6 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x26 = 0x81358459 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x27 = 0x8144c200 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x28 = 0x8153ffa0 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x29 = 0x81633d6a 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x2a = 0x81727ad5 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x2b = 0x8181b892 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x2c = 0x8190f63f 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x2d = 0x81a033df 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x2e = 0x81af7157 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x2f = 0x81beaf0e 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x30 = 0x81cdecb4 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: ERROR: USTORM_ASSERT_INDEX 0x31 = 0x81dd2a54 0x00000008 0x06000000 0x000200ab
Nov 12 21:17:12 strg-shared bxe0: bxe_attn_int_deasserted3 (/freenas-11-releng/freenas/_BE/os/sys/dev/bxe/bxe.c,7999)
Nov 12 21:17:20 strg-shared bxe0: ERROR: ECORE: timeout waiting for state 0
Nov 12 21:17:20 strg-shared bxe0: ERROR: Failed to delete MACs (-4) mac_type 0 wait_for_comp 0x1
Nov 12 21:17:20 strg-shared bxe0: ERROR: Failed to delete UC MACs list (-4)
Nov 12 21:17:25 strg-shared bxe0: ERROR: ECORE: timeout waiting for state 7
Nov 12 21:17:30 strg-shared bxe0: ERROR: ECORE: timeout waiting for state 2
Nov 12 21:17:30 strg-shared bxe0: ERROR: FUNC_STOP ramrod failed. Running a dry transaction (-4)
Nov 12 21:17:30 strg-shared bxe0: ERROR: failed to release tx mbufs (353 left)
Nov 12 21:17:30 strg-shared bxe0: ERROR: failed to release tx mbufs (480 left)
Nov 12 21:17:30 strg-shared bxe0: ERROR: failed to release tx mbufs (31 left)
Nov 12 21:17:30 strg-shared bxe0: ERROR: failed to release tx mbufs (732 left)
Nov 12 21:17:34 strg-shared bxe0: ERROR: TX watchdog timeout on fp[00], resetting!
Nov 12 21:17:34 strg-shared bxe0: ERROR: failed to release tx mbufs (353 left)
Nov 12 21:17:34 strg-shared bxe0: ERROR: failed to release tx mbufs (480 left)
Nov 12 21:17:34 strg-shared bxe0: ERROR: failed to release tx mbufs (31 left)
Nov 12 21:17:34 strg-shared bxe0: ERROR: failed to release tx mbufs (732 left)
Nov 12 21:17:36 strg-shared bxe0: ERROR: TX watchdog timeout on fp[01], resetting!
Nov 12 21:17:36 strg-shared bxe0: ERROR: failed to release tx mbufs (353 left)
Nov 12 21:17:36 strg-shared bxe0: ERROR: failed to release tx mbufs (480 left)
Nov 12 21:17:36 strg-shared bxe0: ERROR: failed to release tx mbufs (31 left)
Nov 12 21:17:36 strg-shared bxe0: ERROR: failed to release tx mbufs (732 left)
Nov 12 21:17:38 strg-shared bxe0: ERROR: TX watchdog timeout on fp[02], resetting!
Nov 12 21:17:38 strg-shared bxe0: ERROR: failed to release tx mbufs (353 left)
Nov 12 21:17:38 strg-shared bxe0: ERROR: failed to release tx mbufs (480 left)
Nov 12 21:17:39 strg-shared bxe0: ERROR: failed to release tx mbufs (31 left)
Nov 12 21:17:39 strg-shared bxe0: ERROR: failed to release tx mbufs (732 left)
Nov 12 21:17:40 strg-shared bxe0: ERROR: TX watchdog timeout on fp[03], resetting!
Nov 12 21:17:41 strg-shared bxe0: ERROR: failed to release tx mbufs (353 left)
Nov 12 21:17:41 strg-shared bxe0: ERROR: failed to release tx mbufs (480 left)
Nov 12 21:17:41 strg-shared bxe0: ERROR: failed to release tx mbufs (31 left)
Nov 12 21:17:41 strg-shared bxe0: ERROR: failed to release tx mbufs (732 left)
Nov 12 21:17:45 strg-shared bxe0: link_vars phy_flags : 0
Nov 12 21:17:45 strg-shared kernel: bxe0: link state changed to DOWN
Nov 12 21:17:45 strg-shared kernel: bxe0: link state changed to DOWN
Nov 12 21:17:45 strg-shared bxe0: link_vars phy_flags : 1
Nov 12 21:17:45 strg-shared bxe0: 
Nov 12 21:17:45 strg-shared kernel: bxe0: link state changed to UP
Nov 12 21:17:45 strg-shared kernel: bxe0: link state changed to UP
Nov 12 21:17:45 strg-shared NIC Link is Up, 10000 Mbps full duplex, Flow control: none
Nov 12 21:18:34 strg-shared bxe0: ERROR: ECORE: timeout waiting for state 0
Nov 12 21:18:34 strg-shared bxe0: ERROR: Failed to delete MACs (-4) mac_type 0 wait_for_comp 0x1
Nov 12 21:18:34 strg-shared bxe0: ERROR: Failed to delete UC MACs list (-4)
Nov 12 21:18:39 strg-shared bxe0: ERROR: ECORE: timeout waiting for state 7
Nov 12 21:18:44 strg-shared bxe0: ERROR: ECORE: timeout waiting for state 2
Nov 12 21:18:44 strg-shared bxe0: ERROR: FUNC_STOP ramrod failed. Running a dry transaction (-4)
Nov 12 21:18:45 strg-shared bxe0: ERROR: failed to release tx mbufs (353 left)
Nov 12 21:18:45 strg-shared bxe0: ERROR: failed to release tx mbufs (480 left)
Nov 12 21:18:45 strg-shared bxe0: ERROR: failed to release tx mbufs (31 left)
Nov 12 21:18:45 strg-shared bxe0: ERROR: failed to release tx mbufs (732 left)
Nov 12 21:18:50 strg-shared bxe0: ERROR: ECORE: timeout waiting for state 1
Nov 12 21:18:50 strg-shared bxe0: ERROR: Queue(0) SETUP failed (rc = -4)
Nov 12 21:18:50 strg-shared bxe0: ERROR: Setup leading failed! rc = -4
Nov 12 21:18:50 strg-shared bxe0: ERROR: failed to release tx mbufs (353 left)
Nov 12 21:18:50 strg-shared bxe0: ERROR: failed to release tx mbufs (480 left)
Nov 12 21:18:50 strg-shared bxe0: ERROR: failed to release tx mbufs (31 left)
Nov 12 21:18:50 strg-shared bxe0: ERROR: failed to release tx mbufs (732 left)
Nov 12 21:18:50 strg-shared bxe0: ERROR: Initialization failed, stack notified driver is NOT running!

 

[kdragon75]

I would have to Google the error...

 

[kdragon75]

Hmm all I can find is the driver source... The comments are not helping me.

 

[mosh]

Hmm all I can find is the driver source... The comments are not helping me.

Thanks for checking out my topic.
I google'd it and nothing came up, I suspect its might be a NIC driver issue..
what do you mean the comments are not helping?

I'm also thinking about upgrade FreeNAS to U6 as a last resort. hoping that might solve the issue

- Moshe

 

[Chris Moore]

OS: FreeNAS-11.1-U4

I would suggest an upgrade to FreeNAS-11.1-U6 as there were some fixes and keep an eye on the situation.

The possibility that came to mind first was that the boot device might be failing.

Can you give us a rundown on your hardware using the following guide:
https://forums.freenas.org/index.php?threads/updated-forum-rules-8-3-16.45124/

 

[kdragon75]

what do you mean the comments are not helping?

The comment (or lack of) in the source code. I'm not smart enough to just read the code ;)

 

[Elliot Dierksen]

Please post the details of your hardware, and whether it is a bare metal or virtualized server. A quick google search on that error message turned up some errors like that under ESXi with a heavy load. Is there anything odd happening on your network?

 

[mosh]

Hi,
Below are the Hardware details

PowerEdge R630
2 x Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
RAM Memory 262021MB
2x SSD 120GB for OS
5x INTEL DC Grade SSD 1.6TB
Intel Ethernet Controller 10-Gigabit X540-AT2
Avago Technologies (LSI) SAS2008 Firmware: 20.00.07.00, Driver: 21.01.00.00-fbsd

 

[mosh]

This issue reoccured
Here is the log from the storage server

Code:

Dec 14 23:13:12 strg-shared bxe0: ERROR: MC assert!
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_LIST_INDEX 0x36
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x0 = 0xb17c47da 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x1 = 0xb18b858a 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x2 = 0xb19ac2eb 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x3 = 0xb1aa00e0 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x4 = 0xb1b93e7f 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x5 = 0xb1c87c12 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x6 = 0xb1d7b961 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x7 = 0xb1e6f6ff 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x8 = 0xb1f634a1 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x9 = 0xb205724e 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xa = 0xb214afdf 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xb = 0xb223ed96 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xc = 0xb2332b20 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xd = 0xb24268b6 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xe = 0xb251a6e9 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0xf = 0xb260e401 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x10 = 0xb2702193 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x11 = 0xb27f5f41 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x12 = 0xb28e9cec 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x13 = 0xb29ddaf9 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x14 = 0xb2ad183f 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:12 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x15 = 0xb2bc55b2 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:13 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x16 = 0xb2cb937d 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:13 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x17 = 0xb2dad13a 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:13 strg-shared bxe0: ERROR: XSTORM_ASSERT_INDEX 0x18 = 0xb2ea0e89 0x00000008 0x06000000 0x000200ab

Dec 14 23:13:13 strg-shared bxe0: ERROR: TSTORM_ASSERT_LIST_INDEX 0x34
Dec 14 23:13:14 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x0 = 0xb17c47e1 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:14 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x1 = 0xb18b858f 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:14 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x2 = 0xb19ac2ef 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:14 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x3 = 0xb1aa00e6 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:14 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x4 = 0xb1b93e84 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:14 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x5 = 0xb1c87c18 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:14 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x6 = 0xb1d7b964 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:14 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x7 = 0xb1e6f703 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:14 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x8 = 0xb1f634a5 0x00000008 0x06000000 0x000200ab
Dec 14 23:13:14 strg-shared bxe0: ERROR: TSTORM_ASSERT_INDEX 0x9 = 0xb2057251 0x00000008 0x06000000 0x000200ab

Dec 14 23:13:18 strg-shared bxe0: bxe_attn_int_deasserted3 (/freenas-11-releng/freenas/_BE/os/sys/dev/bxe/bxe.c,7999)
Dec 14 23:13:21 strg-shared bxe0: ERROR: ECORE: timeout waiting for state 0
Dec 14 23:13:21 strg-shared bxe0: ERROR: Failed to delete MACs (-4) mac_type 0 wait_for_comp 0x1
Dec 14 23:13:21 strg-shared bxe0: ERROR: Failed to delete UC MACs list (-4)
Dec 14 23:13:26 strg-shared bxe0: ERROR: ECORE: timeout waiting for state 7
Dec 14 23:13:31 strg-shared bxe0: ERROR: ECORE: timeout waiting for state 2
Dec 14 23:13:31 strg-shared bxe0: ERROR: FUNC_STOP ramrod failed. Running a dry transaction (-4)
Dec 14 23:13:31 strg-shared bxe0: ERROR: failed to release tx mbufs (872 left)

Dec 14 23:14:53 strg-shared bxe0: bxe_attn_int_deasserted2 (/freenas-11-releng/freenas/_BE/os/sys/dev/bxe/bxe.c,8113)
Dec 14 23:14:53 strg-shared bxe0: ERROR: PXP2_REG_PXP2_INT_STS_CLR_0 0x00000080
Dec 14 23:14:53 strg-shared bxe0: ERROR: PXP2_WR_PGLUE_EOP_ERROR
Dec 14 23:14:53 strg-shared bxe0: ERROR: FATAL HW block attention set2 0x20
Dec 14 23:14:53 strg-shared bxe0: bxe_attn_int_deasserted2 (/freenas-11-releng/freenas/_BE/os/sys/dev/bxe/bxe.c,8113)
Dec 14 23:14:53 strg-shared bxe0: ERROR: ECORE: timeout waiting for state 1
Dec 14 23:14:53 strg-shared bxe0: bxe0: ERROR: PXP2_REG_PXP2_INT_STS_CLR_0 0x00000080
Dec 14 23:14:53 strg-shared bxe0: ERROR: PXP2_WR_PGLUE_EOP_ERROR
Dec 14 23:14:53 strg-shared bxe0: ERROR: FATAL HW block attention set2 0x20
Dec 14 23:14:53 strg-shared bxe0: bxe_attn_int_deasserted2 (/freenas-11-releng/freenas/_BE/os/sys/dev/bxe/bxe.c,8113)
Dec 14 23:14:53 strg-shared ERROR: Queue(0) SETUP failed (rc = -4)
Dec 14 23:14:53 strg-shared bxe0: bxe0: ERROR: Setup leading failed! rc = -4
Dec 14 23:14:53 strg-shared ERROR: PXP2_REG_PXP2_INT_STS_CLR_0 0x00000080
Dec 14 23:14:53 strg-shared bxe0: ERROR: PXP2_WR_PGLUE_EOP_ERROR
Dec 14 23:14:53 strg-shared bxe0: ERROR: FATAL HW block attention set2 0x20
Dec 14 23:14:53 strg-shared bxe0: bxe_attn_int_deasserted2 (/freenas-11-releng/freenas/_BE/os/sys/dev/bxe/bxe.c,8113)
Dec 14 23:14:53 strg-shared bxe0: ERROR: failed to release tx mbufs (872 left)
Dec 14 23:14:53 strg-shared bxe0: ERROR: failed to release tx mbufs (7 left)
Dec 14 23:14:53 strg-shared bxe0: ERROR: failed to release tx mbufs (410 left)
Dec 14 23:14:53 strg-shared bxe0: ERROR: failed to release tx mbufs (646 left)
Dec 14 23:14:53 strg-shared bxe0: ERROR: Initialization failed, stack notified driver is NOT running!

 

[Elliot Dierksen]

Is this occurring during periods of heavy traffic? A quick google search for XSTORM turned up this link. https://anthonyspiteri.net/heads-up-heavy-vxlan-traffic-causing-broadcom-10gb-nics-to-drop/. I don't know if you can pass the "enable_vxlan_ofld=0" option it references in the link to yours using Options: section, but you could try that. Honestly I think my suggestion would be to get a used Chelsio NIC off eBay.

 

[kdragon75]

What is the exact broadcom chip in the NIC? Perhaps we can use the bce driver instead?

 

[mosh]

What is the exact broadcom chip in the NIC? Perhaps we can use the bce driver instead?

Hi
Look here,

Code:

bxe0: <QLogic NetXtreme II BCM57810 10GbE (B0) BXE v:1.78.90
bxe0: PCI BAR0 [10] memory allocated: 0xc9000000-0xc97fffff (8388608) -> 0xfffff800c9000000
bxe0: PCI BAR2 [18] memory allocated: 0xc9800000-0xc9ffffff (8388608) -> 0xfffff800c9800000
bxe0: PCI BAR4 [20] memory allocated: 0xca010000-0xca01ffff (65536) -> 0xfffff800ca010000
bxe0: Found 10GBase-T media.
bxe0: IFMEDIA flags : 20
bxe0: Using defaults for TSO: 65518/35/2048

 

[kdragon75]

Yeah, no luck. The bce(4) only covers 1g.

 

[mosh]

Is this occurring during periods of heavy traffic? A quick google search for XSTORM turned up this link. https://anthonyspiteri.net/heads-up-heavy-vxlan-traffic-causing-broadcom-10gb-nics-to-drop/. I don't know if you can pass the "enable_vxlan_ofld=0" option it references in the link to yours using Options: section, but you could try that. Honestly I think my suggestion would be to get a used Chelsio NIC off eBay.

Hi,
No, I checked the monitors and this is not happening during heavy traffic

- Moshe

 

[mosh]

Hi,
No, I checked the monitors and this is not happening during heavy traffic

- Moshe

I tend to agree with you that replacing the NIC could solve this issue for good.
I saw that you recommended Chelsio NIC.
Why Chelsio? Which model do you recommend for 10G?

- Moshe

 

[Elliot Dierksen]

Why Chelsio? Which model do you recommend for 10G?

The driver support for Chelsio in FreeBSD is very good, and that is what TrueNAS (pay version of FreeNAS) and NetApp both use in their hardware. Almost anything is good. The T3 cards are kind of old, but they are still supported in FreeBSD and are very cheap on eBay. I think VMware and some other have dropped driver support for T3 and T4's. T5's work great. I have them in both my FreeNAS boxes.