Network drops after a degraded volume?

[Raymond Matos]

So i been getting emails every 5 mins about a volume being degraded. I havent had a chance to replace it.

More importantly, i get notification that the freenas server network is offline, then 2-3mins later its back. Im guessing its related, but i dont understand why a degraded volume would cause the network to drop in and out. Also why am i getting emails every 5 mins.

here is the dmesg log output

Code:

Build FreeNAS-9.10.1 (d989edd)
Platform Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
Memory 32623MB

log is here

https://gist.github.com/raymatos/c0b40e53560d5336d64cac6cf71b1fcb

 

[SweetAndLow]

I didn't see anything in that log about network going down. What are these network notifications you are getting?

 

[Raymond Matos]

I have a service that pings the websites that i have running on freenas (ie: sabnzbd, couchpotato, etc) and send me a notification when it goes down.

Any reason why i would get an email every 5 mins? the email is from freenas regarding the degraded volume.

 

[SweetAndLow]

You need to give us way more information. What is your pool layout, what is your freenas version, what hardware are you using(motherboard, cpu, ram, disks, controller). I suspect you have something setup wrong.

 

[Raymond Matos]

Here are the machine Specs
Build FreeNAS-9.10.1 (d989edd)
Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
32631MB - ECC ram
SUPERMICRO MBD-X10SAE-O ATX Server Motherboard

M1015 out to an hp sas expander


3 different volumes


pool: ADE01
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 824K in 1h29m with 1 errors on Sat Sep 17 05:02:24 2016
config:

NAME STATE READ WRITE CKSUM
ADE01 ONLINE 0 0 1
raidz2-0 ONLINE 0 0 4
gptid/57a0c0ab-4283-11e4-9185-00259086c202 ONLINE 0 0 0
gptid/584946a3-4283-11e4-9185-00259086c202 ONLINE 0 0 0
gptid/58f42459-4283-11e4-9185-00259086c202 ONLINE 0 0 0
gptid/5995e289-4283-11e4-9185-00259086c202 ONLINE 0 0 0

errors: 1 data errors, use '-v' for a list

pool: backup1
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 2h33m with 13 errors on Sat Sep 17 08:07:31 2016
config:

NAME STATE READ WRITE CKSUM
backup1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/edd5cd38-ba0e-11e3-922a-00259086c202 ONLINE 0 0 0
gptid/efd3abab-ba0e-11e3-922a-00259086c202 ONLINE 0 0 0

errors: 13 data errors, use '-v' for a list

pool: freenas-boot
state: ONLINE
scan: resilvered 541M in 0h8m with 0 errors on Fri Sep 16 19:37:03 2016
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/e9b1dad7-7c27-11e6-9d07-00259086c202 ONLINE 0 0 0
gptid/b9703b63-7c43-11e6-a3b1-00259086c202 ONLINE 0 0 0

errors: No known data errors

pool: jails
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: resilvered 1.35M in 0h0m with 0 errors on Wed Sep 21 14:56:14 2016
config:

NAME STATE READ WRITE CKSUM
jails DEGRADED 0 0 125
mirror-0 DEGRADED 0 0 250
gptid/3c94e433-7c58-11e6-a900-00259086c202 DEGRADED 0 0 250 too many errors
6213643929067291486 UNAVAIL 30 67 89 was /dev/da23p2

errors: 1 data errors, use '-v' for a list

pool: tank1
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: resilvered 36K in 0h0m with 0 errors on Fri Sep 16 18:19:18 2016
config:

NAME STATE READ WRITE CKSUM
tank1 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/6d8f5cca-aa72-11e3-9202-00259086c202 ONLINE 0 0 0
gptid/6e64de8c-aa72-11e3-9202-00259086c202 ONLINE 0 0 0
gptid/6f3b878c-aa72-11e3-9202-00259086c202 ONLINE 0 0 0
gptid/7012ee03-aa72-11e3-9202-00259086c202 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/27782ca0-f039-11e3-89c2-00259086c202 ONLINE 0 0 0
gptid/2857adb5-f039-11e3-89c2-00259086c202 ONLINE 0 0 0
gptid/29353ec9-f039-11e3-89c2-00259086c202 ONLINE 0 0 0
gptid/2a157610-f039-11e3-89c2-00259086c202 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
gptid/5264f6c7-11ea-11e5-96e7-00259086c202 ONLINE 0 0 0
gptid/52d6cf71-11ea-11e5-96e7-00259086c202 ONLINE 0 0 0
gptid/539468f8-11ea-11e5-96e7-00259086c202 ONLINE 0 0 0
gptid/5406480f-11ea-11e5-96e7-00259086c202 ONLINE 0 0 0
raidz2-3 ONLINE 0 0 0
gptid/843b4ccd-7c5c-11e6-8555-00259086c202 ONLINE 0 0 0
gptid/853e00d6-7c5c-11e6-8555-00259086c202 ONLINE 0 0 0
gptid/863ce193-7c5c-11e6-8555-00259086c202 ONLINE 0 0 0
gptid/873b8b35-7c5c-11e6-8555-00259086c202 ONLINE 0 0 0
cache
gptid/5e8ede6e-4527-11e4-9fc5-00259086c202 ONLINE 0 0 0

errors: 3 data errors, use '-v' for a list


 

[SweetAndLow]

Why do all your pools say 'Otherwise restore the entire pool from backup.'? This is usually very bad, at a minimum you have data loss and possibly loose everything in all your pools. Tell us more about your setup you are doing something that you are not telling us about.

 

[Raymond Matos]

This happens only when you have a corrupted file in a pool, you either restore the file from backup or delete it from the pool. After you run a scrub it should go away.

This has been setup and running for over 3 years now with no issue till now.

I pretty much gave you all the information on my setup, not hiding anything.

Here are the machine Specs
Build FreeNAS-9.10.1 (d989edd)
Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
32631MB - ECC ram
SUPERMICRO MBD-X10SAE-O ATX Server Motherboard

M1015 out to an hp sas expander

total of 24 drives. Some are 3tb other are 2tb.

 

[depasseg]

Im guessing its related, but i dont understand why a degraded volume would cause the network to drop in and out.

You are right, it wouldn't'. It's more likely that they have a similar root cause like system overheating or bad psu would be my guess.