Network Services Drop

Status
Not open for further replies.

Jason Brunk

Dabbler
Joined
Jan 1, 2016
Messages
28
Good afternoon All,
I have been having an issue recently and I thought this last round of update would fix it.

Periodically all networking services on my freenas stop. They start back up after a bit. But it drops EVERYTHING. Pings stop, iscsi from my esxi box drops (crashing my vms in the process), smb shares drop from every device on the network, nfs drops from a secondary esxi box.

But a few minutes later, all is happy again.

This is NOT a supermicro board as obviously recommended, but the nic's I am using are both intel nic's on board. I started with the one, saw this problem happening so I switched to the other nic. Same issue. I am considering putting a pci intel pro nic in and see if that does anything, but I am not hopeful as it seems like its the networking services failing. It is still an option, just kind of bummed I can't use the 2 onboard gig nic's.

Anyone run in to this? any thoughts?


Thanks!
 

Jason Brunk

Dabbler
Joined
Jan 1, 2016
Messages
28
dmesg

[root@skooge] ~# dmesg
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.126 (iqn.1998-01.com.vmware:567a59e4-541c-e991-603a-00e081ef6aea-1c6d65d9): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.12 (iqn.1998-01.com.vmware:55974bbb-9ddd-5751-43da-001cc42ea440-4a2086f0): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 192.168.2.14 (iqn.1991-05.com.microsoft:dib): no ping reply (NOP-Out) after 5 seconds; dropping connection




Specs:
9 drives (3 raidz in 1 zfs pool)
1 120gig ssd cache
ASRock Q87WS-DL ATX Server Motherboard
Intel Celeron G1820 Haswell Dual-Core 2.7 GHz
8 Gig ECC memory
 
D

dlavigne

Guest
Please also post a debug (you can create one in System -> Advanced -> Save Debug.
 

Jason Brunk

Dabbler
Joined
Jan 1, 2016
Messages
28
Yes, its setup as a cache and my system shows 109G l2arc.

I can remove it. Just curious how much memory I need to be able to use that. Any info on why that would cause a problem? Or recommendations on if there is a different better use for that SSD?
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
Just curious how much memory I need to be able to use that.
There's no fixed number, but 8GB is definitely not enough. The simplest rule of thumb is to max out your RAM before considering an L2ARC.

When you install an L2ARC, memory is consumed to track its contents. In systems with minimal RAM (yours has the minimum), that RAM would be better used for ARC.
oh, and why would that cause network drop? :)
I'm speculating that your system is choking due to lack of RAM - perhaps some kind of thrashing behavior.
 

Jason Brunk

Dabbler
Joined
Jan 1, 2016
Messages
28
Seems easy enough to try. Any thing I can use that SSD for elsewhere in freenas? Or would i be better off putting it in my esxi box for now?
 

Jason Brunk

Dabbler
Joined
Jan 1, 2016
Messages
28
roger that. I will try removing the cache ssd and see if it continues. Thanks for the help! :)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Yes, its setup as a cache and my system shows 109G l2arc.

I can remove it. Just curious how much memory I need to be able to use that. Any info on why that would cause a problem? Or recommendations on if there is a different better use for that SSD?

You'd probably want to start off with the L2ARC at around 64GB or 128GB of RAM.

Read all of the first post at https://forums.freenas.org/index.ph...res-more-resources-for-the-same-result.28178/ to understand why.

All your setup is doing is draining the ARC for header entries for the L2ARC, which further cripples its already nearly nonexistent ability to identify useful blocks for the L2ARC.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
oh, and why would that cause network drop? :)

It's probably not a network drop so much as it is you're making ZFS have mini strokes and go catatonic trying to handle the demands you are placing on it with a hopelessly tiny amount of resources.
 

Jason Brunk

Dabbler
Joined
Jan 1, 2016
Messages
28
Thank you jgreco. I will definitely review that link. Especially since my esxi environment is relying heavily on iscsi.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
If you're actually relying heavily on iSCSI, you should probably read most of the posts I've posted on these forums that mention iSCSI.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
ISCSI? On 8gb?

Feeling lucky?

I don't know, seems like we've had a small parade of these intriguing tiny-memory-plus-iSCSI deployments in recent weeks. I wonder if I should go back and try to find them all.
 

Jason Brunk

Dabbler
Joined
Jan 1, 2016
Messages
28
I have never had problems till lately with the network dropping. I am guessing I need to go find the recommended specs on build for iscsi use. Granted I only have a couple vm's running but still.

jgreco I did see on your profile you have ALOT of input on the iscsi. I have been reading through some of the threads already :)
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
Status
Not open for further replies.
Top