Questions on vdev construction after pool re-creation, and performance

Status
Not open for further replies.
Joined
Feb 2, 2016
Messages
574

Can we, just for a second, pause and mourn the passing of these sorts of helpful, useful graphs which are only available in the classic FreeNAS interface?

RIP: meaningful reporting.

Cheers,
Matt
 
Joined
Dec 29, 2014
Messages
1,135
Can we, just for a second, pause and mourn the passing of these sorts of helpful, useful graphs which are only available in the classic FreeNAS interface?
Yikes! I haven't looked at 11.2 yet. Mostly because of a graybeard "Just say no to 0" view. :smile:

I keep running into things that make my head hurt. Check out these graphs as I do a storage Vmotion from one FreeNAS to the other.
Here is the primary which is sending to the secondary.
upload_2018-11-9_15-28-34.png

This is the secondary receiving from the primary.
upload_2018-11-9_15-29-21.png

Umm, what gives? The primary thinks it is sending at around 8G, but the secondary thinks it is receiving at around 5G. This is even borne out by the 10G switch.
LAGG facing primary.
Code:
<HP-5820X-14XG>dis interface Bridge-Aggregation 3
 Bridge-Aggregation3 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bca8
 Description: FreeNAS-G6 2
 20Gbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 PVID: 252
 Port link-type: access
  Tagged   VLAN ID : none
  Untagged VLAN ID : 252
 Last clearing of counters:  Never
 Last 300 seconds input:  703533 packets/sec 1067374567 bytes/sec	43%
 Last 300 seconds output:  51502 packets/sec 5656605 bytes/sec	0%
 Input (total):  37983033556 packets, 53685584417398 bytes
		 37982242793 unicasts, 36584 broadcasts, 743818 multicasts
 Input (normal):  37983023195 packets, - bytes
		 37982242793 unicasts, 36584 broadcasts, 743818 multicasts
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
		 0 CRC, 0 frame, - overruns, 0 aborts
		 - ignored, - parity errors
 Output (total): 18067018507 packets, 21023527476751 bytes
		 17792249715 unicasts, 3089557 broadcasts, 13201255 multicasts, 129239010 pauses
 Output (normal): 17808540527 packets, - bytes
		 17792249715 unicasts, 3089557 broadcasts, 13201255 multicasts, 129239010 pauses
 Output: 0 output errors, - underruns, - buffer failures
		 0 aborts, 0 deferred, 0 collisions, 0 late collisions
		 0 lost carrier, - no carrier


LAGG facing secondary.
Code:
<HP-5820X-14XG>dis interface Bridge-Aggregation 1
 Bridge-Aggregation1 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bcaa
 Description: FreeNAS-G6
 20Gbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 PVID: 252
 Port link-type: access
  Tagged   VLAN ID : none
  Untagged VLAN ID : 252
 Last clearing of counters:  Never
 Last 300 seconds input:  16296 packets/sec 2181418 bytes/sec	0%
 Last 300 seconds output:  293602 packets/sec 436778393 bytes/sec	18%
 Input (total):  15050972756 packets, 18109037742045 bytes
		 15049108198 unicasts, 16023 broadcasts, 181224 multicasts
 Input (normal):  15049305445 packets, - bytes
		 15049108198 unicasts, 16023 broadcasts, 181224 multicasts
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
		 0 CRC, 0 frame, - overruns, 0 aborts
		 - ignored, - parity errors
 Output (total): 29944089642 packets, 41903769448891 bytes
		 29867279196 unicasts, 25236 broadcasts, 3738146 multicasts, 36523842 pauses
 Output (normal): 29871042578 packets, - bytes
		 29867279196 unicasts, 25236 broadcasts, 3738146 multicasts, 36523842 pauses
 Output: 0 output errors, - underruns, - buffer failures
		 0 aborts, 0 deferred, 0 collisions, 0 late collisions

I guess this is more of a VMware question. Why does the storage Vmotion sender have a sending rate almost twice that of the receiver?
 
Joined
Dec 29, 2014
Messages
1,135
I think I have something of an answer. There are a number of 'pause frames' being sent.
Code:
<HP-5820X-14XG>display interface Bridge-Aggregation3
 Bridge-Aggregation3 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bca8
 Description: FreeNAS2
 20Gbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 PVID: 252
 Port link-type: access
  Tagged   VLAN ID : none
  Untagged VLAN ID : 252
 Last clearing of counters:  Never
 Last 300 seconds input:  13948 packets/sec 1436336 bytes/sec	0%
 Last 300 seconds output:  129592 packets/sec 192901360 bytes/sec	8%
 Input (total):  38112222090 packets, 53866276538542 bytes
		 38111441452 unicasts, 36596 broadcasts, 743930 multicasts
 Input (normal):  38112221978 packets, - bytes
		 38111441452 unicasts, 36596 broadcasts, 743930 multicasts
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
		 0 CRC, 0 frame, - overruns, 0 aborts
		 - ignored, - parity errors
 Output (total): 18182773274 packets, 21183582863857 bytes
		 17908002596 unicasts, 3089570 broadcasts, 13203124 multicasts, 129239012 pauses
 Output (normal): 17924295290 packets, - bytes
		 17908002596 unicasts, 3089570 broadcasts, 13203124 multicasts, 129239012 pauses
 Output: 0 output errors, - underruns, - buffer failures
		 0 aborts, 0 deferred, 0 collisions, 0 late collisions
		 0 lost carrier, - no carrier

<HP-5820X-14XG>display interface Bridge-Aggregation1
 Bridge-Aggregation1 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bcaa
 Description: FreeNAS
 20Gbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 PVID: 252
 Port link-type: access
  Tagged   VLAN ID : none
  Untagged VLAN ID : 252
 Last clearing of counters:  Never
 Last 300 seconds input:  311750 packets/sec 473032852 bytes/sec	19%
 Last 300 seconds output:  51160 packets/sec 4413490 bytes/sec	0%
 Input (total):  15286458162 packets, 18461012654191 bytes
		 15284589701 unicasts, 16037 broadcasts, 181336 multicasts
 Input (normal):  15284787074 packets, - bytes
		 15284589701 unicasts, 16037 broadcasts, 181336 multicasts
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
		 0 CRC, 0 frame, - overruns, 0 aborts
		 - ignored, - parity errors
 Output (total): 30054386894 packets, 42018034018751 bytes
		 29961056380 unicasts, 25249 broadcasts, 3740025 multicasts, 44782930 pauses
 Output (normal): 29964821654 packets, - bytes
		 29961056380 unicasts, 25249 broadcasts, 3740025 multicasts, 44782930 pauses
 Output: 0 output errors, - underruns, - buffer failures
		 0 aborts, 0 deferred, 0 collisions, 0 late collisions
		 0 lost carrier, - no carrier

<HP-5820X-14XG>dis interface Ten-GigabitEthernet1/0/11
 Ten-GigabitEthernet1/0/11 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bca8
 Description: FreeNAS2 - P1
 Loopback is not set
 Media type is optical fiber,Port hardware type is 10G_BASE_SR_SFP
 10Gbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 Flow-control is enabled
 The Maximum Frame Length is 10000
 Broadcast MAX-ratio: 100%
 Unicast MAX-ratio: 100%
 Multicast MAX-ratio: 100%
 Allow jumbo frame to pass
 PVID: 252
 Mdi type: auto
 Port link-type: access
  Tagged   VLAN ID : none
  Untagged VLAN ID : 252
 Port priority: 0
 Last clearing of counters:  Never
 Peak value of input: 1067797315 bytes/sec, at 2018-07-06 14:57:20
 Peak value of output: 660706356 bytes/sec, at 2018-11-04 08:54:01
 Last 60 seconds input:  13377 packets/sec 1378730 bytes/sec 0%
 Last 60 seconds output:  2 packets/sec 174 bytes/sec 0%
 Input (total):  26286408207 packets, 37522327854688 bytes
		 26286007187 unicasts, 27466 broadcasts, 373365 multicasts, 0 pauses
 Input (normal):  26286408018 packets, - bytes
		 26286007187 unicasts, 27466 broadcasts, 373365 multicasts, 0 pauses
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
		 0 CRC, 0 frame, - overruns, 0 aborts
		 - ignored, - parity errors
 Output (total): 6672108727 packets, 5437111560168 bytes
		 6602518671 unicasts, 2966113 broadcasts, 11961075 multicasts, 27331434 pauses
 Output (normal): 6617445859 packets, - bytes
		 6602518671 unicasts, 2966113 broadcasts, 11961075 multicasts, 27331434 pauses
 Output: 0 output errors, - underruns, - buffer failures
		 0 aborts, 0 deferred, 0 collisions, 0 late collisions
		 0 lost carrier, - no carrier

<HP-5820X-14XG>dis interface Ten-GigabitEthernet1/0/12
 Ten-GigabitEthernet1/0/12 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bca9
 Description: FreeNAS2 - P2
 Loopback is not set
 Media type is optical fiber,Port hardware type is 10G_BASE_SR_SFP
 10Gbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 Flow-control is enabled
 The Maximum Frame Length is 10000
 Broadcast MAX-ratio: 100%
 Unicast MAX-ratio: 100%
 Multicast MAX-ratio: 100%
 Allow jumbo frame to pass
 PVID: 252
 Mdi type: auto
 Port link-type: access
  Tagged   VLAN ID : none
  Untagged VLAN ID : 252
 Port priority: 0
 Last clearing of counters:  Never
 Peak value of input: 1111514888 bytes/sec, at 2018-10-13 10:30:09
 Peak value of output: 569243839 bytes/sec, at 2018-11-09 14:48:57
 Last 60 seconds input:  0 packets/sec 40 bytes/sec 0%
 Last 60 seconds output:  124607 packets/sec 185479511 bytes/sec 15%
 Input (total):  11826117789 packets, 16343980028720 bytes
		 11825738092 unicasts, 9130 broadcasts, 370567 multicasts, 0 pauses
 Input (normal):  11826117789 packets, - bytes
		 11825738092 unicasts, 9130 broadcasts, 370567 multicasts, 0 pauses
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
		 0 CRC, 0 frame, - overruns, 0 aborts
		 - ignored, - parity errors
 Output (total): 11513907789 packets, 15751298543523 bytes
		 11308727135 unicasts, 123458 broadcasts, 1242076 multicasts, 101907578 pauses
 Output (normal): 11310092669 packets, - bytes
		 11308727135 unicasts, 123458 broadcasts, 1242076 multicasts, 101907578 pauses
 Output: 0 output errors, - underruns, - buffer failures
		 0 aborts, 0 deferred, 0 collisions, 0 late collisions
		 0 lost carrier, - no carrier

<HP-5820X-14XG>dis interface Ten-GigabitEthernet1/0/13
 Ten-GigabitEthernet1/0/13 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bcaa
 Description: FreeNAS - P1
 Loopback is not set
 Media type is optical fiber,Port hardware type is 10G_BASE_SR_SFP
 10Gbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 Flow-control is enabled
 The Maximum Frame Length is 10000
 Broadcast MAX-ratio: 100%
 Unicast MAX-ratio: 100%
 Multicast MAX-ratio: 100%
 Allow jumbo frame to pass
 PVID: 252
 Mdi type: auto
 Port link-type: access
  Tagged   VLAN ID : none
  Untagged VLAN ID : 252
 Port priority: 0
 Last clearing of counters:  Never
 Peak value of input: 369803471 bytes/sec, at 2018-11-02 09:43:40
 Peak value of output: 487844986 bytes/sec, at 2018-07-07 16:14:35
 Last 60 seconds input:  0 packets/sec 71 bytes/sec 0%
 Last 60 seconds output:  3 packets/sec 267 bytes/sec 0%
 Input (total):  846967891 packets, 825862710785 bytes
		 845964657 unicasts, 6602 broadcasts, 90842 multicasts, 905790 pauses
 Input (normal):  846062101 packets, - bytes
		 845964657 unicasts, 6602 broadcasts, 90842 multicasts, 905790 pauses
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
		 0 CRC, 0 frame, - overruns, 0 aborts
		 - ignored, - parity errors
 Output (total): 22223874523 packets, 31368788373114 bytes
		 22192840289 unicasts, 4625 broadcasts, 3178073 multicasts, 13925768 pauses
 Output (normal): 22196022987 packets, - bytes
		 22192840289 unicasts, 4625 broadcasts, 3178073 multicasts, 13925768 pauses
 Output: 0 output errors, - underruns, - buffer failures
		 0 aborts, 0 deferred, 0 collisions, 0 late collisions
		 0 lost carrier, - no carrier

<HP-5820X-14XG>dis interface Ten-GigabitEthernet1/0/14
 Ten-GigabitEthernet1/0/14 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bcab
 Description: FreeNAS - P2
 Loopback is not set
 Media type is optical fiber,Port hardware type is 10G_BASE_SR_SFP
 10Gbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 Flow-control is enabled
 The Maximum Frame Length is 10000
 Broadcast MAX-ratio: 100%
 Unicast MAX-ratio: 100%
 Multicast MAX-ratio: 100%
 Allow jumbo frame to pass
 PVID: 252
 Mdi type: auto
 Port link-type: access
  Tagged   VLAN ID : none
  Untagged VLAN ID : 252
 Port priority: 0
 Last clearing of counters:  Never
 Peak value of input: 1050871408 bytes/sec, at 2018-11-09 14:12:17
 Peak value of output: 591856141 bytes/sec, at 2018-11-09 15:21:27
 Last 60 seconds input:  316684 packets/sec 480523762 bytes/sec 39%
 Last 60 seconds output:  50993 packets/sec 4427540 bytes/sec 0%
 Input (total):  14448754836 packets, 17649207961868 bytes
		 14447889036 unicasts, 9436 broadcasts, 90496 multicasts, 761296 pauses
 Input (normal):  14447988968 packets, - bytes
		 14447889036 unicasts, 9436 broadcasts, 90496 multicasts, 761296 pauses
 Input:  0 input errors, 0 runts, 0 giants, 0 throttles
		 0 CRC, 0 frame, - overruns, 0 aborts
		 - ignored, - parity errors
 Output (total): 7831947009 packets, 10649371113747 bytes
		 7768967085 unicasts, 20624 broadcasts, 561981 multicasts, 31198967 pauses
 Output (normal): 7769549690 packets, - bytes
		 7768967085 unicasts, 20624 broadcasts, 561981 multicasts, 31198967 pauses
 Output: 0 output errors, - underruns, - buffer failures
		 0 aborts, 0 deferred, 0 collisions, 0 late collisions
		 0 lost carrier, - no carrier

The pauses are only displayed on the physical links, not on the bridge-aggregation interface (LAGG in H-P/H3C speak). Perhaps I need to do some iperf stuff between FreeNAS units and see what that gets. It sure looks like something is telling my network to slow down.
 
Joined
Dec 29, 2014
Messages
1,135
It certainly appears that any slowness is not related to the network. See my iperf3 results below:
Code:
root@freenas2:/nonexistent # iperf3 -c 192.168.252.23
Connecting to host 192.168.252.23, port 5201
[  5] local 192.168.252.27 port 16505 connected to 192.168.252.23 port 5201
[ ID] Interval		   Transfer	 Bitrate		 Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.43 Gbits/sec	0   3.19 MBytes
[  5]   1.00-2.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.19 MBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.19 MBytes
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.19 MBytes
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.20 MBytes
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.20 MBytes
[  5]   6.00-7.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.20 MBytes
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.20 MBytes
[  5]   8.00-9.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.20 MBytes
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec	0   3.23 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate		 Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.42 Gbits/sec	0			 sender
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec				  receiver

iperf Done.
root@freenas2:/nonexistent # iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.252.23, port 44637
[  5] local 192.168.252.27 port 5201 connected to 192.168.252.23 port 62120
[ ID] Interval		   Transfer	 Bitrate
[  5]   0.00-1.00   sec  1.02 GBytes  8.80 Gbits/sec
[  5]   1.00-2.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   6.00-7.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   8.00-9.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec
[  5]  10.00-10.06  sec  71.4 MBytes  9.42 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate
[  5]   0.00-10.06  sec  11.0 GBytes  9.35 Gbits/sec				  receiver
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
It certainly appears that any slowness is not related to the network. See my iperf3 results below:
Code:
root@freenas2:/nonexistent # iperf3 -c 192.168.252.23
Connecting to host 192.168.252.23, port 5201
[  5] local 192.168.252.27 port 16505 connected to 192.168.252.23 port 5201
[ ID] Interval		   Transfer	 Bitrate		 Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.43 Gbits/sec	0   3.19 MBytes
[  5]   1.00-2.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.19 MBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.19 MBytes
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.19 MBytes
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.20 MBytes
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.20 MBytes
[  5]   6.00-7.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.20 MBytes
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.20 MBytes
[  5]   8.00-9.00   sec  1.10 GBytes  9.42 Gbits/sec	0   3.20 MBytes
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec	0   3.23 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate		 Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.42 Gbits/sec	0			 sender
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec				  receiver

iperf Done.
root@freenas2:/nonexistent # iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.252.23, port 44637
[  5] local 192.168.252.27 port 5201 connected to 192.168.252.23 port 62120
[ ID] Interval		   Transfer	 Bitrate
[  5]   0.00-1.00   sec  1.02 GBytes  8.80 Gbits/sec
[  5]   1.00-2.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   6.00-7.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec
[  5]   8.00-9.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec
[  5]  10.00-10.06  sec  71.4 MBytes  9.42 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate
[  5]   0.00-10.06  sec  11.0 GBytes  9.35 Gbits/sec				  receiver
Have you tried do io benchmark locally for different recordsizes?
 
Joined
Dec 29, 2014
Messages
1,135
Have you tried do io benchmark locally for different recordsizes?
No, but I am certainly willing to do so. I am not sure how to go about that. I am mostly a network guy these days. I know enough FreeNAS/VMware to get around and do most of the basics, but I would certainly not call myself an advanced storage/virtualization person.
 

Ender117

Patron
Joined
Aug 20, 2018
Messages
219
No, but I am certainly willing to do so. I am not sure how to go about that. I am mostly a network guy these days. I know enough FreeNAS/VMware to get around and do most of the basics, but I would certainly not call myself an advanced storage/virtualization person.
I am certainly not an IT person of any type (my closest connection is probably semiconductor fabrication lol), but I guess the iozone -a -O -s 512M we used above is a place to start
BTW are block size same on the 2 hosts? having a mismatch might slow things down http://www.yellow-bricks.com/2011/02/24/storage-vmotion-performance-difference/
 
Status
Not open for further replies.
Top