Poor 10Gb performance... help!

[Bazant0r]

Hi Forums,


Here is my hardware for FreeNAS:

2X Supermicro 4U Chassis ( 24Bay and external 45 Bay )
MB: Supermicro X10DRH-C/i
CPU: 2X Xeon 2.4 Ghz 2460 v3
MEM: 256GB
RAID: LSI 3108 MegaRAID 6.22.03.0 - FW 24.7.0.-0026

Drives:
1x: 32GB Sandisk USB 3.0 ( boot drive ) - hd0
4X: KINGSTON SUV400S 400GB ea (single L2ARC cache device)
4X: INTEL SSDSC2CT24 240GB ea (single ZIL device)
16X: WDC WD2003FZEX-0 2TB ea
24X: WDC WD2003FZEX-0 2TB ea - Connected via external SAS onto LSI RAID

20x vdevs consisting of ( 2 x 2TB devices )

2 X RJ-45 10G
4 X 10G SFP+ - Intel X710 chipset ( Installed Oct 2016 )

Here is my hardware for my ESXi Hosts:

4X Dell FC630 - 2 ea E5-2697v3 - 512GB RAM - 4X Intel x710 NICs ( passthrough IOM )

FreeNAS and ESXi hosts are connected via Dell S4048-ON 10Gb switch running DNOS 9.14.1 with Intel/Dell Official 10-GBASE-SR Fibermodules and high quality manufactured LC/LC fiber at 5m in the same rack.

What I am seeking help with is to troubleshoot an issue where I am having one way rate limiting on my FreeNAS storage. This has NEVER been an issue until very recently ( 1DEC2018 ) and I've been taking drastic troubleshooting steps to resolve. I am at my wits end and am seeking the assistance of the community at large.

Seeing poor performance on my 10Gb network ESXi -> FreeNAS ( Currently running at about <1 Gbps )

Seeing fair performance on my 10Gb network FreeNAS -> ESXi ( Varies from 10Gbps -> 6 Gbps in hard step downs )

Previously, I have been able to completely saturate a 10Gb link from these hosts to the storage bidirectionally. ( copying from All flash VSAN -> FreeNAS and back )

What I am trying to accomplish:
I am trying to regain the same performance that I had previously as it's required for the environment to operate effectively.


Steps I have taken thus far:
Removed 4x 10Gb lagg0 on FreeNAS to isolate potential bonding issues ( and check SFP+ health )
Created single tagged interface on FreeNAS to investigate and more easily run tcpdumps in FreeBSD.
Removed Jumbo frames from ESXi hosts, FreeNAS interfaces ( all MTU sizes sans switch are now set to default 1500 bytes )

Once I vacate all of my important files to the All-Flash VSAN, I am prepared to do the worst to the FreeNAS pool in the spirit of solving the problem.

Any ideas/thoughts that anyone has would be most appreciated as I am a one man show with no assistance or sounding board to see if my ideas are terrible.

 

[DrKK]

Did anything change on 1 December with the hardware or software? Did that correspond to updating the FreeNAS system? If so, have you rolled back to the previous version to see if the problem persists?

 

[Bazant0r]

Hi DrKK,

Thanks for the quick reply. Nothing configuration-wise. I did have a sudden problem in late November with ESXi hosts being unable to maintain connection to the NFS shares on FreeNAS which caused an APD on the vSphere side, but once I cleared that problem ( quick reboot of FreeNAS, restart of ESXi hosts ) things seemed positive until about the 1st of December. I only upgraded on the 15th ( yesterday ) in a vain hope that it would resolve the issue. I know it's dumb to think that upgrades will solve problems, but I've had really good luck with FreeNAS upgrades solving strange issues, usually involving drivers updates.

I have been going down the road of media errors ( was going to start pulling/replacing SFPs ) but wanted more opinions/ideas before I made the trip to do so. All of my important VMs are still moving today from NFS to FlashVSAN, so once they move, I was going to start doing some physical media troubleshooting and test with tagged/untagged switchports to see if I can isolate the problem.

 

[m0nkey_]

I'd like to know the state of your pool to start with. Please can you paste the output of zpool status and zpool list?

 

[Bazant0r]

Hi m0nkey,

You bet, here is zpool status:
inf-ustor-001# zpool status
pool: VMwareZpool
state: ONLINE
status: One or more devices are configured to use a non-native block size.
Expect reduced performance.
action: Replace affected devices with devices that support the
configured block size, or migrate data to a properly configured
pool.
scan: scrub repaired 0 in 0 days 01:16:55 with 0 errors on Sun Nov 18 01:16:59 2018
config:

NAME STATE READ WRITE CKSUM
VMwareZpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/277fbcd9-d4af-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/28136d0d-d4af-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-1 ONLINE 0 0 0
gptid/d41cc526-d4af-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/d4bb7e80-d4af-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-2 ONLINE 0 0 0
gptid/2ba71e57-d4b0-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/2c341141-d4b0-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-3 ONLINE 0 0 0
gptid/697635cb-d4b4-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/6a114220-d4b4-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-4 ONLINE 0 0 0
gptid/b8b66134-d4b4-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/b9644c03-d4b4-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-5 ONLINE 0 0 0
gptid/251d3e4e-d4b5-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/25baf0ec-d4b5-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-6 ONLINE 0 0 0
gptid/7fbf54b0-d4b5-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/8070f048-d4b5-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-7 ONLINE 0 0 0
gptid/d519bb39-d4b5-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/c5e117cd-ed02-11e7-9761-0cc47a35102d ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-8 ONLINE 0 0 0
gptid/1d9e3327-d4b6-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/1e41042f-d4b6-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-9 ONLINE 0 0 0
gptid/7e36946d-d4b6-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/7ee3e82e-d4b6-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-10 ONLINE 0 0 0
gptid/d25131a9-d4b6-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/d2f29477-d4b6-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-11 ONLINE 0 0 0
gptid/ac8c8e68-d4b7-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/ad21f382-d4b7-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-12 ONLINE 0 0 0
gptid/0d125a59-d4b8-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/0daf1c98-d4b8-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-13 ONLINE 0 0 0
gptid/560fb50d-d4b8-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/56a6f802-d4b8-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-14 ONLINE 0 0 0
gptid/abce5d9a-d4b8-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/ac70b561-d4b8-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-15 ONLINE 0 0 0
gptid/f42dc761-d4b8-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/f4d0b08b-d4b8-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-16 ONLINE 0 0 0
gptid/3265fc34-d4b9-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/33012820-d4b9-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-17 ONLINE 0 0 0
gptid/82df9b31-d4b9-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/837cdb22-d4b9-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-18 ONLINE 0 0 0
gptid/c98e0cb7-d4b9-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/ca324970-d4b9-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
mirror-20 ONLINE 0 0 0
gptid/ca54dd24-eae9-11e6-8ac0-0cc47a35102c ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/cb056816-eae9-11e6-8ac0-0cc47a35102c ONLINE 0 0 0 block size: 512B configured, 4096B native
logs
mirror-19 ONLINE 0 0 0
gptid/197bc6e5-d4bd-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/19cfd071-d4bd-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/1a251c6a-d4bd-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
gptid/1a7f2518-d4bd-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0 block size: 512B configured, 4096B native
cache
da4p1 ONLINE 0 0 0
da23p1 ONLINE 0 0 0
da11p1 ONLINE 0 0 0
gptid/029c8c9a-d4bd-11e6-bc35-90e2bacc4bd5 ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da48p2 ONLINE 0 0 0

errors: No known data errors

zpool list output:

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
VMwareZpool 36.2T 1.40T 34.9T - - 16% 3% 1.00x ONLINE /mnt
freenas-boot 28.5G 810M 27.7G - - - 2% 1.00x ONLINE -

 

[m0nkey_]

OK, you're well above the 80% utilization threshold. ZFS goes from first fit to best fit above this level of utilization. I believe this is the reason for your slow down.

 

[Bazant0r]

Hi m0nkey, maybe the text columns are hard to read?

inf-ustor-001# zpool list
NAME SIZE ALLOC FREE
VMwareZpool 36.2T 1.40T 34.9T

Isn't utilization a function of ALLOC/SIZE ?

 

[m0nkey_]

OK, yes. The text was kinda hard to read and I read it wrong. You're well below the threshold :) You can use the special [ code ] tags or pre-formatted text style options to make it easier.

Did you change anything prior or on beginning of December? Have you checked the SMART status of each drive?

 

[Bazant0r]

Hi m0nkey, sorry about that! You're right, I'll repost the info so it's not a ninja edit.

Code:

Welcome to FreeNAS
inf-ustor-001# zpool status
  pool: VMwareZpool
 state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the
        configured block size, or migrate data to a properly configured
        pool.
  scan: scrub repaired 0 in 0 days 01:16:55 with 0 errors on Sun Nov 18 01:16:59 2018
config:

        NAME                                            STATE     READ WRITE CKSUM
        VMwareZpool                                     ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/277fbcd9-d4af-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/28136d0d-d4af-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-1                                      ONLINE       0     0     0
            gptid/d41cc526-d4af-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/d4bb7e80-d4af-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-2                                      ONLINE       0     0     0
            gptid/2ba71e57-d4b0-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/2c341141-d4b0-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-3                                      ONLINE       0     0     0
            gptid/697635cb-d4b4-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/6a114220-d4b4-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-4                                      ONLINE       0     0     0
            gptid/b8b66134-d4b4-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/b9644c03-d4b4-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-5                                      ONLINE       0     0     0
            gptid/251d3e4e-d4b5-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/25baf0ec-d4b5-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-6                                      ONLINE       0     0     0
            gptid/7fbf54b0-d4b5-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/8070f048-d4b5-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-7                                      ONLINE       0     0     0
            gptid/d519bb39-d4b5-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/c5e117cd-ed02-11e7-9761-0cc47a35102d  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-8                                      ONLINE       0     0     0
            gptid/1d9e3327-d4b6-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/1e41042f-d4b6-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-9                                      ONLINE       0     0     0
            gptid/7e36946d-d4b6-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/7ee3e82e-d4b6-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-10                                     ONLINE       0     0     0
            gptid/d25131a9-d4b6-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/d2f29477-d4b6-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-11                                     ONLINE       0     0     0
            gptid/ac8c8e68-d4b7-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/ad21f382-d4b7-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-12                                     ONLINE       0     0     0
            gptid/0d125a59-d4b8-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/0daf1c98-d4b8-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-13                                     ONLINE       0     0     0
            gptid/560fb50d-d4b8-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/56a6f802-d4b8-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-14                                     ONLINE       0     0     0
            gptid/abce5d9a-d4b8-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/ac70b561-d4b8-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-15                                     ONLINE       0     0     0
            gptid/f42dc761-d4b8-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/f4d0b08b-d4b8-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-16                                     ONLINE       0     0     0
            gptid/3265fc34-d4b9-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/33012820-d4b9-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-17                                     ONLINE       0     0     0
            gptid/82df9b31-d4b9-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/837cdb22-d4b9-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-18                                     ONLINE       0     0     0
            gptid/c98e0cb7-d4b9-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/ca324970-d4b9-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
          mirror-20                                     ONLINE       0     0     0
            gptid/ca54dd24-eae9-11e6-8ac0-0cc47a35102c  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/cb056816-eae9-11e6-8ac0-0cc47a35102c  ONLINE       0     0     0  block size: 512B configured, 4096B native
        logs
          mirror-19                                     ONLINE       0     0     0
            gptid/197bc6e5-d4bd-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/19cfd071-d4bd-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/1a251c6a-d4bd-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
            gptid/1a7f2518-d4bd-11e6-bc35-90e2bacc4bd5  ONLINE       0     0     0  block size: 512B configured, 4096B native
        cache
          da4p1                                         ONLINE       0     0     0
          da23p1                                        ONLINE       0     0     0
          da11p1                                        ONLINE       0     0     0
          gptid/029c8c9a-d4bd-11e6-bc35-90e2bacc4bd5    ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da48p2    ONLINE       0     0     0

errors: No known data errors

inf-ustor-001# zpool list
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
VMwareZpool   36.2T  1.40T  34.9T        -         -    16%     3%  1.00x  ONLINE  /mnt
freenas-boot  28.5G   810M  27.7G        -         -      -     2%  1.00x  ONLINE  -

No changes in the environment configuration wise in late November / early December. I did have an All Paths Down issue when hosts lost connection to the NFS store on FreeNAS in late November, but that issue was cleared. Nothing has really changed in the physical environment since August 2017 when I moved the physical systems to their current resting location.


I did introduce an OSPF zone in Sept 2018 which changed unrelated network topology. ( vlan60, the storage vlan is unrouted )

Since I didn't do anything directly to cause the error that I can attribute, I am walking down the path of physical network media (either SFPs or a network card somewhere ) likely being the culprit, but I still don't know.

Here is my switch config in case you see something obvious.

I removed the port channel on te1/17-te1/20 ( referenced in original post ) today as a troubleshooting measure to try and get to the bottom of what's going on and limit variables for troubleshooting.

Code:

! Version 9.14(1.1)
! Last configuration change at Sun Dec 16 10:52:56 2018 by Bazant0r
! Startup-config last updated at Sun Dec 16 10:53:00 2018 by Bazant0r
!
boot system stack-unit 1 primary system://A
boot system stack-unit 1 default system://A
!
logging console informational
!
logging coredump stack-unit  1
logging coredump stack-unit  2
logging coredump stack-unit  3
logging coredump stack-unit  4
logging coredump stack-unit  5
logging coredump stack-unit  6
!
hostname INF-UFT10-001
!
feature vrf
!
no protocol lldp
!
redundancy auto-synchronize full
!
fips mode enable
!
XXXXXXX ---removed passwords ----- XXXXXXXX
!
ip vrf MainLABData 2
!
ip vrf management
!
stack-unit 1 provision S4048-ON
!
interface TenGigabitEthernet 1/1
 description DellFX2Switch
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/2
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/3
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/4
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/5
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/6
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/7
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/8
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/9
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/10
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/11
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/12
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/13
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/14
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/15
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/16
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/17
 description "FREENAS LACP"
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/18
 description "FREENAS LACP"
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/19
 description "FREENAS LACP"
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/20
 description "FREENAS LACP"
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/21
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/22
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/23
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/24
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/25
 description DellFX2vSphere
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/26
 no ip address
 mtu 9200
 no shutdown
!
interface TenGigabitEthernet 1/27
 no ip address
 mtu 9200
 no shutdown
!
interface TenGigabitEthernet 1/28
 no ip address
 mtu 9200
 no shutdown
!
interface TenGigabitEthernet 1/29
 no ip address
 mtu 9200
 no shutdown
!
interface TenGigabitEthernet 1/30
 no ip address
 mtu 9200
 no shutdown
!
interface TenGigabitEthernet 1/31
 no ip address
 mtu 9200
 no shutdown
!
interface TenGigabitEthernet 1/32
 no ip address
 mtu 9200
 no shutdown
!
interface TenGigabitEthernet 1/33
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/34
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/35
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/36
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/37
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/38
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/39
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/40
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/41
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/42
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/43
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/44
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/45
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/46
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/47
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface TenGigabitEthernet 1/48
 no ip address
 mtu 9200
 switchport
 no shutdown
!
interface fortyGigE 1/49
 no ip address
 shutdown
!
interface fortyGigE 1/50
 no ip address
 shutdown
!
interface fortyGigE 1/51
 no ip address
 shutdown
!
interface fortyGigE 1/52
 no ip address
 shutdown
!
interface fortyGigE 1/53
 no ip address
 shutdown
!
interface fortyGigE 1/54
 no ip address
 shutdown
!
interface ManagementEthernet 1/1
 ip address 10.0.0.103/24
 no shutdown
!
interface ManagementEthernet 2/1
 no shutdown
!
interface ManagementEthernet 3/1
 no shutdown
!
interface ManagementEthernet 4/1
 no shutdown
!
interface ManagementEthernet 5/1
 shutdown
!
interface ManagementEthernet 6/1
 shutdown
!
interface Vlan 1
 mtu 9018
!
interface Vlan 10
 no ip address
 mtu 9200
 tagged TenGigabitEthernet 1/1-1/16,1/21-1/25,1/33-1/48
 no shutdown
!
interface Vlan 20
 ip address 10.0.2.1/24
 mtu 9200
 tagged TenGigabitEthernet 1/1-1/16,1/21-1/25,1/33-1/48
 no shutdown
!
interface Vlan 30
 ip address 10.0.3.1/24
 mtu 9200
 tagged TenGigabitEthernet 1/1-1/16,1/21-1/25,1/33-1/48
 no shutdown
!
interface Vlan 50
 ip address 10.0.5.1/24
 mtu 9200
 tagged TenGigabitEthernet 1/1-1/16,1/21-1/25,1/33-1/48
 no shutdown
!
interface Vlan 60
 no ip address
 mtu 9100
 tagged TenGigabitEthernet 1/1-1/18,1/21-1/25,1/33-1/48
 untagged TenGigabitEthernet 1/19-1/20
 no shutdown
!
interface Vlan 70
 ip address 10.0.7.1/24
 mtu 9018
 tagged TenGigabitEthernet 1/1-1/16,1/21-1/25,1/33-1/48
 ip helper-address 10.0.2.110
 no shutdown
!
interface Vlan 90
 ip address 10.0.9.1/24
 mtu 9100
 tagged TenGigabitEthernet 1/1-1/16,1/21-1/25,1/33-1/48
 ip helper-address 10.0.2.110
 no shutdown
!
interface Vlan 100
 ip address 10.0.10.1/24
 mtu 9018
 tagged TenGigabitEthernet 1/1-1/16,1/21-1/25,1/33-1/48
 no shutdown
!
interface Vlan 999
 no ip address
 mtu 9018
 shutdown
!
interface Vlan 1001
 no ip address
 tagged TenGigabitEthernet 1/1-1/16,1/21-1/25,1/33-1/48
 no shutdown
!
route-map continue permit 10
!
ip prefix-list mainlabprefixes
 seq 10 permit 10.0.0.0/16
 seq 4094 deny any
!
router ospf 1
 network 10.0.2.0/24 area 0
 network 10.0.9.0/24 area 0
 network 10.0.3.0/24 area 0
 network 10.0.10.0/24 area 0
!
logging 10.0.2.46
!
no ip telnet server enable
!
ip helper-address hop-count disable
!
ip ssh server enable
!
line console 0
line vty 0
line vty 1
line vty 2
line vty 3
line vty 4
line vty 5
line vty 6
line vty 7
line vty 8
line vty 9
!
reload-type
 boot-type normal-reload
 config-scr-download enable
!
end

 

[Bazant0r]

So, I've removed the networking aspect as a possibility for poor performance. I get 8Gbps + on read, but only about 1.25 - 1.5 Gpbs on write. I am beginning to suspect that my ZIL drives may be slowing down for some reason. They are about two years old at this point on a pool that's heavily utilized and I think I used Intel MLC drives. I am going to replace them as a prophylactic measure and see if my performance improves. I am discounting the network since I am seeing my VSAN traffic exit the FreeNAS array at 8+Gpbs.

This is probably a positive change anyways, since I can go ahead and utilize a native 4k block size, since all of my drives support it.

 

[Bazant0r]

Just an update after some fairly extensive testing.

Just to recap: No changes on storage/network/compute when troubles began. Began to see poor performance on roughly 1DEC2018 with no discernible cause. I'm the only one in the environment with Storage/Compute/Network creds, so I can discount other admin change.

Steps I've now taken-

1.)

Installed:​

2 * Samsung 840 PRO SSDs for ZIL to replace 4X: INTEL SSDSC2CT24 240GB​

6 * Intel 545s for L2ARC to replace 4X: KINGSTON SUV400S 400GB​

Replaced the boot flash drive for good measure.​

​

Outcome: Reloaded my pool and had the same problems. ( can't get decent performance on 10G links, failures to mount NFS shares )​

Discounts Configuration/Pool Composition as cause​

​

2.)
Destroyed original pool, recreated with same layout to take advantage of 4k block sizes on drives

Outcome: No improvement.​

Discounts block size mismatch as possible cause.​

​

3.)

Destroyed all Logical Interfaces/Adapters to remove possible bottlenecks in FreeNAS ( used untagged/access ports versus tagged/trunk ports.​

Replaced SFP+, cables, and swapped out to test. Tested using the 10G RJ45 links on baseboard instead of the Intel X710 card.​

Outcome: No improvement.​

Discounts switching infrastructure/SFP/Cabling​

I can only think this leaves FreeNAS as a possible culprit, but I am hoping for some kind assistance from the community to help me think of possible troubleshooting avenues.

Attached is the demonstration of poor write performance/high read performance.

 

[Bazant0r]

Output of zpool status from new pool for layout info.

Code:

root@freenas[~]# zpool status
  pool: ZFSPool
 state: ONLINE
  scan: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        ZFSPool                                         ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/4171d4a6-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/43a46c2a-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/45d242e2-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/5081f277-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-2                                      ONLINE       0     0     0
            gptid/5471522a-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/56840955-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-3                                      ONLINE       0     0     0
            gptid/58caf08e-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/5af1b2d0-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-4                                      ONLINE       0     0     0
            gptid/5d522c82-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/5f90a3a6-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-5                                      ONLINE       0     0     0
            gptid/61e3b5d7-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/6400cc4c-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-6                                      ONLINE       0     0     0
            gptid/664f1e6a-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/6882cb85-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-7                                      ONLINE       0     0     0
            gptid/6b2d4d32-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/7d9efe68-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-8                                      ONLINE       0     0     0
            gptid/81ada395-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/83c2b87d-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-9                                      ONLINE       0     0     0
            gptid/8600f44a-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/88459bab-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-10                                     ONLINE       0     0     0
            gptid/8a88fb9e-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/9597e550-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-11                                     ONLINE       0     0     0
            gptid/a3a2d065-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/b0e42d8e-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-12                                     ONLINE       0     0     0
            gptid/bd58bb99-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/c9ba6257-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-13                                     ONLINE       0     0     0
            gptid/d63c58d6-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/e28606c3-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-14                                     ONLINE       0     0     0
            gptid/eecb7f80-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/fb19cbef-0b20-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-15                                     ONLINE       0     0     0
            gptid/0974ce09-0b21-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/15ca3c66-0b21-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-16                                     ONLINE       0     0     0
            gptid/245f4c55-0b21-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/309462a8-0b21-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-17                                     ONLINE       0     0     0
            gptid/3f63d7bf-0b21-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/4d81c4f0-0b21-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
          mirror-18                                     ONLINE       0     0     0
            gptid/5c24983a-0b21-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/6880dc7f-0b21-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
        logs
          mirror-19                                     ONLINE       0     0     0
            gptid/71c03097-0b21-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
            gptid/72dc55fc-0b21-11e9-a20d-0cc47a35102c  ONLINE       0     0     0
        cache
          gptid/6b5a4974-0b21-11e9-a20d-0cc47a35102c    ONLINE       0     0     0
          gptid/6c5c4286-0b21-11e9-a20d-0cc47a35102c    ONLINE       0     0     0
          gptid/6d4d1b4d-0b21-11e9-a20d-0cc47a35102c    ONLINE       0     0     0
          gptid/6e48f4c9-0b21-11e9-a20d-0cc47a35102c    ONLINE       0     0     0
          gptid/6f4500b1-0b21-11e9-a20d-0cc47a35102c    ONLINE       0     0     0
          gptid/70494347-0b21-11e9-a20d-0cc47a35102c    ONLINE       0     0     0
        spares
          gptid/7fcf3ab3-0b21-11e9-a20d-0cc47a35102c    AVAIL
          gptid/8c40a421-0b21-11e9-a20d-0cc47a35102c    AVAIL

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da48p2    ONLINE       0     0     0

errors: No known data errors

 

[depasseg]

@Bazant0r - I'm also trying to configure a fresh FreeNAS install on an FC630 (with an attached FD332 and 8 SSDs) to a pair of Dell S40408 switches in VLT. How did you config your switches to bring up LACP in freenas? I can't get the port-channel to come up. I'm wondering if there's a network issue going on that's related to what you are experiencing. If I can get this working, I'll give you some performance stats.

 

[Bazant0r]

Hi @depasseg - I created an active untagged port channel using 4 links on one switch ( I only had one when I was doing the aggregation originally ) and required a minimum of 2 links to bring it up, I defined the IP interface on the switch. I used the "LACP" option in FreeNAS when I created the aggregation interface with the address specified on the switch. I've found that FreeNAS, despite the fact that it shouldn't, does require a reboot to make the link aggregation interface fully come up. ( Not sure how it would work with an FX2 where the links never actually go down on the switch. )

I am interested in hearing more about your setup since it sounds like we are working with the same switch INF and I am still very basic in my switch configs. Are you using the I/O passthroughs on your FX2 or do you have the I/O aggregators? I'm not sure how to do this across a VLT, but am very interested in hearing your final setup as I just bought another S4048-ON for redundancy and am interested in optimizing everything as much as possible.

 

[Elliot Dierksen]

I've found that FreeNAS, despite the fact that it shouldn't, does require a reboot to make the link aggregation interface fully come up.

This is just a guess, but I suspect that is because LACP negotiation is done when the port comes up. The switch may not have initiated that since the link state did not change. FreeNAS will always do LACP active mode, but the switch could be in active or passive depending on how you configure it. It is too late now, but a switch port shut/no shut or disconnect/re-connect of the LAN cables would have probably done the trick as well.

 

[depasseg]

Yes, I have 10GbE I/O passthrough ports (not the IOA). Between the 2 switches I've configured a VLT using 2 40GbE ports on each switch.

Then in the port channel configuration on each switch (which must be identical on each switch), just add a "vlt-peer-lag port-channel 102". I have this working to an LACP aggregate on a Palo Alto NGFW from the switches.

I've got 4 ports coming from the FC630 (2 going to each switch)

I'm assuming you are using BIOS mode (I couldn't get it to install using UEFI).

I've also got Qlogic BCM57840 10GbE NDCs (not Intel) :-( They appear to work fine without LACP.

 

[depasseg]

I've got the switch port-channel LACP setup as active, but I've also tried passive. And I've tried shut/no shut all the interfaces. Rebooting the server, rebooting the switches. pulling the DACs. I've got no idea. I see the LACP counters (Xmit and Recv) increasing. Still nothing. Grrrr

 

[depasseg]

Woo hooo! I at least found my issue - known Qlogic driver bug. The interfaces need to be in promiscuous mode to negotiate.

ifconfig bxe0 (through bxe3) promisc did the trick (plus a shut no shut on the switch side to force the renegotiation).

Now on to configuring storage.

For future ref:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213606
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227743