Troubleshooting Slow Down

[NASbox]

I have a FreeNAS unit:
FreeNAS-11.1-U7
GA-Z77X-UD5H Rev 1.1, i7-3770 CPU @ 3.40GHz ~ 32GB (DDR3)
LSI 9211-8i 8-port PCI-E SATA HBA
RAIDZ2: WD60EFRX x 8 [30TB]
RAIDZ HDS724040ALE640 x 2

I am trying to stream video to a media player, and for some reason I get stalls that cause the media player to buffer. The Video is being streamed from the RAIDZ2 Pool (The Second RaidZ Pool is used for backup and is normally "spun down"as it is rarely used. The FreeNAS box is not running any jails or other software and no other tasks are using the network or accessing the FreeNAS, so the load on the machine and network are very very light as this is a home installation, and when we are watching TV.

I don't suspect the "external network" because I have my personal workstation (i5-6500) which is on the same network segment, and I have no problems streaming video to the media player.

I'm not sure if this is an networking issue, or a FreeNAS tuning issue or a network setup issue. Any suggestions as to how I would go about troubleshooting this situation would be much appreciated.

 

[dlavigne]

Anything in /var/log/messages around the time of the stall? What is the full output of ifconfig within code tags?

 

[NASbox]

Anything in /var/log/messages around the time of the stall? What is the full output of ifconfig within code tags?

There is absolutely nothing in /var/log/messages except the messages that aree produced whenever the KVM switch brieftly connects a keyboard/mouse to the freenas box when I am switching between systems - but that wasn't happening when I was playing media. Here is the ifconfig:

Code:

ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:15:17:d2:67:7a
        hwaddr 00:15:17:d2:67:7a
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:15:17:d2:67:7a
        hwaddr 00:15:17:d2:67:7b
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em2: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 90:2b:34:d5:fa:e5
        hwaddr 90:2b:34:d5:fa:e5
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
alc0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c319a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MCAST,WOL_MAGIC,VLAN_HWTSO,LINKSTATE>
        ether 90:2b:34:d5:fa:e7
        hwaddr 90:2b:34:d5:fa:e7
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:15:17:d2:67:7a
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        groups: lagg
        laggproto lacp lagghash l2,l3,l4
        laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:15:17:d2:67:7a
        inet 172.16.50.141 netmask 0xffffff00 broadcast 172.16.50.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        vlan: 50 vlanpcp: 0 parent interface: lagg0
        groups: vlan
vlan192: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:15:17:d2:67:7a
        inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        vlan: 192 vlanpcp: 0 parent interface: lagg0
        groups: vlan
vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:15:17:d2:67:7a
        inet 172.16.11.10 netmask 0xffffff00 broadcast 172.16.11.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        vlan: 11 vlanpcp: 0 parent interface: lagg0
        groups: vlan

vlan11 is a private network with no access to the internet used for media playing only. Lagg0 connects to two ports on an SG200-20. The linux system that works also has a NIC on vlan11, but there is no LAGG.

 

[dlavigne]

Were you able to determine the cause of the slowdown?

 

[NASbox]

Were you able to determine the cause of the slowdown?

Thanks for asking. Unfortunately no. I'm not sure where to begin with troubleshooting this.
Is it something to do with FreeNAS buffering, the LAGG or something else.
Any suggestions as to where to start with troubleshooting this would be much appreciated.

 

[pschatz100]

So to be clear, you do not have any problem streaming video from your personal workstation to the media player. And, the FreeNAS system is acting only as a file server?

What is the media player you are streaming to? What is the format of the media you are streaming? What speed do you get if you just copy the file from FreeNAS to your workstation?

 

[SweetAndLow]

If the media player wireless? What do you mean by 'stream'?

 

[NASbox]

Thanks @pschatz100 & @SweetAndLow for the reply

So to be clear, you do not have any problem streaming video from your personal workstation to the media player. And, the FreeNAS system is acting only as a file server?

What is the media player you are streaming to? What is the format of the media you are streaming? What speed do you get if you just copy the file from FreeNAS to your workstation?

Yes, I have no problem streaming from my personal workstation. The FreeNAS box is only acting as a file server, and the only significant activity is the Samba share that is providing the video feed. There may have been an idle read only sshfs connection from the workstation, but it would not have been pushing/pulling any data, as I wouldn't be at the keyboard, and I wasn't runny any copy/backup operations.

the media player is an old Patriot Box Office "PBO Core" - a small ARM box. Media can be any one of a number of formats AVI, MP4, MKV most common. I can copy from FreeNAS to the workstation @ ~100MB/S, (GigEthernet) but the path is different. The path from both workstation and freenas is over a Powerline ethernet link. It's only good for about 100Mbits/S... but the player likely can't handle things that fast.

If the media player wireless? What do you mean by 'stream'?

The connection in all cases is ethernet of powerline, and when I say "stream", I mean playing a video file from a Samba Share

While responding to this, I put a movie on to test, and I was getting ~3-5 second lockups every 3-5 minutes, but when I played the same movie at 8X speed, I only experienced one very brief ~1-2Second lockup. (Each test was about 10 minutes.)

I'm thinking that it may be because of the LAGG. I suspect that at normal speed the buffer fills and data stops flowing, and then for some reason it takes too long to get the connection back when the media player calls for more data. At 8X, the media player never stops making requests as it isn't able to completely fill the buffer. Just a guess - not sure how to verify or troubleshoot. I can telnet into the media player, but it's very basic... busybox, and there isn't really much in the way of diagnostcs. I'm likely stuck with doing cat /proc/xxx or /sys/xxx to get raw information. The media player is on it's own private VLAN which is isolated from the internet since it has no security and would be very easy to compromise. Media player is connected by ethernet to a switch (Tomato running on an E3000, which is connected to the powerline ethernet adapter which trunks back to an SG300 Smart switch. The workstation has 5 NICs, one of which is on the same VLAN as the media player. FreeNAS connects the the SG300 over a 2 port LAG that trunks 4 VLANS, one of which is the media player VLAN. Other than a bit of mdns or other noise floating around the FreeNAS shouldn't be passing traffic over any of the other VLANS during the tests.

Does that provide any useful clues that give rise to suggestions?

 

[SweetAndLow]

The connection in all cases is ethernet of powerline

this will bring your problem 100% of the time. Power line networking is not fast. You should do an iperf test to be sure. I use LAGG and don't have strange budget issues you talk about. And in reality there is no bigger issue like you describe. To test this you should just remove the lagg. What kind of lagg is it anyways?

I get the impression you have absolutely no clue what you are doing. With all the vlans and laggs you doing be way now well versed than you are. You should look into removing all that stuff until you actually have a need for it and know what it is.

 

[pschatz100]

The PBO Core is a pretty old device, and I was thinking you may have some issues with support for your media. I used to use some old WD players and they had trouble with certain combinations of video codecs and audio codecs. Now, I use 4K compatible Roku's and they seem able to consume just about everything.

Do you get the lags and pauses on all media types? If yes, then almost certainly the problem is related to your network.

 

[NASbox]

Thanks @pschatz100 & @SweetAndLow for the reply. Working through your questions and replies has given me some things to try and helped me think things through in detail. At least now I have some things to try and my action plan is listed at the end and is open to suggestions.

this will bring your problem 100% of the time. Power line networking is not fast. You should do an iperf test to be sure. I use LAGG and don't have strange budget issues you talk about. And in reality there is no bigger issue like you describe. To test this you should just remove the lagg. What kind of lagg is it anyways?

I get the impression you have absolutely no clue what you are doing. With all the vlans and laggs you doing be way now well versed than you are. You should look into removing all that stuff until you actually have a need for it and know what it is.

I haven't tested the powerline network recently, but I don't suspect it is the major problem because it is common to both the working setup and non-working setup. The powerline network is one of the faster models-it's no GigE, but throughput is about equal to 1000BaseT - except there may be some dropouts/retrys - not sure how many.

I'm certainly no networking expert, and maybe have not set things up correctly, but VLANs serve a purpose and while I am certainly not there yet, I am working toward a high security network with VLANs that isolate traffic/firewall services (WiFi, IoT, Smartphones etc.) and carry a narrow and predicable class of traffic so it is easier to spot abnormalities such as rogue applications/root kits.

I suspect that you might be correct that the LAGG isn't configured correctly either on FreeNAS or on the switch.

What kind of lagg is it anyways?

It's a 2 port lacp from a dual port intel nic with 3 VLANs, the media VLAN and VLANs for 2 separate data networks (Both of which should idle during media watching.)

This might be significant:
I did notice that the PBO connection message shows a significantly higher connection rate when logging in to the workstation ~12000kbs vs ~8000kbs with FreeNAS. This must be network setup of some sort either FreeNAS or the Switch.

I do have the ability to monitor from the switch (and save to pcap/ run wireshark) at any of the following points: both Lagg Connections, the workstation ethernet conection, and the cable from the switch to the poweline ethernet adapter. Any suggestions as to what to look for?

The PBO Core is a pretty old device, and I was thinking you may have some issues with support for your media. I used to use some old WD players and they had trouble with certain combinations of video codecs and audio codecs. Now, I use 4K compatible Roku's and they seem able to consume just about everything.

Do you get the lags and pauses on all media types? If yes, then almost certainly the problem is related to your network.

Yes the media player is old and low powered (and a newer/faster player might be able to compensate), but it is again common to both cases.
I will recheck, but I'm also pretty sure it is not the media as I have played the same file over both paths and it works OK from the workstation.

Side Discussion
@SweetAndLow Is a Roku a viable local media player - i.e. to play from a samba share if it doesn't have access to the internet? I've got a Visio "Smart TV", but I haven't enabled any of that crap as I don't want to be "monitored", I don't trust the security, and the WiFi can't be turned off.

I'm likely going to get a small PC or maybe an android box for the TV. With all the tracing and other shady crap going on I'd like to get to the point where each session starts from a fresh image - likely a small linux machine with lxd containers. My design philosophy is isolate, non-persistant, firewall (and maybe monitor)

Action Items:

1. Pcap/Wireshark on line from switch to powerline Ethernet link
2. Setup linux laptop on powerline and run iperf
3. Replace media player with laptop running wireshark and VLC
4. Remove media player VLAN from lagg and assign it's own access port/NIC.