Interference between jail and afp share

[kjp4756]

I'm experiencing the same problem here. This just started happening within the last week or so.

I have 2 jails. One for deluge/flexget and another for emby server. I've been running these for a few months and never had any issues until recently. If I stop both jails then time machine backups work as they should. With the jails running , automatic backups do not work. I also am having issues connecting to CIFS shares on freenas as well. I get the same message "connection failed". If I stop the jails then everything works as it should.

There is definitely something up with jails and mdns/zeroconf.

UPDATE: Did a clean install of FreeNAS-9.3-STABLE-201503270027 and now time machine and cifs works when jails are running. I'll stay with this version until either I or someone else can figure out what exactly is going on and submit a bug report. I did some watching in wireshark and saw my mac send out a mdns query but it never gets a reply back when 1 or more jails are running. It doesn't matter which jail it is either.

UPDATE 2: It seems the problem is back even on 201503270027. I need to have all of my jails stopped before AFP or CIFS can connect from the finder. When any or all jails are running I am seeing some interesting behavior with tcpdump running on freenas. When I have connection problems I am seeing this:

IP truncated-ip - 31855 bytes missing! 10.1.1.20.mdns > 224.0.0.251.mdns: 0*- [0q] 1/0/3 (Cache flush) SRV freenas.local.:445 0 0 (31972)

10.1.1.20 is my freenas box. I was trying to browse my cifs shares in finder when I got that. Eventually I got "Connection failed" in finder.

 

[adrianwi]

I'm having a similar problem that I asked for some help here > https://forums.freenas.org/index.php?threads/afp-share-issue-on-os-x-10-10-3.30423/#post-196379

Having ruled out any possible DNS issues by stopping the unbound jail and pointing everything back to my router and out to Google, I was running out of ideas, but it could be related to a conflict with jails.

I have 2 freenas machines running. freenas1 is my main box and is running a number of jail (owncloud, plex, openvpn) and this is the one I'm having irritate AFP connection issues with and also have 3 TM datasets (all at the root) for backing up 3 different Macs. freenas2 is just used as a backup (using ZFS Replication) and this works without any issues, and has no jails running.

Did anyone log this as a possible bug?

UPDATE: I've just done some testing by switching all my jails off and connecting via AFP works perfectly, with no lag between clicking the connect button in finder and entering the username/password. Switched them back on one by one and no problems with any of the 3 Plex jails (for Media Server, PlexConnect and PlexWatch) or OpenVPN. Switched on the jail based on VirtualBox-4.3.12 and the AFP problem comes back. Switch it off and AFP works fine again. The switched on ownCloud jail based on standard template (all others are except the VM one) and it stops working again. Switched off and it works! Finally switch on unbound jail and similar result as the ownCloud jail.

So it looks like something in the configuration of the ownCloud, unbound and VM jails which is causing my problem. The ownCloud jail is build on Josh's guide on here, the unbound one based around DrKK's youtube guide, and the VM jail is just the standard template with nothing changed other than creating a Window VM through the console.

Any ideas what things to check between the config of the OK jails and the problematic ones? Don't really know enough about FreeNAS or networking to know where to start :D

I guess the one think both of the standard jails have in common in a firewall running - ownCloud jail also has Fail2Ban running inside. Does the standard VM template have a firewall running?

UPDATE 2: Although I suspect there is still something in the update above, I tested this on my iMac. When I just tried on my MBA I will still having some issues connecting to the AFP shares - sometimes it just wouldn't connect and then other times it works perfectly connecting and disconnecting. Tried back on the iMac and the same intermittent problems with what I though might be the problem jails switched off. This is the error reported in the Mac Console:

Code:

27/04/2015 17:48:27.623 NetAuthSysAgent[4084]: DNSAddressResolver:Resolve CFNetServiceResolveWithTimeout failed
27/04/2015 17:48:27.624 NetAuthSysAgent[4084]: ERROR: AFP_GetServerInfo - connect failed 64

Both have a single DNS entry set to the router (Airport Extreme) which is pointing out to Google (8.8.8.8/8.8.4.4) :(

 

[adrianwi]

Not sure if this is related, but when I enter 'afpusers' at the CLI I get the following error:

Code:

afpusers                                                    
PID      UID      Username         Name                 Logintime Mac          
Traceback (most recent call last):                                             
  File "/usr/local/bin/afpusers", line 95, in <module>                         
    for (pid, uid, user, fname, time, mac) in AFPUsers():                      
  File "/usr/local/bin/afpusers", line 50, in AFPUsers                         
    host = socket.gethostbyaddr(m.group(2))[0]                                 
socket.herror: [Errno 1] Unknown host 

 

[kb0]

No pointer records for hosts on the network. I get the same error. Guess it would be fixed by giving rDNS info for all the hosts on my domain to DNS but this seems a bit extreme...I have no idea if this bug is related to the weird problem of jails interfering with afp.

 

[kjp4756]

I'm having no problems running afpusers. I do have proper internal DNS set up though.

I think I have an idea of what is happening. The vmnet driver for jails is causing issues with mdns. For some reason packets are getting truncated. I've been watching mdns traffic on my freenas server using tcpdump.

Here is a successful connection to an AFP share. I do use hostname=tmsrv in my afp service settings. This allows me to have CIFS and AFP separate in finder.

Code:

listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:43:50.876846 IP 10.1.1.154.5353 > 224.0.0.251.5353: 0 SRV (QU)? tmsrv._afpovertcp._tcp.local. (46)
15:43:50.876922 IP 10.1.1.20.5353 > 10.1.1.154.5353: 0*- [0q] 1/0/1 (Cache flush) SRV freenas.local.:548 0 0 (84)
15:43:51.224577 IP 10.1.1.154.5353 > 224.0.0.251.5353: 0 AAAA (QU)? freenas.local. (31)
15:43:51.224655 IP 10.1.1.20.5353 > 224.0.0.251.5353: 0*- [0q] 0/0/1 (42)

Now, here is a failed attempt to connect using AFP. The responses from 10.1.1.20 are being truncated. 10.1.1.20 is my freenas box. 10.1.1.154 is my OS X 10.10.3 machine.
Basically OS X isn't getting the response it requires from the server which is causing the connection to fail.

Code:

listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:46:20.973143 IP 10.1.1.154.5353 > 224.0.0.251.5353: 0 SRV (QM)? tmsrv._afpovertcp._tcp.local. (46)
15:46:20.973240 IP truncated-ip - 33130 bytes missing! 10.1.1.20.5353 > 224.0.0.251.5353: 0*- [0q] 1/0/3 (Cache flush) SRV freenas.local.:548 0 0 (33252)
15:46:29.954706 IP 10.1.1.154.5353 > 224.0.0.251.5353: 0 SRV (QM)? tmsrv._afpovertcp._tcp.local. (46)
15:46:29.954807 IP truncated-ip - 33130 bytes missing! 10.1.1.20.5353 > 224.0.0.251.5353: 0*- [0q] 1/0/3 (Cache flush) SRV freenas.local.:548 0 0 (33252)

 

[kb0]

You're right. As soon as I told the DNS service on pfSense to register DHCP leases and static mappings, afpusers started working like a charm. Thanks!

 

[kb0]

Here are are the mdns records from a tcpdump on my Mac. A jail is running on FreeNAS and I made a single (failed) attempt to 'Back Up Now' from TimeMachine:

Code:

15:34:21.797925 IP (tos 0x0, ttl 255, id 57589, offset 0, flags [none], proto UDP (17), length 82) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 SRV (QU)? Time Machines._afpovertcp._tcp.local. (54)
15:34:21.798015 IP6 (flowlabel 0x58709, hlim 255, next-header UDP (17) payload length: 62) fe80::260a:64ff:fe95:91f9.mdns > ff02::fb.mdns: [udp sum ok] 0 SRV (QU)? Time Machines._afpovertcp._tcp.local. (54)
15:34:22.882650 IP (tos 0x0, ttl 255, id 19423, offset 0, flags [none], proto UDP (17), length 82) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)
15:34:22.882734 IP6 (flowlabel 0x58709, hlim 255, next-header UDP (17) payload length: 62) fe80::260a:64ff:fe95:91f9.mdns > ff02::fb.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)
15:34:26.008764 IP (tos 0x0, ttl 255, id 55862, offset 0, flags [none], proto UDP (17), length 82) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)
15:34:26.008857 IP6 (flowlabel 0x58709, hlim 255, next-header UDP (17) payload length: 62) fe80::260a:64ff:fe95:91f9.mdns > ff02::fb.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)
15:34:35.094869 IP (tos 0x0, ttl 255, id 20732, offset 0, flags [none], proto UDP (17), length 82) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)
15:34:35.094929 IP6 (flowlabel 0x58709, hlim 255, next-header UDP (17) payload length: 62) fe80::260a:64ff:fe95:91f9.mdns > ff02::fb.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)

And here are the same records when I stop the jail and make a single (successful) attempt to 'Back Up Now' from TimeMachine:

Code:

15:36:42.185537 IP (tos 0x0, ttl 255, id 65482, offset 0, flags [none], proto UDP (17), length 77) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 [3q] AAAA (QU)? FreeNAS.local. A (QU)? khack.local. AAAA (QU)? khack.local. (49)
15:36:42.428765 IP (tos 0x0, ttl 255, id 2910, offset 0, flags [none], proto UDP (17), length 139) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0*- [0q] 2/0/2 khack.local. (Cache flush) [2m] AAAA fe80::260a:64ff:fe95:91f9, khack.local. (Cache flush) [2m] A 192.168.101.228 ar: khack.local. (Cache flush) [2m] A 192.168.101.228, khack.local. (Cache flush) [2m] AAAA fe80::260a:64ff:fe95:91f9 (111)
15:36:42.815938 IP (tos 0x0, ttl 255, id 64154, offset 0, flags [none], proto UDP (17), length 59) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 AAAA (QU)? FreeNAS.local. (31)
15:43:43.291808 IP (tos 0x0, ttl 255, id 59757, offset 0, flags [none], proto UDP (17), length 64) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 PTR (QM)? _privet._tcp.local. (36)

mdns name resolution is different. Nothing changes aside from starting and stopping a jail instance.

 

[kjp4756]

Here are are the mdns records from a tcpdump on my Mac. A jail is running on FreeNAS and I made a single (failed) attempt to 'Back Up Now' from TimeMachine:

Code:

15:34:21.797925 IP (tos 0x0, ttl 255, id 57589, offset 0, flags [none], proto UDP (17), length 82) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 SRV (QU)? Time Machines._afpovertcp._tcp.local. (54)
15:34:21.798015 IP6 (flowlabel 0x58709, hlim 255, next-header UDP (17) payload length: 62) fe80::260a:64ff:fe95:91f9.mdns > ff02::fb.mdns: [udp sum ok] 0 SRV (QU)? Time Machines._afpovertcp._tcp.local. (54)
15:34:22.882650 IP (tos 0x0, ttl 255, id 19423, offset 0, flags [none], proto UDP (17), length 82) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)
15:34:22.882734 IP6 (flowlabel 0x58709, hlim 255, next-header UDP (17) payload length: 62) fe80::260a:64ff:fe95:91f9.mdns > ff02::fb.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)
15:34:26.008764 IP (tos 0x0, ttl 255, id 55862, offset 0, flags [none], proto UDP (17), length 82) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)
15:34:26.008857 IP6 (flowlabel 0x58709, hlim 255, next-header UDP (17) payload length: 62) fe80::260a:64ff:fe95:91f9.mdns > ff02::fb.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)
15:34:35.094869 IP (tos 0x0, ttl 255, id 20732, offset 0, flags [none], proto UDP (17), length 82) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)
15:34:35.094929 IP6 (flowlabel 0x58709, hlim 255, next-header UDP (17) payload length: 62) fe80::260a:64ff:fe95:91f9.mdns > ff02::fb.mdns: [udp sum ok] 0 SRV (QM)? Time Machines._afpovertcp._tcp.local. (54)

And here are the same records when I stop the jail and make a single (successful) attempt to 'Back Up Now' from TimeMachine:

Code:

15:36:42.185537 IP (tos 0x0, ttl 255, id 65482, offset 0, flags [none], proto UDP (17), length 77) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 [3q] AAAA (QU)? FreeNAS.local. A (QU)? khack.local. AAAA (QU)? khack.local. (49)
15:36:42.428765 IP (tos 0x0, ttl 255, id 2910, offset 0, flags [none], proto UDP (17), length 139) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0*- [0q] 2/0/2 khack.local. (Cache flush) [2m] AAAA fe80::260a:64ff:fe95:91f9, khack.local. (Cache flush) [2m] A 192.168.101.228 ar: khack.local. (Cache flush) [2m] A 192.168.101.228, khack.local. (Cache flush) [2m] AAAA fe80::260a:64ff:fe95:91f9 (111)
15:36:42.815938 IP (tos 0x0, ttl 255, id 64154, offset 0, flags [none], proto UDP (17), length 59) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 AAAA (QU)? FreeNAS.local. (31)
15:43:43.291808 IP (tos 0x0, ttl 255, id 59757, offset 0, flags [none], proto UDP (17), length 64) khack.localdomain.mdns > 224.0.0.251.mdns: [udp sum ok] 0 PTR (QM)? _privet._tcp.local. (36)

mdns name resolution is different. Nothing changes aside from starting and stopping a jail instance.

You're only showing the mdns packets going to your server. What I can tell from the first one is the server is not responding to ip4 mdns so the mac is then trying ipv6 to get a response. I've seen the same behavior on mine.

My time machine and AFP have been working great since yesterday afternoon. It seemed to all start working properly after a ran 'tcpdump -i bridge0 udp port 5353' on my freenas server. I wonder if throwing bridge0 in and out of promiscuous mode some how fixes it. Please give it a try and see if it works for you.

 

[kb0]

Because mdns appears to be using ipv6 to resolve I went to Network>Interfaces, edited my principal interface and checked 'Autoconfigure IPv6'. Connections to AFP shares are no longer dropping out immediately when I turn on a jail(!!). However, Time Machine backups are taking longer than normal to complete. Not sure what the issue is.

 

[adrianwi]

I've just set interface to Autoconfigure IPv6 and AFP issues appears to be resolved, but I can't get into the GUI anymore :O

Think I changed WebGUI IPv6 Address: on the System > General tab too, so it has both IPv4 and IPv6 set.

Help!!!

 

[kb0]

Do you have physical access to your FreeNAS box? Just attach a monitor to the machine and use the CLI to reconfigure the interface.

 

[adrianwi]

I'm connecting via IPMI but when I try and select any of the Console setup options (1-14) I just get

FreeBSD/amd64 (freeness.apeconsulting.local) (ttyv0)

flash up and it doesn't accept the command.

I can SSH into it from my iMac if I can change it there?

 

[kb0]

Use ifconfig to delete the ipv6 address:

/sbin/ifconfig <interface> inet6 del <ipv6address>/<prefixlength>

 

[adrianwi]

Firstly, thanks for you help!

I'd had a look at ifconfig as I though I might be able to use the IPv6 address to access the GUI, but not sure the string it's listing is a correct value. This is the detail for the interface:

Code:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
    ether 0c:c4:7a:33:59:a2
    inet 192.168.168.15 netmask 0xffffff00 broadcast 192.168.168.255
    inet6 fe80::ec4:7aff:fe33:59a2%igb0 prefixlen 64 scopeid 0x2
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active

So I tried

Code:

/sbin/ifconfig igb0 inet6 del fe80::ec4:7aff:fe33:59a2/64

and I get

Code:

ifconfig: del: bad value

 

[kb0]

did you execute the command as root or using sudo?

 

[adrianwi]

I'm connecting via SSH as root

 

[kb0]

I'm a Linux guy and the syntax may be different for FreeBSD. Let me look into it...

 

[kb0]

Try replacing the delete (del) with -alias

 

[adrianwi]

That returns:

Code:

ifconfig: fe80::ec4:7aff:fe33:59a2/64: bad value

 

[kb0]

Try using fe80::ec4:7aff:fe33:59a2%igb0 as the ipv6 address? nevermind...