"pkg" problem when trying to install jail plugin

[Louis2]

Hello,

I am running true nas core 12-U6.1 and would like to install some jails running default packages like e.g. famp, plex and nextcloud.

The true nas system is connected to my network using a trunk carrying multiple vlans. The intention is as follows:
- one vlan connected to the true nas core application and
- one vlan per jail e.g.
- vlan10 => famp(1) running webserver for domain-A
- vlan20 => famp(2) running webserver for domain-B
- vlan30 => nextcloud
I need a strict vlan separation, so I think I need to use vnet.

I do not manage

Actual base situation is like this:
- the gateway for all vlans is a pfSense router
- the trunk is ix0
- I did define an interface for each vlan
- I did assign IP-addresses to the vlans intended for the truenas core application and
I did NOT assign IP-addresses to the vlans intended for the jails (or not yet present vm's)

Than I start creating jails by adding plugins:
- The first plugin I tried was famp. That failed (I do not remember the exact messages)
- the second try was Nextcloud and that worked
- after that I tried to install famp in an second jail ... which again failed.
- I did try plex ... same issue

Error: ^myJail^ had a failure Exception: RuntimeError Message: pkg error: - pkg-static: http://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/meta.txz: No address record pkg-static: http://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/packagesite.txz: No address record Please check your network Partial plugin destroyed

I tried to install nextcloud in an second jail, which ..... does fail as well .... perhaps for the same reason .... !!??

pfSense capture shows
11:08:03.416079 IP 192.168.110.10.35864 > 192.168.1.1.53: UDP, length 33
11:08:03.416167 IP 192.168.110.1.53 > 192.168.110.10.35864: UDP, length 74
11:08:03.417187 IP 192.168.110.10 > 192.168.110.1: ICMP 192.168.110.10 udp port 35864 unreachable, length 110
11:08:13.454390 IP 192.168.110.10.54738 > 192.168.1.1.53: UDP, length 39
11:08:13.454586 IP 192.168.110.1.53 > 192.168.110.10.54738: UDP, length 114

Decoded with wireshark
No. Time Source Destination Protocol Length Address Info
1 11:08:03,416079 192.168.110.10 192.168.1.1 DNS 75 6c:b3:11:09:07:48,6e:b3:11:b6:3f:ec Standard query 0xc357 A pkg.freebsd.org
No. Time Source Destination Protocol Length Address Info
2 11:08:03,416167 192.168.110.1 192.168.110.10 DNS 116 6e:b3:11:b6:3f:ec,6c:b3:11:09:07:48 Standard query response 0xc357 A pkg.freebsd.org CNAME pkgmir.geo.freebsd.org A 139.178.72.201
No. Time Source Destination Protocol Length Address Info
3 11:08:03,417187 192.168.110.10 192.168.110.1 ICMP 144 6c:b3:11:09:07:48,6e:b3:11:b6:3f:ec Destination unreachable (Port unreachable)

Note that 110.10 is the jail and 110.1 is the gateway

Also got a message like

Error: <some-jail> had a failure Exception: RuntimeError Message: Stopped <same-jail> due to VNET failure Partial plugin destroyed

In jail setup 'Basic Properties' I tried a couple of things but here an example for what I did for nextcloud
DHCP = NO (I define an fixed IP)
NAT = NO (NAT is a task for pfSense not for TrueNAS)
VNET = YES (vlan's should be 100% separated from each other)
Packet filter = NO (I am not sure, about if and why)
vnet_default_interface = vlan111:Nextcloud
ipv4 interface = vnet0 ipv4 address = 192.168.111.10 ipv4 NetMask = 32

So bottom line:
- I try to install multiple jails each connected to their own vlan
- the vlans are arriving via a trunk. The Gateways are on pfSense
- the problems seems to be related to the communication stack
- the first Nextcloud instance however .... using the same principles .... does work

So if someone could tell me what I am doing wrong .... It would be appreciated


Louis

 

[sretalla]

The Gateways are on pfSense

ipv4 interface = vnet0 ipv4 address = 192.168.111.10 ipv4 NetMask = 32

I don't think you're subnetting it right... a single address can't really route like that, you need at least 2, 1 for pfSense, 1 for the jail.

 

[Patrick M. Hausen]

Apart from the addressing issue - you really cannot use a /32 netmask on a broadcast medium like Ethernet except as an alias, do this:

Set all your jails autostart to off and reboot your TrueNAS.

For each VLAN

• create a matching interface of type bridge, e.g. bridge10 for vlan10
• make the vlan interface the only member of that bridge interface
• if there is an IP address for the NAS host on that VLAN, remove it and put it on the bridge interface instead

test, save, reboot (just to make sure)

For each jail:

• set vnet_default_interface from "auto" to "none"
• set "interfaces" field (down in the "Network Properties" section) to e.g. vnet0:bridge10 for bridge10. "vnet0" is alway "0", don't change that number - just the bridge

Start the jails and test.
Re-enable autostart for the jails.

 

[Louis2]

I don't think you're subnetting it right... a single address can't really route like that, you need at least 2, 1 for pfSense, 1 for the jail.

Yep, of course I have been testing around to much. I "changed it back" to /24. But that alone did not solve the problem. I noticed there is already another reaction, which I need to think about / to test.

Thanks!

Louis

 

[Louis2]

Apart from the addressing issue - you really cannot use a /32 netmask on a broadcast medium like Ethernet except as an alias, do this:

Set all your jails autostart to off and reboot your TrueNAS.

For each VLAN

• create a matching interface of type bridge, e.g. bridge10 for vlan10
• make the vlan interface the only member of that bridge interface
• if there is an IP address for the NAS host on that VLAN, remove it and put it on the bridge interface instead

test, save, reboot (just to make sure)

For each jail:

• set vnet_default_interface from "auto" to "none"
• set "interfaces" field (down in the "Network Properties" section) to e.g. vnet0:bridge10 for bridge10. "vnet0" is alway "0", don't change that number - just the bridge

Start the jails and test.
Re-enable autostart for the jails.

Ok, thanks!

Since a bridge is ihmo just a sort of unmanaged switch and all vlans are as default attached to bridge0, I can image that this could work. I will try that probably later today

Louis

 

[Patrick M. Hausen]

A bridge in FreeBSD is a vSwitch, yes. And it cannot be further divided into VLANs. So you need one bridge per VLAN. That's not a bug but the architecture - it's supposed to work that way.

So from a management viewpoint, and if you are familiar with ESXi, it's s port group. Technically it's a limited vSwitch.

 

[Louis2]

Testing ......
- I defined the bridges and assigned the vlans to the bridges. I did not give the jaiul related bridges and vlans any ip adress / subnet
- I did try to install nextcloud as first jail, which did did not work (connection problem)

This is what I tried:
- plugins select nextcloud
- advanced nat=off, vnet=on, vnet_default_interface=none, ipv4 address=192.168.111.2, netmask=/24, default router=192.168.111.1
- name=Nextcloud,
- autostart=off
- network properties. vnet0:bridge111; ip4 inherit; ipv6 disable (for the moment)

I tried a few more things. Still the same problem

Error: Nextcloud had a failure Exception: RuntimeError Message: pkg.FreeBSD.org could not be reached via DNS, check Nextcloud's network configuration Partial plugin destroyed

 

[Patrick M. Hausen]

Sorry - no clue about the plugin system. We exclusively run our own standard jails. Could you try if you can get one to connect as expected, so we could work from there with the plugin?

One thing: all my jails have "IPv4: New" instead of "inherit" ...

 

[Louis2]

OK,

I just did two tests:
1) just the default jail ........ it works; note that I used basicsettings vnet_default_interface = auto!
2) again with nextcloud where I
- also used "auto"
- and did remove the static routes towards the gateways, since they are not needed when using separate stacks for each vlan

Also note that I have and had activated some tunables, which imho does not affect the behavoir being:
- net.add_addr_allfibbs;value=0; sysctl; Do not create default routing tables for all FIB's
- net.fibs;value=12;loader;Define the (max) number of routing tables (max definable =16)
- vfs.zfs.arc_max;value=17179869184;sysctl;Limit ARC max size to 16GB

I will also try with plex, but do not expact a different outcome

 

[Louis2]

A few additional tests:
- disable tunables does not change any thing
- installing two default jails are no problem. They can ping each other as well
- installing plex does not work, neither with vnet_default_interface = none or auto

 

[Louis2]

I found an problem ...... however still not OK
- in network properties there is a field resolver ...... which by default point to ...... a kind of host file.
- that is IMHO far from OK. It should point to a couple of DNS servers at least one for IPV4 and IPV6.
- in my case the gw-address is also the portal to the DNS, however of course 8.8.8.8 (google) workjs as well
- however ..... that works for the default jails (which can by the way by default not access a dns!!)
- however it does not work for the plugins .... so there is more to solve .....

 

[Patrick M. Hausen]

The "resolver" field takes a syntax similar to /etc/resolv.conf. So if you want to use e.g. 8.8.8.8 you need to put "nameserver 8.8.8.8" in there.
If the content is "/etc/resolv.conf" that means "copy the nameserver config from the TrueNAS host."

What do you mean by default jail, btw? They are connected to the interface that has your default route? Or did you possibly name your bridge "bridge0"?

I assumed you had problems assigning different jails to different VLANs ... that's why I went on that lengthy explanation.

 

[Louis2]

No, you are mistaken. "none" is referring to the name server of the default host! But apart from that IMHO using the gateway is a more common way of accessing the DNS.

I solved this problem by putting the gateway adress(es IP{V4 and IPV6) on the line. But of course other DNS server IP's do work as well.

With default jail I mend just stating an empty jail from the jail menu in oposit from creating a jail by installing a plugin.

 

[Patrick M. Hausen]

No, you are mistaken. "none" is referring to the name server of the default host!

/etc/resolv.conf does the same. I checked the iocage source code on github.

 

[Louis2]

/etc/resolv.conf does the same. I checked the iocage source code on github.

Yep I tested that. And its true, however note that:
- a jail should not use definitions from the host
- and it does only work if the host is referring to an public DNS (like 8.8.8.8)
- and it does not work is the host is referring to an DNS in its own vlan (like 192.168.<hostvlan>.1)

So I absolutely do not like the use of "/etc/resolv.conf" or "none"

 

[Patrick M. Hausen]

It does work in the default case of one LAN, one bridge and all jails sharing that with the host.
Which is actually what most users run and the reason why iX got away with the wrong layer 3 setup for so long.

 

[Louis2]

Not sure if this is going to work using the actual TrueNas version.
- still DNS related problems, at least during installation. However you can work around that by using e.g. 8.8.8.8 in the golbal network settings
- however, that you try to install a plugin and .... the network performance is so slow ....... the speed of a 1970 modem ....... that even after hours there is not yet a working plugin-jail
- I noticed that there are more complains about that, and also advice from Patrick to turn hardware acceleration off ...... because of bugs of course, which is ridiculous. I never the less tried "igb0 -vlanhwtag" without success.

So ...... I got a strong feeling that I have to wait for
- Truenas core 13 or (not even a public beta yet)
- Truenas scale (there is a RC, which is IMHO not yet really an RC)

 

[Patrick M. Hausen]

I noticed that there are more complains about that, and also advice from Patrick to turn hardware acceleration off ...... because of bugs of course, which is ridiculous.

What bugs? Hardware assisted manipulation of e.g. TCP headers is a layer violation if the connection is not terminating at the host but the host is merely serving as a bridge or router to the final stack receiving the packet. So of course it needs to be turned off. All hardware acceleration.

I run jails on VLANs with or without lagg by the hundreds. I might miss certain things particular to your installation and being human I might jump to conclusions faster than helpful. But all of this is working now and FreeBSD 13 is not going to change a bit about how the stack is supposed to work.

Please disable all hardware acceleration on the physical interfaces that build your lagg, if you want to use a bridge interface on them.
And if there is any host based IP address on that interface (VLAN?) put that on the bridge and not the vlan interface ...

 

[Louis2]

It really feels strange to me to me that offloading of processing should be turned off. That is normally very beneficial! However OK lets give it a change.

I am using an intel 550 board. And one of the interfaces ix0 is my trunk. I am not using a lagg on this machine.

I assume that the additional settings should be placed in the interface menu on the options line. I also assume that all options should be semicolon(?) separated. And that the line should look like this ix0 -vlanhwtag; etc ..

Do you have an example of what should be on that line?

I will start a test with the supposed "ix0 -vlanhwtag" to start with

 

[Patrick M. Hausen]

There is a "disable hardware offloading" checkbox. Ticking that should be enough. If that does not disable all options, then I'd agree that's a bug in the TrueNAS UI/middleware. You can then proceed using the options field, yes. And only space separated. E.g. "-tso -hwvlantag"