TrueNAS 13.0-U6 is Now Available

[Etorix]

"The tool in the UI."

Do you mean a… plugin ?

 

[Patrick M. Hausen]

You need to use the UI to create a bridge interface before you create any jails and move the IP address from the physical interface to the bridge. This is mandatory by the FreeBSD documentation and has been repeatedly confirmed by the FreeBSD developers concerned with the network stack, but iX seem to just ignore that.

 

[c32767a]

"The tool in the UI."

Do you mean a… plugin ?

I mean the plugin setup process.

Sorry if the terminology is wrong. We turned this up in our testing and dealt with it.

I was just trying to provide an observation of a potential negative effect of following a recommendation published in the release notes.

I'll remember it's more important to call a plugin a plugin if I post agian.

 

[blanchet]

Due to security vulnerabilities and maintainability issues, the S3 service is deprecated in TrueNAS CORE 13.0 and scheduled for removal in CORE 13.1. Beginning in CORE 13.0-U6, the CORE web interface generates an alert when the deprecated service is either actively running or is enabled to start on boot. Users should plan to migrate to a separately maintained MinIO plugin or otherwise move any production data away from the S3 service storage location.

The main issue is that Minio has deprecated both the S3 Gateway and Filesystem services.

Deprecation of the MinIO gateway

MinIO is deprecating the gateway and will be completely removed in six months. This should not come as a surprise, we began informing the community in 2020 and have steadily removed unpopular gateways. In the last ten months, MinIO has only made bug fixes. The community can continue to use

blog.min.io

It means that even if you migrate to minio-plugin or minio-in-your-custom-jail, you cannot access anymore to the same files through NFS and S3.

It is very annoying for many workflows
For example, if you upload files with a modern web application (S3), but finally you process the data with an application that needs regular files (NFS, Samba)

 

[Juan Manuel Palacios]

You need to use the UI to create a bridge interface before you create any jails and move the IP address from the physical interface to the bridge. This is mandatory by the FreeBSD documentation and has been repeatedly confirmed by the FreeBSD developers concerned with the network stack, but iX seem to just ignore that.

I don't recall having done that manually, and I do have a bridge interface that plugs my jails' interfaces to my host's igb0 interface, and networking works like charm for me here at home.

Could that be because I always setup my jails in DHCP mode? When I select that option, the TrueNAS CORE UI disables interface selection, and seems to do the right thing automagically.

 

[Patrick M. Hausen]

It does the wrong thing automatically because when you now type ifconfig in a shell you will find that your bridge interface does not have an IP address while the hosts igb0 interface does. Which is explicitly and very clearly forbidden by FreeBSD documentation.

Among other things that configuration of yours breaks multicast. And IPv6.

Why? Ask iX. That was documented since day 1 of the introduction of the bridge interface to FreeBSD:

Chapter 34. Advanced Networking

Advanced networking in FreeBSD: basics of gateways and routes, CARP, how to configure multiple VLANs on FreeBSD, etc

docs.freebsd.org

If the bridge host needs an IP address, set it on the bridge interface, not on the member interfaces.

To reiterate: a bridge member interface MUST NOT have an IP address. This is not debatable. TrueNAS does it wrong. That's why I have describes again and again how to perform a manual bridge setup and not rely on the "magic".

 

[Juan Manuel Palacios]

I did not know that, thanks for the explanation!

And you're indeed correct, what you describe is exactly what I see in my NAS' network setup. But, just wondering here, if that setup is not just discouraged, but outright forbidden, presumably because it's somehow inherently broken, how come then my NAS' networking appears to work just fine? Here's a summary of what ifconfig says for it:

Code:

igb0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: igb0
    options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
(…)
    inet 10.0.0.5 netmask 0xffffff00 broadcast 10.0.0.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
(…)
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
(…)
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
(…)
    member: vnet0.28 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 13 priority 128 path cost 2000
(several other similar bridge members)
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
(…)
vnet0.28: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
    description: associated with jail: unifi-controller as nic: epair0b
    options=8<VLAN_MTU>
(…)
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=2b<PERFORMNUD,ACCEPT_RTADV,IFDISABLED,AUTO_LINKLOCAL>

And from within my unifi-controller jail:

Code:

epair0b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    options=8<VLAN_MTU>
(...)
    inet 10.0.0.9 netmask 0xffffff00 broadcast 10.0.0.255
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

And networking for my NAS host and all of its jails works just fine, like I said above. They all have IPv4 LAN connectivity, and both IPv4 & IPv6 WAN connectivity, so if anything is broken about this setup, either I just haven't noticed it, or it hasn't year rear its ugly head.

Thanks in advance for any FreeBSD networking wisdom you might be able to offer!

 

[Patrick M. Hausen]

But, just wondering here, if that setup is not just discouraged, but outright forbidden, presumably because it's somehow inherently broken, how come then my NAS' networking appears to work just fine?

You are not using multicast and you are not using IPv6 I assume

 

[Juan Manuel Palacios]

You are not using multicast and you are not using IPv6 I assume

IPv6 in my LAN, nah, haven't had the need for it so far, and I don't think I'm using multicast from my NAS either. Maybe my plex jail would be the only one that'd need it, and as far as I've been able to tell, if it does indeed use multicast, it seems to work without any problems because I can connect it to my TV just fine (and my UniFi Controller running in its jail can find my UniFi devices without problems either); but, other than that, I don't think anything else is using multicast (if those two examples indeed are).

In any case, outside of IPv6 and multicast, should I expect other networking problems with my setup? Or would you say it's an "all bests are off" type of situation?

Thanks!

 

[Etorix]

Sorry if the terminology is wrong. We turned this up in our testing and dealt with it.

I was just trying to provide an observation of a potential negative effect of following a recommendation published in the release notes.

I'll remember it's more important to call a plugin a plugin if I post agian.

The issue here is not terminology, or using jails. The issue is that "plugins" were not followed up, and have eventually been deprecated—with official notice to NOT use them. But plugins are still there, in GUI and iX documentation.

@eturgeon, @morganL, and all iXSytem employees reading this thread: Please act consistently and remove plugins for good. Plugins being still present in GUI is harmful to users, some of which will try to use them, assuming that they should work (see my emphasis in the above quote).

 

[morganL]

The issue here is not terminology, or using jails. The issue is that "plugins" were not followed up, and have eventually been deprecated—with official notice to NOT use them. But plugins are still there, in GUI and iX documentation.

@eturgeon, @morganL, and all iXSytem employees reading this thread: Please act consistently and remove plugins for good. Plugins being still present in GUI is harmful to users, some of which will try to use them, assuming that they should work (see my emphasis in the above quote).

Some plugins do work and are useful. We'd be open to a process of disabling specific plugins that don't work. Perhaps we should start a new thread to discuss these plugins in detail?

We are actively pushing users to jails or SCALE Apps, However, we will not remove plugins that do work and have active users. This would be unreasonable to those users without their permission.

 

[c32767a]

We are actively pushing users to jails or SCALE Apps, However, we will not remove plugins that do work and have active users. This would be unreasonable to those users without their permission.

I may be in the minority, but generally speaking, I don't see the value of running non storage related tasks on a storage appliance, so I don't really see the need for a plugin/jail system..
Having the ability to run an S3 service on the storage appliance is useful. If you're not planning on supporting S3 as a peer storage service to say SMB or NFS, I would retire it as a service or plugin, at least for Core.. Maybe scale's multi-node model would make it more practical, but a single minio instance running in a Core jail isn't really a practical deployment, given how minio does redundancy and scaling.

 

[Juan Manuel Palacios]

I may be in the minority, but generally speaking, I don't see the value of running non storage related tasks on a storage appliance, so I don't really see the need for a plugin/jail system..
Having the ability to run an S3 service on the storage appliance is useful. If you're not planning on supporting S3 as a peer storage service to say SMB or NFS, I would retire it as a service or plugin, at least for Core.. Maybe scale's multi-node model would make it more practical, but a single minio instance running in a Core jail isn't really a practical deployment, given how minio does redundancy and scaling.

There are many tasks that benefit a great deal from storage availability, like databases, web servers, video libraries, video storage, and many others, and all of those are actual jails I have running on my TrueNAS CORE system (MySQL jails, jails for various PHP runtimes that read code & assets from deployments produced by GitLab CI/CD pipelines, which is in itself another set of jails I run, one for the main application itself and others the various runners I use --which also benefit from storage availability--, Plex, Zoneminder, and others).

I initially started with plugins for some of those, but have since been moving them to homegrown jails, of which I could contribute to some kind of community-maintained catalog if time would permit, to help in the effort of keeping alive high quality "plugins" offering for TrueNAS CORE.

If I had my druthers, I'd finish learning Bastille and would put my energy behind that toolset, because I loved the infrastructure-as-code ring it had to it, but the key takeaways here are 1) TrueNAS CORE only offers iocage out-of-the-box, and 2) time availability, of which there aint much currently :(

 

[blanchet]

Having the possibility to run plugins, jails and VM directly on the NAS to support other protocols is a major feature of TrueNAS.

IMHO, TrueNAS is the better storage system for Veeam thank to the virtual machines

• one windows VM to run the Veeam application
• one linux VM to run the Veeam hardened Linux repository

With this setup you have an all-in-one Veeam appliance that supports deduplication (thank to XFS fast cloning).