Accessing NAS from VM Still Broken without Bridge

[NickF]

Ahh. There's the problem. You seem to think TrueNAS is a "normal Hypervisor". It isn't. It's a NAS platform, which is optimized towards serving files. This collides with a completely different role of acting as a hypervisor, where hypervisors typically do include bridges by default because that is their raison d'etre. However, hypervisors also don't generally include NAS functionality, so you're failing to account for the TrueNAS developers aiming to develop a product that serves its primary role to the best of its ability. I suggest you go back and read this entire thread in its totality.

Historically, I would agree with your assertion here. But the entire reason why SCALE exists in parallel to CORE is because SCALE is focused on HCI while CORE keeps it's focus on storage.

So expecting it to function like other hypervisors is not really a stretch IMO. While VMWare VSAN is the closest example I can draw, the guests in a VMWare environment can absolutely ping their hosts.
You can also make file shares like TrueNAS: http://www.vmwarearena.com/create-a-vsan-file-share/#:~:text=To Create a vSAN file share, Login to,Enter the name for the vSAN file share. A VSAN is not far off from a scale-out ZFS cluster with Gluster on top.

 

[jgreco]

the guests in a VMWare environment can absolutely ping their hosts.

Only in insecure environments where you have let vSwitch0 handle both the vmk0 interface and also guest interfaces. I operate lots of VMware environments and your statement is false in ALL of them. It's a huge security hole.

SCALE is focused on HCI

Perhaps it will eventually be, but right now it is still a NAS platform. If you look at all of the stuff on the main GUI, this is clear. SCALE was done on Linux because some of the technologies such as Kubernetes were basically not possible to do on CORE. Given that both CORE and SCALE have usable virtualization stacks, I wouldn't read too much into what the intent was there.

 

[ptyork]

Honestly, I think that easy access to the host makes even MORE sense if the primary purpose of the system is to be a NAS. If that's the case I can see really only two purposes to use the virtualization stack on TNS.

1) You're a relatively small shop and this is maybe your only on-prem server. You spin up a couple of VMs for, say, running your BlueIris NVR and perhaps a back-end database or two for some applications. Whatever. In such cases, easy and direct access to host storage would make a ton of sense. ESPECIALLY if it doesn't need to use of NFS...but I again digress.

2) You're a larger shop with a number of servers already. Likely they're already running VMWare or Proxmox or XCP-ng or another far more mature hypervisor. And TNS is really meant to be for shared storage. And if that's the case, then the only real reason I can see to leverage TNS virtualization is to co-locate some very specific apps with the data they depend on, minimizing latency and network traffic. And thus having easy and direct access to the datasets once again just makes sense.

This is ignoring the potential of the system as the foundation for HCI (or hyperconverged storage or whatever the buzzword is). However, I think that's a far different use case involving multiple nodes, TrueCommand, Gluster, etc. No matter how great TC+TNS is, you're already adding a ton of complexity, so perhaps easy access from VM to host might be the last thing you're concerned about. ;)

 

[jgreco]

Honestly, I think that easy access to the host makes even MORE sense if the primary purpose of the system is to be a NAS.

Yes, but it can significantly impact NAS performance, and the way that it would need to be set up for "easy" default use conflicts significantly with more complex configurations. It's pointless to continue to argue for this unless you have a suggestion that would allow this to be implemented in a flexible and efficient manner. The above discussion already touches on all the relevant options and does not yield a clear path to doing this.

 

[ptyork]

Yes, but it can significantly impact NAS performance, and the way that it would need to be set up for "easy" default use conflicts significantly with more complex configurations.

I'm kinda confused about what would impact NAS performance? An optional "easy switch" in the GUI to create and assign a bridge? A translation layer like virtio-9p that simply translates IO from a virtual guest disk to a host disk in order to bypass the network?

Sure, having a bridge be the default might add a fraction of a percent overhead to network throughput, but having it as an "easy" option would do nothing but make things "easier" for the end user. And the negative TNS is getting from the 'tubers might well make even that fraction of a percent penalty worth bridging the physical interface by default.

It's pointless to continue to argue for this unless you have a suggestion that would allow this to be implemented in a flexible and efficient manner. The above discussion already touches on all the relevant options and does not yield a clear path to doing this.

I'm confused. I think @NickF suggested at least two viable options.

I also think I pointed to a very viable option. Better for most uses, I think, because it entirely bypasses the networking stack and the related issues. There aren't many NAS OS options that also serve as hypervisors, but I know for sure that Unraid offers this solution exactly (https://wiki.unraid.net/Manual/VM_Management#Advanced_Options) and Asustor uses VirtualBox which allows for host folder passthrough. (as an aside, Synology doesn't, though they also don't really operate as a hypervisor, instead installing one or more instances of the NAS OS as clients onto a base hypervisor alongside any other hosted VMs...I found that interesting) I don't pretend to know if 9p is great or if perhaps there's an issue with 9p + ZFS or something, but FWIW it's part of the QEMU package (https://wiki.qemu.org/Documentation/9psetup) and the client is part of the Linux kernel (https://www.kernel.org/doc/html/latest/filesystems/9p.html).

And money-where-mouth-is, I actually AM a developer and would consider digging in deeper and even taking a stab at hacking on a new "Passthrough Folder Device" option if iXsystems was earnestly interested in the contribution and willing to work on integrating it into the product.

Regardless, I think most of us are just "arguing" for anything better than "search 10 forum posts to figure out how to share files from a host to a VM since it's not in the create wizard; debate on the merits of SMB vs. NFS; decide on NFS; set up some shares; try for an hour to debug why your mount command you put in that /etc/fstab file is failing; finally realize your VM's can't see your host; search through docs and find an article that discusses this; try to create a bridge and fail; search through another 20 posts; find a command-line option; finally can ping the host; finally get the mount to work; oh crap, what's all this weird stuff with UID 1000; shrug and go home to drink a few fingers of scotch wondering where half your day went." Obviously much of the preceding isn't an issue for an expert. But then, I don't think "expert" is the sole target demographic for iXsystems.

TBH, for now, I'm fine with TNS as a VM on Proxmox and just using it for NAS duties. I like the interface and I like watching it "grow," but I don't use the hypervisor or even Kubernetes for anything beyond just tinkering at the moment. It doesn't yet fit my needs. It's in a home lab. I'm not a paid user. I don't really have a meaningful horse in the race. However that doesn't stop me from wanting to try to help TNS improve and hopefully evolve into the true "holy-grail" solution that it has the potential to be. And I bet for most of us, that's all this is--just trying to help. Even if the tone doesn't always come across that way.

 

[urza]

Ahh. There's the problem. You seem to think TrueNAS is a "normal Hypervisor". It isn't. It's a NAS platform, which is optimized towards serving files. This collides with a completely different role of acting as a hypervisor, where hypervisors typically do include bridges by default because that is their raison d'etre. However, hypervisors also don't generally include NAS functionality, so you're failing to account for the TrueNAS developers aiming to develop a product that serves its primary role to the best of its ability. I suggest you go back and read this entire thread in its totality.



Sure. You've got unreasonable expectations. The feature doesn't do exactly what you think it does, just like Tesla Autopilot doesn't do what many people "expect" it to.

Then this should be communicated more clearly. The promotional materials by ix about Scale have Linux KVM virtualization as one of their main features. It's in every graphic and table.

I would also understand explanation that it is just "young product" and devs will work on it. What I don't understand very much is agressive takes like yours, dismissing the problem.

This problem has been reported by numerous people, here, on Jira, even famous youtubers like Wendell and Lawrence mention this issue explicitly and repeatedly. Many people wasted many hours or days with this. There are whole workaround tutorials about this issue. So it woud seem, it is you, who has misjudged the importance of this issue and you would serve better the project if you would translate the frustrations of MANY people on this issue to develeopers, and urge them to prioritize this, since it looks like you are close to them.

 

[jgreco]

I'm kinda confused about what would impact NAS performance?

A number of users have been advocating a default bridge configuration. The network bridge device adds an extra layer and also some new potential failure modes because packet loss shows up with some types of ethernet interfaces under heavy load as well. The extra virtual network layer results in performance loss, as does any retries due to packet loss. The latter might not be a problem if we could get all users to use non-problematic ethernet interfaces.

Sure, having a bridge be the default might add a fraction of a percent overhead to network throughput,

I've been working with networking on FreeBSD for more than a quarter of a century, and I have not noticed it to be "a fraction of a percent" (deemed to mean something between 0% and 1%). It can be substantially greater.

And the negative TNS is getting from the 'tubers might well make even that fraction of a percent penalty worth bridging the physical interface by default.

I kinda doubt that YouTube videos are going to drive such an impactful design decision. Again, not a "fraction of a percent penalty." Once you get into the higher capacity interfaces such as 10/40Gbps, every little bit hurts.

I'm confused. I think @NickF suggested at least two viable options.

I've suggested one as well. Unfortunately, there is not a clear winner that works in all cases. Actually implementing any of this crap means that you need to have something that handles, at a minimum:

1) Multiple physical interface concerns, which should theoretically be handled by the LACP layer, but we do know that some users use bridging as an alternative to buying a 10G switch

2) Multiple bridged network setups, where you have more than one bridge in order to implement more complex topologies,

3) VLAN setups, which apparently can be made to work either underneath or on top of a bridge, so which one should be the default,

4) Jumbo MTU setups, where the stack needs to be set up to handle jumbo up through all the necessary layers.

Quite frankly, I'm amazed that they've gotten the stuff that they have for network setup to work as well as it does. I have not seen any viable options that cleanly cover all deployment models. And remember, where a model isn't covered, the system shouldn't get in the way of someone who has to set up that model by trying to force an incompatible model on them. This makes it very complicated because the potential layering options in the network stack are VERY flexible. Some of us watched for years as the developers tried to grapple with the combination of LAGG and VLAN, for example.

Regardless, I think most of us are just "arguing" for anything better than "search 10 forum posts to figure out how to share files from a host to a VM since it's not in the create wizard; debate on the merits of SMB vs. NFS; decide on NFS; set up some shares; try for an hour to debug why your mount command you put in that /etc/fstab file is failing; finally realize your VM's can't see your host; search through docs and find an article that discusses this; try to create a bridge and fail; search through another 20 posts; find a command-line option; finally can ping the host; finally get the mount to work; oh crap, what's all this weird stuff with UID 1000; shrug and go home to drink a few fingers of scotch wondering where half your day went." Obviously much of the preceding isn't an issue for an expert. But then, I don't think "expert" is the sole target demographic for iXsystems.

This could be read as a strong argument in favor of limiting the options available in order to reduce the complexity of potential configurations. I'm not sure that's a good idea.

My solution to this has been to generate documentation. I believe my posts currently make up... maybe half? Of the resources section. I try to make difficult topics accessible. There's lots of problems. Got a Realtek ethernet? Got a RAID controller? I can tell you why these things won't work correctly and why they should be avoided. Want to virtualize TrueNAS? I can tell you how to do so successfully. It should be clear that I've got my own ideas about how to move the project forward and keep people from dangerous off-road misadventures. But I don't use CORE or SCALE for virtualization, so I don't really have any documentation for this. Someone else could potentially step up and handle this, though. Expecting the developers to solve every problem may be unrealistic.

 

[alpha754293]

I'm just going to dive on in with my $.02 here:

virtio-fs (or virtfs as @ptyork refers to it) might not have been around for very long (since 2018 according to this presentation from Stefan Hajnoczi, Senior Principal Software Engineer, RedHat: slides (pdf)), but it looks like that it was built on top of virtio-9p, where also according to the same presentation, states that active development of 9p ceased in 2012.

Thus, to the question of "is it production ready?" - I don't know if there are companies that are actively using this, but being that it is developed by the folks at RedHat, I would guess that there might not be very many reasons why it can't be used for production.

Again, referencing the same presentation and also a presentation that Stefan gave at FOSDEM a year later, (video), he also talked about some use cases beyond a home lab/NAS environment, which might suggest that either they are using it themselves or that they have clients who might be using it.

I read through the thread here.

And whilst I understand how some people would most definitely WANT to keep the VMs completely separated and isolated from each other, all the way to the CPU cache levels, there are other use cases (not only for the homelab, but also outside of the homelab environment) where companies and/or people might want to use this.

I can't speak for the companies, so I'll narrow down the scope by speaking for myself:

I'm currently in the middle of a massive migration project where I am consolidating 5 different and separate NAS server down to one.

Right now, I am using Oracle VirtualBox as my "hypervisor" (running on top of a Windows 11 host, which runs on a Beelink mini PC). The Oracle Virtual Box VM disks are hosted on one of my QNAP NAS units.

Media is stored on another (which also runs Plex) and applications run on another, etc. etc. etc. You get the idea.

So, right now, this massive migration projection (it's massive to me), is to pull/suck all of that up into a single 36-bay 4U server, and all of those things that I used to do with my QNAP NAS systems will now run as VMs.

And then on top of that, I'm also virtualising my gaming tower systems as well, with GPU passthrough.

In my homelab environment, all of the systems talk to the various NAS systems, fairly regularly, so they all talk to each other.

When I migrate the QNAP systems over, whilst I COULD have set up a NFS and/or a SMB/CIFS share and then have it route through the virtio NIC (which is a 10 Gbps NIC), if I can SKIP the network stack entirely, that would make things run faster by having a more direct access to all of the data that the hypervisor host is now hosting -- i.e. TrueNAS SCALE.

It's a NAS first and the hypervisor is built on top of it so that I can create one ZFS pool (which also leads to better overall utilisation of my available hard drive space vs. it being dispersed into 5 separate NAS servers), and then have the VMs interact with said ZFS pool directly.

This is where virtio-fs comes into play.

As a part of this massive server migration project, I've been testing out xcp-ng, Proxmox, and TrueNAS SCALE.

xcp-ng can't do virtio-fs at all (despite it's CentOS origins), so that got eliminated.

Proxmox can do both virtio-fs and GPU passthrough, but it's not exactly straightforward, but there are enough forum posts that you can eventually cobble together the "How To" deployment instructions.

TrueNAS SCALE made GPU passthrough practically a trivial task, but unfortunately, because I have no direct way of interacting with the ZFS storage pool that was set up by TrueNAS -- the lack of immediate virtio-fs support made it such that, unfortunately, TrueNAS SCALE eliminated itself from this competition between the different solutions for this server consolidation/migration project.

I was hoping that, since I was already learning how to enable virtio-fs in Proxmox, that I was going to be able to take the <<VMID>>.conf files and be able to use some of the stuff that's in there to be able to enable virtio-fs in TrueNAS SCALE. And then that's when I smacked the wall of the websocket API protocol.

Whilst I understand why there are use cases where you wouldn't WANT to allow this to happen, but the way that the current Hypervisor is set up in TrueNAS SCALE also makes it very difficult to enable virtio-fs if you aren't a programmer/developer who understands how the websocket API works (which I, unfortunately, don't).

So, it would be nice that for those of us who want to consolidate multiple servers down to a single server like this, enabling virtio-fs (or at least giving us this option to enable it, if we want to), would make TrueNAS SCALE an even easier platform to use than Proxmox.

In my testing of Proxmox, I had to create the ZFS pool via CLI myself. (Because if you do it through their GUI, you can only put certain types of content in it for some inexplicable reason.). And then I also had to set up the iSCSI target via the CLI as well, along with a NFS export, and also a SMB/CIFS share.

These are all of the things that TrueNAS is already GREAT at doing.

Now if I can just access a ZFS pool made of NVMe SSDs at speeds faster than 10 Gbps (because that's the fastest speed that the virtio NIC allows), that would've made TrueNAS SCALE amazing.

And from reading the forum posts, it would appear that I am not the only who wished that TrueNAS (SCALE) enabled virtio-fs.

(And from my testing with it in Proxmox, it seems quite stable. I haven't noticed any major issues with it so far. I need to test out the Windows regkey edit that Xiaoling Gao (virtio-fs team, RedHat) sent to me so that I can get the Windows 10 VM to "auto-mount" the virtio-fs share on something OTHER than the Z drive.)

But beyond that, for the systems where it works, it's been working great! I was getting, I think something around like 236 MB/s writes in an Ubuntu VM running on Proxmox when it was interacting with a virtio-fs share on the Proxmox host on a single HGST 3 TB SATA 3 Gbps HDD. (Core i7-6700K, Asus Z170-E motherboard, 64 GB of DDR4-2400 RAM).

 

[icemannz]

Hi All,

I just wanted to circle back to this since I haven't seen any update on this topic in a while. Is there an actual ticket number I can reference @morganL ?

Tom Lawrence (@LawrenceSystems on here) Brought this up in his video on Bluefin today (and in previous videos):

TrueNAS SCALE 22.12.0 Bluefin Is a Whale of an Update!

TrueNAS SCALE Beta vs TrueNAS CORE Performance Comparisonhttps://youtu.be/uFJaYXUWEb8Release noteshttps://www.truenas.com/docs/scale/scale22.12/Adding admin ...

www.youtube.com

And Wendell from Level1Techs brought this up back like 4 months ago:

TrueNAS: Full Setup Guide for Setting Up Portainer, Containers and Tailscale #Ultimatehomeserver

Thank you to Fractal for sponsoring this video! Check out the Fractal Meshify 2 Lite here: https://www.fractal-design.com/products/cases/meshify/meshify-2-li...

www.youtube.com

Basically, I am just complaining that VMs can't access the NAS they are running on without a network bridge (most folks followed this: https://www.truenas.com/docs/scale/scaletutorials/virtualization/accessingnasfromvm/) being manually created, or unless they go out to the switch and back over a different VLAN.

In my configuration I am doing the latter currently, which I think is totally silly. Looking at the past 24 hours, 5 out of 6 of the top addresses talking THROUGH my switch back to my NAS are VMs running on my NAS.
View attachment 60988

If I expand the resolution out to a month, my Plex server literally used a Terrabyte of network bandwidth to talk to the server it lives on.....

View attachment 60989
What's going on? Is this really not something folks other than me and a few tech YouTubers care about?

Hi, I had this issue and was able to get it working.
I have two network cards in my Truenas Scale 22.12.0 machine.
I noticed that the virtual machines could not talk to the Truenas host and started playing around with the settings.
My Truenas host shows two network cards and two different ip addresses.
Both cards on on the same LAN.
The network cards are shown as enp4so and eno1.
It depends on which network card i select for the virtual machine to use,
if I select the first one -the virtual machine will work and run well but it will not be able to communicate with the host on either of the host's ip addresses.
If I select the 2nd network card for a virtual machine, it will then be able to talk to the host machine using the ip address from the opposite network card.
So my virtual machine is now connected to the eno1 netwrok card and it can ping and communicate with the host via the ip address assigned to the enp4so card.
Hope that makes sense and hopefully it may help someone else.

 

[Patrick M. Hausen]

Two interfaces in the same subnet is not a valid configuration. You can probably remove the IP address or DHCP from the interface you use for VMs.

 

[-jim]

There should be a checkbox in the VM to allow access to the NAS.
Most user for TrueNAS are not large enterprises and this is a "pretty basic expectation" as pointed out by MANY of your user's.

 

[jgreco]

There should be a checkbox in the VM to allow access to the NAS.

The problem here is that this is much more complex than a mere checkbox; your question misunderstands the issue as though there was something forbidding access to the NAS. There isn't. What you need is for there to be a certain virtual network setup on your NAS that is appropriate to your environment. This COULD be just a simple bridge to the primary ethernet that the NAS is using, but it could also instead be a bridge to a secondary ethernet dedicated to jails/VM's, or a bridge to a link aggregation, or a bridge to a particular VLAN, or even a bridge to a particular VLAN that's configured on a given link aggregation. The FreeBSD and Linux networking stacks allow much more complex network configurations than your typical Synology or QNAP boxes. What's probably necessary for TrueNAS is a visual network editor, because it's pretty clear that the current network configuration tools are extremely difficult for the average user to navigate even without the complication of bridging. Some years ago, I worked with the cloud engineering group at a high end colo/cloud firm and they had a very cool visual network design tool so that you could see how your VM's were arranged. The general absence of this is one of those things that makes setting up networks very difficult for non-network-gurus, and leads to this misunderstanding that it's just a matter of some checkbox somewhere that would make this work.

 

[Nicholas Flamy]

Y'all, could we simply get a popup that warns about this and links to the documentation about accessing TrueNAS from a VM where it explains how to create the bridge, when you open the VM page?

 

[sfatula]

I will merely add that creating a bridge interface via 23.10.2 was significantly easier than on previous non Cobia versions. I didn't encounter any bugs, didn't have to use the console, etc. It was < 1 minute effort.

 

[Nicholas Flamy]

I will merely add that creating a bridge interface via 23.10.2 was significantly easier than on previous non Cobia versions. I didn't encounter any bugs, didn't have to use the console, etc. It was < 1 minute effort.

I agree.

 

[Nicholas Flamy]

I agree.

Actually, I have had the problem of apps not starting because the bridge interface was not found because the apps start on boot before the bridge is created or something. I have to manually choose apps pool so that it will start the apps.