Are the VMs in Scale Isolated or should I Virtualize TrueNAS Core on Proxmox?

[Jazz30-06]

<div class="bbWrapper">Hello all,<br /> <br /> I finally have hardware sorted for a small TrueNAS box:<br /> HP Z440<br /> E5-1650-V4<br /> 32G ECC @ 2133<br /> GTX 1060 6GB<br /> 2x 4TB WD Red Plus<br /> Crucial BX500 or a Samsung 870 Evo for a boot drive<br /> <br /> I would like to keep TrueNAS isolated from the internet for security purposes, but I would like to run a container or two that would have access to the internet such as NextCloud and the like. I originally started down this road with a laptop trying to run Core in Proxmox with these HDD on an external drive, but that was... problematic. Anyway, I learned some stuff in terms of running Proxmox and all that fun jazz and my understanding is that the VMs run completely independently. I know basically nothing about is how Scale handles VMs besides that they both run on KVM. Does this mean that the VMs inside are secure from the main storage or should I run Core on Proxmox to retain that independence? Or, should I just configure the laptop to run Proxmox to run the other VMs that would be connected to the internet? FWIW, I have a NUC that my old roommate helped set up running a Sophos firewall so, in theory, I am slightly more secure than the home router firewall, but I'm still ver new to all of this, so I am hoping someone can point me in the right direction.<br /> <br /> Thanks!</div>

 

[jgreco]

<div class="bbWrapper">Please see <br /> <br /> <div class="bbCodeBlock bbCodeBlock--unfurl js-unfurl fauxBlockLink" data-unfurl="true" data-result-id="7290" data-url="https://www.truenas.com/community/threads/absolutely-must-virtualize-freenas-a-guide-to-not-completely-losing-your-data.12714/" data-host="www.truenas.com" data-pending="false"> <div class="contentRow"> <div class="contentRow-main"> <h3 class="contentRow-header js-unfurl-title"> <a href="https://www.truenas.com/community/threads/absolutely-must-virtualize-freenas-a-guide-to-not-completely-losing-your-data.12714/" class="link link--internal fauxBlockLink-blockLink" target="" rel="" data-proxy-href=""> &quot;Absolutely must virtualize FreeNAS!&quot; ... a guide to not completely losing your data. </a> </h3> <div class="contentRow-snippet js-unfurl-desc">[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I&#039;ve updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...</div> <div class="contentRow-minor contentRow-minor--hideLinks"> <span class="js-unfurl-favicon"> </span> www.truenas.com </div> </div> </div> </div> <br /> And note in particular that Proxmox is not a recommended hypervisor. It tends to either work well or fail spectacularly, but I would still suggest extensive (like as in month-long) burn-in before trusting a virtualized TrueNAS instance to it.<br /> <br /> As for security, in the '80's and '90's, we forecasted the eventual ability of blackhats to exfil data from systems using nothing more than the blink of an LED, the sound from a speaker, the vibrations of an HDD. Modern blackhats can identify keystrokes by their sound, and all sorts of other nefarious stuff.<br /> <br /> So part of the answer to your question is how secure do you need it to be. As suggested above, even airgapping is not an absolute technique.<br /> <br /> For the average user, I suggest you pull your physical keys out of your pocket and look at them. Got some Schlage, Quikset, Yale on there? These are locks that are easily picked by an experienced person (heck even I can do them with some minutes of effort). If you are just looking to provide basic security, then an OPNsense firewall VM and some proper network bridging design may be sufficient to the task. But just as with physical security, you can get more paranoid. Do you need just a door lock, or do you need a deadbolt too? Wood door? Steel door? Reinforced strike plate? Monitored alarm system?<br /> <br /> There is no such thing as VM's running &quot;completely independently&quot;; they share a compute platform, and that is a targetable vulnerability. See for example <i>CVE</i>-2021-22045. Once you start connecting things together, those connections are a potential risk.</div>

 

[Etorix]

<div class="bbWrapper">VMs are &quot;isolated&quot; on SCALE (KVM) or CORE (bhyve), and it may be easier and safer for your storage to run TrueNAS bare metal and further services as VMs from TrueNAS than to setup everything in ESXi according the above resource.<br /> What services do you want to run, and how? There also was a mention of &quot;containers&quot;.<br /> <br /> But with &quot;only&quot; 32 GB, RAM may be tight to run TrueNAS and extra services on top.</div>

 

[Jazz30-06]

<div class="bbWrapper"><blockquote data-attributes="member: 115" data-quote="jgreco" data-source="post: 719079" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=719079" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-719079">jgreco said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Please see<br /> <br /> <div class="bbCodeBlock bbCodeBlock--unfurl js-unfurl fauxBlockLink" data-unfurl="true" data-result-id="7290" data-url="https://www.truenas.com/community/threads/absolutely-must-virtualize-freenas-a-guide-to-not-completely-losing-your-data.12714/" data-host="www.truenas.com" data-pending="false"> <div class="contentRow"> <div class="contentRow-main"> <h3 class="contentRow-header js-unfurl-title"> <a href="https://www.truenas.com/community/threads/absolutely-must-virtualize-freenas-a-guide-to-not-completely-losing-your-data.12714/" class="link link--internal fauxBlockLink-blockLink" target="" rel="" data-proxy-href=""> &quot;Absolutely must virtualize FreeNAS!&quot; ... a guide to not completely losing your data. </a> </h3> <div class="contentRow-snippet js-unfurl-desc">[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I&#039;ve updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...</div> <div class="contentRow-minor contentRow-minor--hideLinks"> <span class="js-unfurl-favicon"> </span> www.truenas.com </div> </div> </div> </div> <br /> And note in particular that Proxmox is not a recommended hypervisor. It tends to either work well or fail spectacularly, but I would still suggest extensive (like as in month-long) burn-in before trusting a virtualized TrueNAS instance to it.<br /> <br /> As for security, in the '80's and '90's, we forecasted the eventual ability of blackhats to exfil data from systems using nothing more than the blink of an LED, the sound from a speaker, the vibrations of an HDD. Modern blackhats can identify keystrokes by their sound, and all sorts of other nefarious stuff.<br /> <br /> So part of the answer to your question is how secure do you need it to be. As suggested above, even airgapping is not an absolute technique.<br /> <br /> For the average user, I suggest you pull your physical keys out of your pocket and look at them. Got some Schlage, Quikset, Yale on there? These are locks that are easily picked by an experienced person (heck even I can do them with some minutes of effort). If you are just looking to provide basic security, then an OPNsense firewall VM and some proper network bridging design may be sufficient to the task. But just as with physical security, you can get more paranoid. Do you need just a door lock, or do you need a deadbolt too? Wood door? Steel door? Reinforced strike plate? Monitored alarm system?<br /> <br /> There is no such thing as VM's running &quot;completely independently&quot;; they share a compute platform, and that is a targetable vulnerability. See for example <i>CVE</i>-2021-22045. Once you start connecting things together, those connections are a potential risk. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>I'm just shooting for something semi-representative of the security of something like google drive. I am well aware of physical security risks, but there is virtually nothing I can do about that while renting. I just want to have the NAS at least somewhat isolated from a bot attack or the like.</div>

 

[Jazz30-06]

<div class="bbWrapper"><blockquote data-attributes="member: 106202" data-quote="Etorix" data-source="post: 719089" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=719089" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-719089">Etorix said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> VMs are &quot;isolated&quot; on SCALE (KVM) or CORE (bhyve), and it may be easier and safer for your storage to run TrueNAS bare metal and further services as VMs from TrueNAS than to setup everything in ESXi according the above resource.<br /> What services do you want to run, and how? There also was a mention of &quot;containers&quot;.<br /> <br /> But with &quot;only&quot; 32 GB, RAM may be tight to run TrueNAS and extra services on top. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Not shooting for anything too crazy. Just Next Cloud, Plex or similar, maybe some smart home stuff, and some cameras later.</div>

 

[jgreco]

<div class="bbWrapper"><blockquote data-attributes="member: 119397" data-quote="Jazz30-06" data-source="post: 719535" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=719535" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-719535">Jazz30-06 said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> I'm just shooting for something semi-representative of the security of something like google drive. I am well aware of physical security risks, but there is virtually nothing I can do about that while renting. I just want to have the NAS at least somewhat isolated from a bot attack or the like. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote><br /> The main advantage to something like Google Drive is simply that the Goog is likely to be proactive about security issues. This is balanced out by the relatively easy pwnage of G accounts and how difficult it is to recover them if you manage to get fully phished.<br /> <br /> By way of comparison, your local Nextcloud instance will always leave your files in a location where you can lay hands on the server, so you can not easily lose your data to a phish. You can also snapshot your data to protect against certain types of deletion attacks. However, it becomes your responsibility to keep your software - both Nextcloud and TrueNAS - up to date and configured with reasonable security.<br /> <br /> You have to consider what risks you're willing to accept. Personally, I am fine with owning my own problems. Occasionally it is annoying, but usually it works out well IMHO.</div>

 

[Jazz30-06]

<div class="bbWrapper"><blockquote data-attributes="member: 115" data-quote="jgreco" data-source="post: 719538" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=719538" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-719538">jgreco said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> The main advantage to something like Google Drive is simply that the Goog is likely to be proactive about security issues. This is balanced out by the relatively easy pwnage of G accounts and how difficult it is to recover them if you manage to get fully phished.<br /> <br /> By way of comparison, your local Nextcloud instance will always leave your files in a location where you can lay hands on the server, so you can not easily lose your data to a phish. You can also snapshot your data to protect against certain types of deletion attacks. However, it becomes your responsibility to keep your software - both Nextcloud and TrueNAS - up to date and configured with reasonable security.<br /> <br /> You have to consider what risks you're willing to accept. Personally, I am fine with owning my own problems. Occasionally it is annoying, but usually it works out well IMHO. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>If I could host my own email, I would, but I think that is getting too far into opening myself up for risk with very little to nothing to gain. I certainly like the idea of always having access to MY data and trying to share less of it with the mega-corps like google, apple, etc. It's part of why I have largely cut social media out of my life. I don't think I'll have any issue keeping things up to date, but I guess I will find out.</div>

 

[Davvo]

<div class="bbWrapper"><blockquote data-attributes="member: 119397" data-quote="Jazz30-06" data-source="post: 719542" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=719542" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-719542">Jazz30-06 said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> If I could host my own email, I would, but I think that is getting too far into opening myself up for risk with very little to nothing to gain. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>How would that get yourself opened to more risks?<br /> It's more likely that anyone would target ie Google's mail servers than your own.<br /> Plus, you don't give Google any data.</div>

 

[Patrick M. Hausen]

<div class="bbWrapper">I would run CORE on the metal and use VMs and/or jails as fits best. The bridged networking can need a bit of fiddling to get right, but once set up correctly this solution is rock solid. I have several units in production 24x7.</div>

 

[Jazz30-06]

<div class="bbWrapper"><blockquote data-attributes="member: 117605" data-quote="Davvo" data-source="post: 719818" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=719818" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-719818">Davvo said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> How would that get yourself opened to more risks?<br /> It's more likely that anyone would target ie Google's mail servers than your own.<br /> Plus, you don't give Google any data. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>I guess I just assumed they were more capable of dealing with attacks than I could ever set something up to be.</div>

 

[Jazz30-06]

<div class="bbWrapper"><blockquote data-attributes="member: 34727" data-quote="Patrick M. Hausen" data-source="post: 719832" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=719832" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-719832">Patrick M. Hausen said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> I would run CORE on the metal and use VMs and/or jails as fits best. The bridged networking can need a bit of fiddling to get right, but once set up correctly this solution is rock solid. I have several units in production 24x7. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Why CORE? I thought SCALE was the one intended for VMs and such?</div>

 

[Patrick M. Hausen]

<div class="bbWrapper">Because jails? Best container foundation there is. Plus VMs in case some software is Linux only.</div>

 

[jgreco]

<div class="bbWrapper"><blockquote data-attributes="member: 119397" data-quote="Jazz30-06" data-source="post: 719980" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=719980" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-719980">Jazz30-06 said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> I thought SCALE was the one intended for VMs and such? </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote><br /> SCALE supports VM's with KVM. CORE supports VM's with bhyve.</div>

 

[Patrick M. Hausen]

<div class="bbWrapper">To elaborate: VMs in CORE are a bit rough and bhyve lacks features present in KVM or other hypervisors. Some intentionally to keep the code base clean.<br /> <br /> Some of our Windows VMs very rarely shut down instead of rebooting when patches are installed automatically. We &quot;fixed&quot; that by monitoring and operator intervention.<br /> OTOH all Ubuntu VMs in bhyve have been rock solid for years with performance absolutely en par with other platforms. So for my use case CORE is the most versatile &quot;jails plus Linux&quot; platform at the moment. I run only few services in Docker and I use Ubuntu and docker-compose in a VM for that. Other VMs are &quot;heavyweight&quot; full installations like Confluence or SOGo. We also run FreeBSD and poudriere in a VM on TrueNAS because our large TrueNAS hosts are the fastest machines we have and not nearly busy CPU wise.</div>

 

[Davvo]

<div class="bbWrapper"><blockquote data-attributes="member: 119397" data-quote="Jazz30-06" data-source="post: 719979" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=719979" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-719979">Jazz30-06 said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> I guess I just assumed they were more capable of dealing with attacks than I could ever set something up to be. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>That's true, but we can also assume they are way more exposed to attacks than a unknown, personal mail server.</div>

 

[Whattteva]

<div class="bbWrapper"><blockquote data-attributes="member: 119397" data-quote="Jazz30-06" data-source="post: 719980" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=719980" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-719980">Jazz30-06 said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> Why CORE? I thought SCALE was the one intended for VMs and such? </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>I'm running Win 10 VM and a Debian VM on CORE right now just fine. Performance seems to be pretty solid from what I've observed. It has some issues with certain Linux distros (i.e. Debian) during installation with garbled display unless you force resolution down to 800x600. Also, bhyve does not have MacOS support (Well, MacOS is iffy even on Linux KVM, but at least it will still run though). Bhyve also has problems booting any of the non-UEFI Linux distros (usually the super minimal ones like SliTaz). I did have random stability issues though when I only had 16 GB of RAM, but once I doubled my RAM to 32 GB, those stability issues disappeared.<br /> <br /> I generally prefer CORE just cause I prefer FreeBSD base (it's a much more sane OS than Linux IMO), but for VM's, Linux-based solution is definitely more polished and mature. Of course, for a dedicated production level hypervisor though, I would use neither SCALE nor CORE.</div>

 

[Jazz30-06]

<div class="bbWrapper">What about PCIe passthrough? I'm running the 1060 and would like to use it for transcoding and whatever else I can find. It seems that CORE has some issues with passthrough. Has that been solved or am I missing something?</div>

 

[Whattteva]

<div class="bbWrapper">Yeah, for PCIe pass-through, you probably want to go with SCALE. I'm not sure that Bhyve even supports it right now (someone else can correct me on this). I hope it would in the future.<br /> <br /> Disclaimer: I have never run SCALE. This is purely just from my knowledge that it is based on Debian Linux, which does, indeed, have that support.</div>

 

[ChrisRJ]

<div class="bbWrapper"><blockquote data-attributes="member: 117605" data-quote="Davvo" data-source="post: 720009" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=720009" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-720009">Davvo said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> That's true, but we can also assume they are way more exposed to attacks than a unknown, personal mail server. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Define &quot;exposed&quot;. These attacks, unless we are talking about a targeted operation, run in an automated fashion, just like sending out spam emails. My gut feeling tells me that it wouldn't take a day (probably less than 3 hours) until I was under attack.<br /> <br /> If you are in the mood, why not set up a honeypot system and see what is happening?</div>

 

[Davvo]

<div class="bbWrapper"><blockquote data-attributes="member: 104866" data-quote="ChrisRJ" data-source="post: 720084" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=720084" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-720084">ChrisRJ said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> If you are in the mood, why not set up a honeypot system and see what is happening? </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>I would like to, but don't have the resources or the knowledge to (never created a honeypot, is it just a normal server in a dmz?).<br /> Btw, Sorry for hijacking the thread.</div>