Migrating FreeNAS 11.2-U7 to ESXi 6.5

[kargnos8]

I'm taking on a fun project at home and am just trying to get ahead of some steps before my new hardware arrives next week.

Question: What's the best way to migrate my existing physical FreeNAS 11.2-U7 server to a new FreeNAS VM

Background:
Existing FreeNAS is on a dedicated 9-year old PC. Boots off of USB. Uses onboard SATA for existing 5 x 3TB HDDs. Currently stores movies and system backups for existing VMs and my home PC. 1 Volume, 2 Datasets, 1 Jail (Plex), connected to Active Directory for user/group auth. Is there anything else that would be helpful to know?

Future:
I've been reading through a few threads to prepare for this project...

FreeNAS virtualized on ESXi or VM's in FreeNAS?

At this moment I am running FreeNAS (11.0-U3 Stable) as a fileserver and as backup for a couple of workstations. No Jails or VM's. The hardware I am running it on is more than capable of doing that job: SuperMicro X10SL7-F - On board SAS2 controller flashed to IT Mode Edit: Intel Core i3-4130T...

www.ixsystems.com

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]

So you end up with them both virtualized and that isn't an issue at all? How much work is it to make this type of setup actually work, and what spec changes would I need to make it not slow and suck? Sent from my iPhone using Tapatalk The biggest issue is that if you have to restart/upgrade...

www.ixsystems.com

With the above threads, I am pretty confident I have a working configuration:
Motherboard: ASUS ROG Strix X470-F Gaming
CPU: Ryzen 5 3600
RAM: 2 x 16GB DDR4 3600
ESXi Boot: USB
ESXi Datastores: 2 x 500GB Samsung 970 EVO Plus NVMe
FreeNAS Passthrough Controller: LSI 9207-8i IT-mode V20
NIC: Intel PRO1000/PT Dual-port PCIe
Case: Silverstone CS380B
Power: Corsair RM650x

My Plan:

• Build ESXi host with 6.5-U2 and get all hardware functional (including SAS passthrough)
• Create FreeNAS VM per threads above
• Backup config from existing physical FreeNAS
• Restore config to new FreeNAS VM
• Import Pool

Thoughts/Concerns:
Do I need to export the pool before physically moving the disks between the 2 systems?
Does the backup contain jail config info too?
Iocage is on the 1 volume that is coming over with the disks, so can I assume that the jail will work once the volume is reconnected?
Is there anything else I should be considering or preparing for?

Thanks in advance!

Mike

 

[blueether]

I can remember if ESXi can pass through a single disk, but if it can I would pass that through to the VM for it's boot disk and install Freenas on there. This is how I run my freenas (on 2 servers) with proxmox. With this config I can pull all disks and place in another box (or pull the proxmox boot drive) and still have a functional freenas box with any jigery-pokery to get to work. I think one of the articals on virtulisuation covers this way.

So I would get freenas working on new hardware then get ESXi working on same hardware and then passthrough all the disks/controlers to a VM

 

[kargnos8]

Wanted to provide a quick update and see if anyone had thoughts on some troubleshooting...

So to start...thanks @blueether for the suggestion. I wanted to try and virtualize FreeNAS as part of the main datastores instead of a passthrough, primarily to keep everything consistent. This machine is going to run another 4-5 VMs, so wanted to keep everything altogether.

I was able to follow the documentation from my original post and was able to get everything to work. The ESXi setup was no problem. Created the virtual disk as 16GB lazy zeroed (which got rid of some error, though I don't notice any performance difference). I reserved the memory at 8GB (for now). Setup the 9207-8i in passthrough mode, everything looks fantastic. Went through a fresh install of 11.2 U7, clean as a whistle. Did some basic setup stuff, restored my config from my old physical box (and I mean AMD Neo K625 old), rebooted, fixed a couple minor things (so minor they don't stick out), and my volume came back online. Everything looks beautiful!

3 things that required more in depth troubleshooting...2 of which is still not resolved so am looking for help.

Resolved issue:
My Plex Media Server jail started just fine, but I couldn't connect or ping (in or out). I tried promiscuous mode both at the vSwitch as well as the port group level...no luck. Finally, I just decided to try the Plex plugin and see what that did. Much to my surprise, it actually worked! Of course, it created a new jail, so none of my PMS config was there. In taking a closer look at what the plugin did...it actually unchecked VNET and did not use vnet0 as the adapter. Instead, it used vnx0 which is my physical adapter. I uninstalled the plugin, went back to my original jail, made those changed and PMS came alive. SWEET!

<EDIT> Disabling VNET, while making the networking completely stable, prevented clients from playing movies that needed on the fly transcoding. This is a known issue or requirement from Plex (https://forums.plex.tv/t/pms-not-sending-transcoded-files-to-clients/390867/17). Turning VNET back on and enabling Promiscuous Mode and Forged Packets at the Port Group level allows clients to play, but network reliability suffers. Still searching for an answer...could just be my setup.

Unresolved issues:
FreeNAS Dashboard just spins. No metrics other than volume capacity and disk temp. Read through various threads, but none have helped. When I booted FreeNAS from bare metal, dashboard worked fine. Well, everything except CPU temp which read at -270 degrees. Haha! I'm fine with the dashboard not working, but would be great if there was a solution.

The bigger issue...Plex Media Server transcoding is unusable. While I use DirectPlay for pretty much everything at home, I know I will need to transcode when I am remote (which was actually the main reason I started this project since my old CPU's Passmark was like 977 with a single core score of 510). I figured with a 6-core Ryzen 5 3600 that has a Passmark score of nearly 20000 and a single thread score of 2800, this would be amazing...even if there was a litttle virtualization overhead. Yeah, not the case at all. Steps I've taken:

• Booted FreeNAS from bare metal. 1080p 20Mbps transcoding screams. Probably a 5 second pause to load the movie and then boom, fires right up. No buffering, no stutters, plays perfectly. CPU utilization hits about 100% for a little bit, but then drops off. Problem DEFINITELY seems to be with FreeNAS in a VM and Plex running in a jail.
• Loaded back up as a VM...
  • Tried every type of transcoding all the way down to 240p and the movie never plays. Just sits there and crushes the vCPU.
  • I started with 4 vCPUs...all pegged at 100%, no playback
  • Tried setting CPU affinity in ESXi...those 4 vCPUs pegged at 100%, no playback
  • Moved down to 1 vCPU...pegged at 100%, no playback
  • Memory use is as expected. Disk and Network show no activity.
  • Tried checking "expose hardware assisted virtualization..." ESXi returns an because passthrough is enabled
  • Tried checking "expose IOMMU to the guest OS" no effect.

I realize this is probably more of a Plex issue, but wanted to toss it out to the FreeNAS group to see if anyone else has tried to load plugins or apps inside of jails and ran into CPU performance problems. I am going to post this to the Plex forum too.

Thanks in advance for any help!

Mike

 

[Dice]

Problem DEFINITELY seems to be with FreeNAS in a VM and Plex running in a jail.

Have you looked into bios options for nested virtualization?
I can't recall the terminology for AMD.


Regarding the migration part, I'm not sure if it still applies (I'm slightlly FreeNAS rusty since a few patches), back then it was sufficient to import the config file, make sure the drives were connected and then everything else followed suit.

I expect that behavior still, as I plan to migrate from ESXi to proxmox very soon.

 

[kargnos8]

Yes, enabled SVM mode in BIOS. This enables AMD Virtualization.

As for the migration, yeah was piece of cake. Had to reset the networking and then update the Plex jail, other than that, restoring the config brought everything back.

Another thing I noticed...during transcoding, my plex jail grows in size VERY VERY slowly. I'm assuming this is the temp file of the transcoded video that's getting created. After nearly 20 minutes of transcoding to 720p 2Mbps, the plex iocage has grown by about 50MB. Having a hard time figuring out where that growth is exactly located. As soon as I stop transcoding, the space starts to go back down.

 

[jgreco]

These guides are still the gold standard.

https://www.ixsystems.com/community...ide-to-not-completely-losing-your-data.12714/

https://www.ixsystems.com/community...ative-for-those-seeking-virtualization.26095/

If you're not doing one of these, or something very close to one of these, you're likely to get pants'ed at some point.

 

[kargnos8]

Quick update...

Upped vCPUs back to 4 and also added a new virtual disk to freenas VM so I could create a Transcoder temp directory that was off of my passthrough controller. Now I have 1200 .ts file sitting on flash and after about 10 mins of transcoding, I have nearly 1GB of transcoded data. I guess I don't know what is typical here but I thought I would post.

 

[kargnos8]

These guides are still the gold standard.

https://www.ixsystems.com/community...ide-to-not-completely-losing-your-data.12714/

https://www.ixsystems.com/community...ative-for-those-seeking-virtualization.26095/

If you're not doing one of these, or something very close to one of these, you're likely to get pants'ed at some point.

Thanks @jgreco ... I will check both of those now just to make sure I'm doing this up to the gold standard.

 

[kargnos8]

OK...I've more or less isolated the problem for anyone following...

In order for Plex in a jail on FreeNAS to play movies that are transcoded, the jail's networking has to use VNET. From all of my testing...I can tell you that the actual playing of the movie, from what I can tell, is the only thing that doesn't work. Plex is online. DirectPlay works fine. Transcoding actually happens. But the ability to actually send the movie to the client doesn't work. I've sent that to Plex for any updates.

So in my first update, I said that I was able to get Plex to work by unchecking VNET. I will edit that next.

So, if you follow the links from my first post as well as from @jgreco, you know that in order to get VNET to work when FreeNAS is a VM, you have to enable Promiscuous Mode and Allow Forged Packets on either the vSwitch or the VM Port Group. When I did my original build, this is exactly what I did. The problem was that I had a tremendous amount of packet loss (60%-80% loss) when trying to ping the jail. Outbound ping from the jail to the FreeNAS VM was great. Ping to the host was terrible. Ping from my desktop which is wired 1Gbps is worse. And the Plex Media Server looks like it is up and down when managing it over the internet.

So that's where I'm at now. I am going to play around more with the ESXi networking to see if there's something there getting in the way.

That all said, I was able to successfully watch a 720p transcoded movie without a problem. 4 vCPUs get hit pretty hard initially, but within about 2-3 minutes, they settle in to a low % utilization.

If I can get the traffic reliability up and the Dashboard to actually display stats instead of just spinning the whole time, I'll be super happy. Quite a fun project!

 

[kargnos8]

I've played with just about every setting I can think of on the networking side of ESXi stripping this all the way down to just a single attached port using no NIC Teaming...nothing seems to change the network stability. After some reading, I decided to run iperf3 to get an idea of what performance would look like.

192.168.10.18 is my FreeNAS 11.2-U7 VM

192.168.10.19 is my plex jail. You can see performance is all over the place, from really high, to somewhere in the middle, to complete lack of transmission

Probably just more of a datapoint for anyone that is following that might have any suggestions. Seems very much like an iocage/VNET on ESXi 6.7 U3 issue.

 

[Dice]

Ugh, that last graph did not look appetizing.

Thank you for sharing.

 

[kargnos8]

Probably my last update for now as I think things have settled in...

First off...I removed Plex from FreeNAS. I could not get the VNET to perform and was just tired of seeing my Plex Media Server bounce up and down from my clients. I was also tired of seeing my FreeNAS vCPU get spun up when running iPerf. There is definitely something going on with VNET where it's crushing the CPU to virtualize the network stack. I love the consolidation idea, but for now, I can live with separate VMs. I installed a small Ubuntu VM, gave it some vCPUs for transcoding, migrated the PMS info over and I'm up and running at full speed. Still using FreeNAS as a VM on ESXi 6.7-U3, dropped it down to 1 vCPU, gave it 12GB of RAM and she looks pretty happy. Except for the dashboard. That is still broken under 11.2.

I think the dashboard is fixed in 11.3! In my troubleshooting, I built a test FreeNAS VM and upgraded it to 11.3 RC. It didn't seem to help the VNET issues, but my dashboard looked a LOT better. I'm looking forward to upgrading to 11.3 when I get back from vacation!