Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]

[ChrisReeve]

I wanted to do an update here, with my experiences/troubles with virtualizing freeNAS.

First of all, I upgraded to the following specs:
MB: Supermicro X9DRL-3F
CPU: 2x Intel E5-2650 v2 (16C/32T total)
RAM: 128GB 1600MHz DDR3 ECC
OS Disk (ESXi + VMs, including freeNAS): Samsung 850 EVO 250GB
Main pool: 10x10TB WD Red (White labels, shucked from WD External drives). (8 drives connected via on-boad SAS-controller, 2 via LSI 9211-8i, both passed through to freeNAS)
Cache-card: Intel DC P3700 400GB (hw-passthrough to freeNAS, with 20GB SLOG, 128GB L2ARC, 16GB Swap) (NOT IN USE! See below)
NIC: Intel X540-T2 dual 10GbE

First time setup took a lot of time, but was eventually successfull. I didnt have to do anything to be able to pass through the on-boad SAS-controller, or the SATA-controller, but I left the SATA controller for my ESXi drives. This left me with (only) 8 drives connected via the MB. The last two had to go through my HBA, which also showed up as available for passthrough on first boot. I also passed through my Intel DC P3700 400GB, partitioned to two partitions (256GB L2ARC, 20GB SLOG, rest for OP).

Basically, no SSH-ing needed. I was also able to set up NFS share and iSCSI, which showed up in ESXi, and I installed a few VMs on it.

Then I started getting the following error in freeNAS about once every 60 seconds (on the VM command line itself, no errors appeared in the web GUI):

Code:

nvme0: Missing interrupt

This didnt affect performance or anything else, bot freeNAS, and VMs running off it worked fine, when I ran CrystalDiskMark on a Windows 10 VM installed on the iSCSI partition, it reported close to 1,5GB/s sequential reads and writes (1GB test file), but when I tried to shut down all the VMs, then I was unable to do a soft power off of freeNAS. Had to force it to power off.

I force rebooted freeNAS in ESXi, and I got it to boot, with several errors, and the pool showed up as healthy, but with 0 bytes available, which worried me. I shut it off again, rebooted my server, and barebone booted into freeNAS (thanks again to the guide and mirror with USB key), and the pool showed up fine. I detatched both my SLOG and L2ARC, shut down, rebooted into ESXi, and started freeNAS as a VM again, removing the hw passthrough of my Intel DC P3700. Now things worked again.

I have been able to reproduce this error (running ESXi 6.7), and I have tried the following fix without success: https://redmine.ixsystems.com/issues/26508

I decided that even though it is cool to be able to have high performance VMs running off my pool, it isn't necessary for my use case, and I dont even gain any performance for my use case with L2ARC/SLOG, besides from running VMs off it. I decided to give up (for now) on running the DC P3700 in virtualized freeNAS, and just run VMs from SSDs directly.

Because I am new to this, and because I had screwed around in freeNAS, I decided to delete it, and start fresh. I installed freeNAS 11.3RC, without DC P3700 this time, and it has been running flawlessly since then. High performance, and no errors (allthough I have just been running it for a few days).

With this solution, I dont have to hassle around with scripts to start/stop/ VMs, and refresh iSCSI mounts etc. Just keeping it simple for my use.

I wasn't able to run plex inside virtualized freeNAS. Unable to install as a plugin (possibly known bug). I passed through a physical NIC (one of the onboard 1GbE), and was able to install the plugin successfully. Plex went online, and was able to mount my media folder, and scan it, but for some reason, it was unable to playback anything at all. I guess I give up fast, and wanted a few more features, so I decided to run plex either in Ubuntu, or in Windows 10. I tried around in Ubuntu, but was halted by something as easy as being able to add media on an SMB drive, so I gave up, made a new Windows 10 VM, just for plex, which has been working flawlessly. I know this is a pretty inefficient way of running Plex, but at least it is very easy, and for now, stable.

PS: I also decided to turn Hyper Threading OFF. The limitations in ESXi free licence of 8 vCPUs per VM, effectively limits me to 8 threads (i.e. 4 CORES) with HT turned on. Turning off HT gave me the possibility to dedicate 8 physical cores to a single VM.

This leaves me with the following setup:
ESXi 6.7 with the following VMs:
freeNAS: 4 cores, 90GB RAM
Win10Plex: 8 cores, 12GB RAM
Win10Torrenting:8 cores, 6GB RAM (high speed VPN using UDP protocol eats alot of resources)
Ubuntu 18.04 (TeamSpeak server, and other servers throuh LinuxGSM on demand): 4 cores, 4GB RAM (for now).

Everything works, is stable, and high performant.

Also: I tried to install OS X High Sierra, but was unable. Did the unlocker thing, rebooted, and was unable to boot any VMs. Uninstalled unlocker 208, installed unlocker 209Unofficial (cant find 209 Release Candidate), with the same issue (Error 45 I believe). Uninstalled, and VMs start again. Was able to make the VM, and MacOS showed up in ESXi, and it boots, but the VM is stuck in an infinite loop during boot. I see the loading bar filling up, but when it gets full, it starts over again. Any way to solve this? KEEP IN MIND, this was with unlocker UNINSTALLED, since I am unable to boot ANY VM with it installed.

But all in all, I am very happy that I followed your guide, and virutalized freeNAS!

PS: I dont know why, but freeNAS performs BETTER when virtualized. I wasnt able to reach 1GB/s read speeds when running barebone, but I am now. With the exact same config, and same SMB optimizations in tunables. I dont know why, but running barebone, I am stuck at around 700-800MB/s. Now I see stable 1.00 to 1.10 GB/s when reading from ARC.

 

[listhor]

Networking question, I've created in ESXi Storage Net with Switch1as per instruction:

and all (loopback) works fine until I add to Freenas second vmnic/vmx connected to other ports group and Switch2 with physical uplink wmmnic7:

Somehow vmnic7 gets Freenas Storage Net ip - 10.55.1.2 (visible on physical router/switch side) and of course there's no direct communication between Freenas and Esxi via iSCSI or anything else.
Is there anything I could set differently?

 

[listhor]

Networking question, I've created in ESXi Storage Net with Switch1as per instruction:
and all (loopback) works fine until I add to Freenas second vmnic/vmx connected to other ports group and Switch2 with physical uplink wmmnic7:
Somehow vmnic7 gets Freenas Storage Net IP - 10.55.1.2 (visible on physical router/switch side) and of course there's no direct communication between Freenas and Esxi via iSCSI or anything else.
Is there anything I could set differently?

In my case solution was to set Storage Net back to Network adapter 1- after adding second adapter Storage Net was "pushed down" to adapter 2. Now it works again...

 

[alieNus]

Two questions to ESXi: Do you have two VMs, so one for FreeNAS and then one for Docker like Ubuntu with Docker installation or does ESXi have some native (free) installation there (like e.g. ProxMox with LXC)?

AND

if ESXi doesn't have native Docker support, why not use open source Proxmox instead? Are there some big disadvantages?

 

[sretalla]

if ESXi doesn't have native Docker support, why not use open source Proxmox instead? Are there some big disadvantages?

It doesn't.

Proxmox is great, but not nearly as mature and well documented as ESXi. For serious stuff, I would stick with ESXi, but maybe proxmox for sandbox environments until the maturity is there.

 

[alieNus]

It doesn't.

Proxmox is great, but not nearly as mature and well documented as ESXi. For serious stuff, I would stick with ESXi, but maybe proxmox for sandbox environments until the maturity is there.

ok, but would be the free ESXi version be enough to have FreeNAS VM and Ubuntu VM with Docker? Are there some limitations maybe?

 

[sretalla]

would be the free ESXi version be enough to have FreeNAS VM and Ubuntu VM with Docker? Are there some limitations maybe?

I do exactly that in my second system (see signature), it works fine.

You can do a lot with the help to be found in the forum such as scripts to work with the storage being provided by the FreeNAS VM back to the other guests... this will be difficult to figure out without that.

See Here:

GitHub - Spearfoot/utility-scripts-for-freenas-and-vmware-esxi: All-In-One utility scripts for FreeNAS and VMware ESXi

All-In-One utility scripts for FreeNAS and VMware ESXi - Spearfoot/utility-scripts-for-freenas-and-vmware-esxi

github.com

and

Utility scripts for FreeNAS and VMware ESXi

I've created a GitHub repository containing the scripts I've written for use with my FreeNAS-on-ESXi All-In-One systems: https://github.com/Spearfoot/utility-scripts-for-freenas-and-vmware-esxi These might be handy for folks who haven't already...

www.ixsystems.com

 

[sretalla]

Also, make sure you read this:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.ixsystems.com

 

[listhor]

View attachment 20206
And there we have it...

Now... that's only a 100GB zvol... lets resize it!

I've done all as per your great tuto but upon restarting ESXi, iscsi store is not attached - obviously because FreeNAS is not yet up and running. Is there any way to delay iscsi service in ESXi?

 

[sretalla]

Is there any way to delay iscsi service in ESXi?

Look at my post 2 above yours at the spearfoot scripts, there's one there you can call to cause ESXi to re-scan the datastores after FreeNAS is up.

 

[listhor]

Look at my post 2 above yours at the spearfoot scripts, there's one there you can call to cause ESXi to re-scan the datastores after FreeNAS is up.

Thanks, but how safe are they? Is there any chance I could have something messed up in ESXi?

 

[sretalla]

Thanks, but how safe are they? Is there any chance I could have something messed up in ESXi?

The scripts are using API calls to ESX, so functions that are called are 100% ESXi, so I would say as safe as for you to manually initiate a rescan.

 

[listhor]

The scripts are using API calls to ESX, so functions that are called are 100% ESXi, so I would say as safe as for you to manually initiate a rescan.

Thanks again, I will give a shot during weekend...
BTW, do you know how to set network interfaces' order, when adding new ones to VM? I have Alpine VM with 2 virtual network adapters (eth0 and eth1) and when I add another one, it goes as first one or second, not the 3rd. And it results in destroying my static IP assignment in VM plus static iscsi iface... When I reassign IPs, I end up with RTNETLINK errors: file exists... Flushing of interface doesn't help.

 

[sretalla]

BTW, do you know how to set network interfaces' order,

I guess the question is for ESXi?

The order they appear in the GUI is the order that they will present to the OS, so you can either change the assignment of first and second (etc.) NICs or re-add them in the right order.

 

[listhor]

I guess the question is for ESXi?

The order they appear in the GUI is the order that they will present to the OS, so you can either change the assignment of first and second (etc.) NICs or re-add them in the right order.

Yes, for ESXi. I'm testing Alpine and docker together and for sure NICs don't appear in Alpine in the same order as they are in VM settings. And I can't get Alpine to work with more than one subnet with gateway (one default subnet and i.e storage network without gateway are ok). I mean if I have 2 subnets (different) with their respective gateways configured - connectivity is only on nic with lower interface number (on eth0 and not on eth1).
Looks like there's some routing error:

Code:

docker:~# ifup -v -f eth1
run-parts /etc/network/if-pre-up.d
ip addr add 172.16.1.21/26 dev eth1 label eth1
ip link set eth1 up
ip route add default via 172.16.1.1 dev eth1
ip: RTNETLINK answers: File exists

Despite of existing default/correct routing table... I flushed interface and it didn't help.

EDIT:
Solution was here: https://wiki.alpinelinux.org/wiki/Multi_ISP
I had to add "metric" option in /etc/network/interfaces, i.e.:

Code:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 172.16.1.20
        netmask 255.255.255.192
        gateway 172.16.1.1
        metric 200
        post-up ip link set dev eth0 mtu 9000
        hostname alpine

auto eth1
iface eth1 inet static
        address 10.55.1.3
        netmask 255.255.0.0
        post-up ip link set dev eth1 mtu 9000

auto eth2
iface eth2 inet static
        address 172.16.0.20
        netmask 255.255.255.0
        gateway 172.16.0.1
        metric 300
        post-up ip link set dev eth2 mtu 9000
        hostname alpine

 

[mattheja]

edit: so I've got it working and I'll follow up once I've finished everything, ty.

So, posting here to see if someone can help, I'm at my wit's end. I've read through the VMware documentation and google'd for a couple hours now.

I can't get the networking to work right.

1. I created a new virtual switch vSwitch1 with defaults (identical to vswitch0).
2. I then created 2 new Port Groups attached to vSwitch1

-> Here is the issue. If I don't add vmKernel NIC's, I can attach these port groups to network adapters on my VM's. If I create vmKernel NIC's with IP addresses for loopback communication between VM's, then I cannot add the port groups to network adapters.

3. Create 2 vmKernel NIC's to associate with the port groups for loopback communication between VM's and FreeNAS

4. As seen below, I can no longer assign those port groups.

I'm on ESXi 7.0.0... if that's the issue, then I guess I'll downgrade, but if someone can spot an issue, I'm all ears. Thanks!

 

[Yooshua]

edit: so I've got it working and I'll follow up once I've finished everything, ty.

So, posting here to see if someone can help, I'm at my wit's end. I've read through the VMware documentation and google'd for a couple hours now.

I can't get the networking to work right.

Kind of a late reply but I thought I would give a suggestion. Put them in different port groups. Port Groups don't really matter unless you assign a VLAN to them. They should still be able to communicate even though they are in different port groups. If you assign a VLAN to them you just have to have a router between the vlans.

 

[listhor]

I've recently installed Ubuntu server on iscsi datastore (connected to Freenas through Storage net) and I started getting on FreeNAS console following errors:

Nov 21 02:27:25 freenas ctl_datamove: tag 0x1306878 on (0:5:0) aborted
Nov 21 02:27:25 freenas ctl_datamove: tag 0x1306879 on (0:5:0) aborted
Nov 21 02:27:25 freenas ctl_datamove: tag 0x130687a on (0:5:0) aborted
Nov 21 02:27:25 freenas ctl_datamove: tag 0x130687b on (0:5:0) aborted

accompanied by errors on Esxi:

Successfully restored access to volume 5e5905d6-3205db70-87ee-ac1f6bd7fbc4 (Store_iscsi) following connectivity issues. Saturday, November 21, 2020, 02:27:07 +0100 Warning None
Lost access to volume 5e5905d6-3205db70-87ee-ac1f6bd7fbc4 (Store_iscsi) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly. Saturday, November 21, 2020, 02:27:06 +0100 Warning None

I have other VM (win10) on this datastore and have never seen these errors before installing Ubuntu server. And I don't see any I/O errors in Ubuntu itself.
Thus how to troubleshoot it, what might be the root cause - more intensive I/O operations?

EDIT:
When I do HDD speed checks in that win10 I get quite high results and no previously mentioned errors...

------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo
Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential 1MiB (Q= 8, T= 1): 1400.482 MB/s [ 1335.6 IOPS] < 5981.49 us>
Sequential 1MiB (Q= 1, T= 1): 973.495 MB/s [ 928.4 IOPS] < 1075.58 us>
Random 4KiB (Q= 32, T=16): 335.230 MB/s [ 81843.3 IOPS] < 6246.61 us>
Random 4KiB (Q= 1, T= 1): 20.280 MB/s [ 4951.2 IOPS] < 201.42 us>

[Write]
Sequential 1MiB (Q= 8, T= 1): 1686.052 MB/s [ 1607.9 IOPS] < 3989.38 us>
Sequential 1MiB (Q= 1, T= 1): 1140.973 MB/s [ 1088.1 IOPS] < 917.26 us>
Random 4KiB (Q= 32, T=16): 333.879 MB/s [ 81513.4 IOPS] < 6270.61 us>
Random 4KiB (Q= 1, T= 1): 14.876 MB/s [ 3631.8 IOPS] < 270.75 us>

Profile: Default
Test: 128 MiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED>
Date: 2020/11/21 10:15:00
OS: Windows 10 Professional [10.0 Build 17763] (x64)

 

[achmetinternet]

Thanks for this awesome build instructions. I followed them for my bare metal build. However, it seems that I don't understand the speaker right

When I start up there is a series of beeps but none of them really make sense. According to the manual there are several beep sequences possible:

It sounds close to the Memory error message but TrueNAS boots up fine and it recognizes all the 64GB RAM inserted. Anything I am missing?

 

[zbyte]

but TrueNAS boots up fine and it recognizes all the 64GB RAM inserted

Even if it boots and detects the memory the memory could still be faulty

When you setup the system did you changed any BIOS options? if you did, resetting the BIOS to defaults do you still get the beeps?

Go to https://www.memtest86.com/ and run memtest and see what you get. What type of memory is it?