FreeNAS on ESXi 7.0 - The Good, Bad, and Ugly from previous ESXi Versions

[joeschmuck]

So this conversation started in another thread and I didn't feel like we should be taking over that thread so another one has been created. Basically come here to see what others have been doing with testing ESXi 7.0 noting any new limitations, changes, enhancements, or concerns/complements that you find when compared to ESXi 6.5 or 6.7.

I have just upgraded my test system (Supermicro A1SAM-F, 16GB RAM, one SSD for ESXI Boot, two 500GB laptop hard drives for datastores) with ESXi 7.0, I created a new FreeNAS VM using TrueNAS 12.1Beta. All was working fine. I'm in my 60 day evaluation period where all the bells and whistles are allowed. I plan to do the unthinkable, enter my license and use ESXi 7.0 in the free version configuration. That will give me a true assessment. Also, I have no desire to upgrade my main server at all, not until I'm satisfied there will be no problems.

But so far so good. As for time devoted to this endevour, I've got a busy life having recently moved and started a new job, buying a house, unpacking. So I will be happy if I get 4 hours or more a weekend. I'm willing to assist others with testing somethings out, such as backup software for ESXi VMs. I do plan to test 33Hops software, possibly tomorrow if I can get the time.

So I hope this thread turns out to become helpful to all the folks using ESXi.

Cheers,
-Mark (aka Joe)

 

[Evertb1]

Good work Joe. Happy experimenting. At this moment I am busy with selecting parts for a, way overdue, build of a new workstation. I still prefer a desktop over a laptop when I am working at a desk. After many years of being a loyal Intel customer I switch to the AMD Ryzen platform. Now before I throw Windows at my new workhorse I intend to see if I can run ESXi 7.0 on it. Because there will be a new ESXi server in my future to replace my FreeNAS main file server.

 

[joeschmuck]

There is nothing wrong with AMD. I recall Cyrix, IDT, NEC all in the game back in the day and making cost effective components. I like the AMD Ryzen and when I started FreeNAS 8.0 I was using an AMD system. I was told that it would never work, but it did. I changed to Intel after that and have been pretty happy. Like you, I prefer a desktop system when I am at home. I do have a nice laptop but that is mainly for travel or when I'm in my recliner. The feel of a full size keyboard and screen is more pleasing. My desktop is an Intel i930 w/24GB RAM (triple channel). Back in the day it was considered fast. I created a RAM Drive and used it to compile code. I've been dreaming about a new system, and I have the money, but I have a hard time justifying it right now. I have 3 servers (one active, one testing, one sitting in a corner), one laptop, one desktop, a pile of parts. My wife might kill me if I bought another computer.

And I'd love to hear about your final parts selection for the workstation.

 

[Evertb1]

I recall Cyrix, IDT, NEC

That I recognize those names must mean that I am getting old.

And I'd love to hear about your final parts selection for the workstation.

I wil certainly come back to this thread when I have made my choises. The cpu will most likely be a Ryzen 7 or 9. 5 or 7. At the moment I can buy them with a nice discount. I think that is because of the coming 4000 series. I, like everybody else, am curious about the pricing and when they will be available.

The AMD Ryzen CPU's are a no-brainer. More cores then the competition with high clockspeeds and the capability to support a lot of memory. Very effective recipy. Though if I Google "AMD workstation" Threadrippers are advized. Almost without an exception. But you are not the only one with a wife . No Threadripper for me.

It's funny, but this time around I have more of a hard time selecting a motherboard. I will probably look for something with the b550 chipset, though they are way more expensive then their predecessor the b450. Most advizers go for the x570 chipset but I don't want a motherboard with a fan on the chipset cooler. I have a mistrust for the durability (and noise) of those tiny fans.

 

[Evertb1]

About ESXi 7.0 and FreeNAS.

I Upgraded from ESXi 6.7 to ESXi 7.0. I had a FreeNAS 11.xx VM running on that server. The guest OS for that VM was of course FreeBSD 11.
I upgraded that VM to TrueNAS 12.0-BETA by changing the release train. Without a problem by the way. I did this before I executed the upgrade to ESXi 7.0.

But since that upgrade to ESXi 7.0 I have this warning in ESXi with that VM.

That is not a big deal I think, as FreeNAS is still running OK. But I feel you all should know.

Edit: I just updated that VM and changed the guest OS version to FreeBSD 12 or later. Strange that I never realized that was possible.

 

[Yorick]

I have more of a hard time selecting a motherboard

If you are going B550 I’d recommend the Asrock B550 Pro4. Asrock because they are the only vendor who adds ECC support to every board. When we built a Ryzen desktop for the husband, we used ECC so we’d never have to troubleshoot a potential memory flaw, ever again. Instead, we’ll know if memory goes bad.

I’m with you on the merits of x570. I don’t see the benefit for a workstation.

 

[Evertb1]

When we built a Ryzen desktop for the husband, we used ECC so we’d never have to troubleshoot a potential memory flaw, ever again.

ECC on my workstation is an actractive option. Now if we only could convince MS to support ZFS . I am still depending on Windows for most of my development and egineering tooling on my workstation.

 

[joeschmuck]

I am still depending on Windows for most of my development and egineering tooling on my workstation.

I fully understand that, I'm in the same predicament as well. But I do need to learn RedHat Linux for work, we are changing from Windoze for most of our projects. I'm sure it's a cost savings and a security risk avoidance.

Are you planning to run Windoze on ESXi or on bare metal?

Edit: I just updated that VM and changed the guest OS version to FreeBSD 12 or later. Strange that I never realized that was possible.

I didn't know that was possible either, not without manually editing the .vmx file which I have done before because it was the only way on ESXi 6.5 that I was aware of. Maybe that is a difference between the 60 day evaluation and when it goes on lockdown after 60 days.

It's funny, but this time around I have more of a hard time selecting a motherboard. I will probably look for something with the b550 chipset, though they are way more expensive then their predecessor the b450. Most advizers go for the x570 chipset but I don't want a motherboard with a fan on the chipset cooler. I have a mistrust for the durability (and noise) of those tiny fans.

I agree, I do not like small fans keeping things cool on a motherboard. In the past I have replaced the heatsink and fan on the chip with a larger heatsink, it had the same footprint but was significantly taller and this allowed me to remove the fan. I also ensured that there was good air flow through the case so I didn't have any hot spots. Sometimes it takes a bit of work to design a proper system. And I always design one with Silence in mind. The only time I can hear anything from my computers is when it powers on and the motherboard runs the fans at full speed for a few seconds, after that, silence. I also use large slow turning fans, my case fans all run at 7VDC vice 12VDC and are all 120mm or larger. On my test server I have a cardboard cover (made from a cereal box) attached to an exhaust fan and the cover pulls air across the fanless CPU heatsink. It wasn't required that I do that but proper cooling is a must for me. Now I want to look into these motherboards and see what the pros and cons are. I'm still not ready to replace my desktop but I can dream.

If you are going B550 I’d recommend the Asrock B550 Pro4. Asrock because they are the only vendor who adds ECC support to every board. When we built a Ryzen desktop for the husband, we used ECC so we’d never have to troubleshoot a potential memory flaw, ever again. Instead, we’ll know if memory goes bad.

I’m with you on the merits of x570. I don’t see the benefit for a workstation.

With RAM prices being what they are today (unlike the prices after Japan was wiped out years ago in 2011 by the earthquake) I can see purchasing ECC RAM for my desktop computer. I need to look up those chipsets and do a comparison. If I see anything obvious that I would stay away from, I'll mention it.

 

[Evertb1]

Are you planning to run Windoze on ESXi or on bare metal?

I think that would be bare metal. I need to do some graphical work every now and then as well and that's best done on bare metal.

I didn't know that was possible either, not without manually editing the .vmx file which I have done before because it was the only way on ESXi 6.5 that I was aware of.

I have never needed it when I was running ESXi 6.7 so I have never looked for it. With 7.0 it's available under Edit --> VM Options --> General Options. It's grayed out until you stop the VM.

With RAM prices being what they are today (unlike the prices after Japan was wiped out years ago in 2011 by the earthquake) I can see purchasing ECC RAM for my desktop computer.

Agree. If I look at a brand like Kingston, the price difference between ECC/non-ECC for comparible 16 GB modules DDR4-2666 Hz is about 2 euro's (2.38 US Dollar) on the Dutch market. The only thing to think about is that the Ryzen platform seems to benefit from higher memory speeds. Though the role of the motherboard should be more important. Especially when it comes to tight timing.

 

[joeschmuck]

So I priced out an upgrade for my desktop and it would cost me $842 (US Dollar) for the upgrade, and it's a big upgrade compared to my over 10 year old system.
The upgrade would be Ryzen 5 3600X, ASRock B550 Pro4 Micro ATX (I have not looked at others yet but it does look nice), 64GB of Kingston 3200MHz Non-ECC RAM (only because there was no tested 16GB sticks of ECC RAM so if I do go the ECC route, I'd have to do more research), and a 1TB Rocket SSD for the m.2 PCIe4.0 x4. I thought about just using a 500GB M.2 but if I'm going to upgrade, might as well go for it. I would not replace the case or power supply, those are all perfectly fine. Well I might replace the case eventually since I am going with the Micro ATX motherboard. Small, quiet, that is what I like. No spinning rust, two 120mm fans and the CPU fan. No complaints.

I would try to run Windoze on ESXi, I do have an RC Flight Simulator, about the only graphics going on in my world, not sure it would run. So I might just get one 32GB stick of RAM to start with and if Windoze works fine, then upgrade to 64GB.

Okay, enough day dreaming, time to go make a laundry table and then I can come back and work on ESXi 7.0 a bit more. So far I have no complaints about it.

 

[Evertb1]

I think I have finished my shopping list for my new workstation. I will "only" buy a new motherboard, new memory and a new CPU. The CPU will be temporary for my workstation. It will find its way in my new ESXi server end this year - early next year. In the mean time the 4000 series Ryzen should be available and one of them will be put in my workstation I think.

CPU Ryzen 7 3700x - eur. 287.00 eur (usd 339.62)
Motherboard Gigabyte B550 Aorus Pro - eur. 165.90 (usd 196.32)
Memory HyperX 32 GB DDR4-3200 (2 x 16) - eur. 159.80 (usd 189.10)

Boot SSD: ADATA XPG Gammix S11 Pro, 256 GB SSD (I own that already)
Local storage SSD: Samsung 860 PRO, 1 TB SSD (I own that already)
PSU: Corsair RM550X (I own that already)
Case: Fractal Design Define R5 (I own that already)
GPU: Gigabyte GeForce GTX 1660 (I own that already)

All the spinning rust is already banned to my FreeNAS home server.

I don't need to say much about the CPU. By now it's clear that they are great and I think it will do good in an ESXi server as well.

The motherboard is another story. @Yorick advized an Asrock B550 Pro4 but after reading some tests I decided to go for the Gigabyte. One of the reasons (but not the only one) is the less then great VRM on the Asrock board. Its not bad though, but it can be better. Especially with the more demanding Ryzen 7 and 9 series. I have chosen an ATX format as I like it when my system has sufficient space for a decent airflow between the components. And my case acommodate it with ease. One thing is hard to avoid with those B550 motherboards and that is all the, for me, unnecessary RGB headers and other RGB sh.... Oh well.

The memory is not ECC. While nice to have it is not a must on a workstation. To avoid discussions: that's what I come to think after endless reading. Everybody can decide for them self if ECC is important on a workstation or not. It's not supported by my motherboard of choise anyway. The modules are on the QVL of the motherboard.

EDIT: I just read in the motherboard manual that it supports ECC-unbuffered memory. I am not sure if that means that it allows ECC-unbuffered modules to run in non-ECC mode or that it really supports the ECC-mode. I contacted Gigabyte and their service team will asap send me a list of the CPU's that work in that motherboard with full ECC support. Nice.

I order the new stuff somewhere this week. As soon as I have it I will "steal" some stuff from another build to test it and to play around with ESXi 7.0 before it will be loaded with Windows. I am anctious to know how well the Ryzen platform handles virtualization and if PCIe passthrough will work correctly. And I intent to test XSI Backup on ESXi 7.0. Officialy it is not supported (yet) according to the 33hops website. We will see.

If I have my way my future new ESXi build will consolidate my homelab server and FreeNAS home server in one system.

 

[joeschmuck]

Very nice. The conversation we have had in this thread had me looking at a Ryzen 3800X today, I was just looking at specs, not prices.
I just looked into the motherboard, looks very good as well. Did you download the user manual and read it? I plan to soon to see if there are any warning signs like if you use an M.2 connector, which SATA port are terminated.

One thing is hard to avoid with those B550 motherboards and that is all the, for me, unnecessary RGB headers and other RGB sh.... Oh well.

It was designed to be sold to gamers who want a pretty looking box. I too would not connect it to anything.

The memory is not ECC. While nice to have it is not a must on a workstation. To avoid discussions: that's what I come to think after endless reading. Everybody can decide for them self if ECC is important on a workstation or not. It's not supported by my motherboard of choise anyway. The modules are on the QVL of the motherboard.

My desktop does not have ECC RAM, only my ESXi server does. I'm with you on this one. If I get a bit-flip, oh well, it's a risk I take. As for on the QVL, on many occasions I would have to choose something that was not tested by the manufacturer. You take your best guess and roll the dice.

And I intent to test XSI Backup on ESXi 7.0. Officialy it is not supported (yet) according to the 33hops website. We will see.

I have upgraded my ESXi to 7.0, I will be testing this out as well soon, maybe this weekend.

If I have my way my future new ESXi build will consolidate my homelab server and FreeNAS home server in one system.

Very possible. Just make sure you have enough RAM to run both very comfortably. Looks like the graphics can use up to 16GB so setting up the BIOS will be very important.

 

[Evertb1]

Did you download the user manual and read it? I plan to soon to see if there are any warning signs like if you use an M.2 connector, which SATA port are terminated

The board has 2 M.2 connectors. M2A_CPU is connected to the CPU and M2B_SB is connected to the chipset. As far as I can see no sata ports are terminated when a M.2 connector is in use. However, if the M2B_SB is used, one of the PCIE slots (PCIEX4) will become unavailable as it shares its bandwidth with the M2B_SB connector. I can live with that because I won't go SLI anytime soon. And I won't havy many expansion cards.

Very possible. Just make sure you have enough RAM to run both very comfortably. Looks like the graphics can use up to 16GB so setting up the BIOS will be very important.

I am not sure what you mean here Joe. My future ESXi build will be with an Asrock rack motherboard with IPMI and it will be headless. It will inherit the 3700X as I plan to buy a 4000 series CPU for this workstation as soon as they become available at a comparible price point. If the 4000 series are a no go, say the first quarter of the coming year I just buy another 3000 series CPU.

So my plans are for 2 systems.
A new workstation (to be build coming weekend) with an discrete graphics card, so no internal memory needed.

A new ESXi server (to be build at the end of the year or early in the coming year). It will be headles so no or little memory should be needed for that. As far as I know IPMI is an autonomus subsystem and has no need for a piece of the internal memory. Anyway, I plan to start out with 64 GB (2 x 32) and then have room to expand to 128 GB. That should be more then suffcicient for my needs. The only real memory hungry beast will be the FreeNAS VM. All the other VM's have modest requierements or are not running unless I need them.

Budget wise it will be a bit juggling. I have a private budget and a business budget. Need to keep on friendly terms with the bookkeeper and the "belastingdienst" (Dutch IRS). And let's not forget the wife. Sleeping in the quest room is not my idea of living the life

 

[joeschmuck]

I am not sure what you mean here Joe.

My misunderstanding.

And let's not forget the wife. Sleeping in the quest room is not my idea of living the life

Very true!

The motherboard M.2 connections sound good to me too. I did download the user manual and plan to read it tomorrow (have family coming in tonight) to see what it says. I'm now considering this motherboard too. I do have a discrete video card but not sure I would be using it, although it's very good at crunching numbers for BOINC. I still have other house projects that will demand some money first but eventually I will upgrade the desktop computer. I am very curious how yours works out and any observations you make, good and bad.

As for my ESXi and VM's, so I updated to ESXi 7.0 yesterday, also updated to FreeNAS 11.3-U4.1, and I have the current version of Ubuntu in a VM. All are still running just fine after the night and I expect them to be running when I get home from work. So far so good. I doubt I'll have time to play with it tonight but after the visiting family departs tomorrow, I'll have time to play. I really want to test out 33Hops product to see if it still works. The one problem I had with it in the past was some of the VM's needed to be shutdown to make a backup of the VM. The program would force the shutdown if you told it to and then restart the VM once done, but some VMs would backup just fine while running.

 

[Evertb1]

I do have a discrete video card but not sure I would be using it

I don't think you have much of a choise. As far as I know only Ryzen 3000 cpu's with the suffix "G" like the Ryzen 5 3400G have integrated graphics. And without it the graphical connectors on the motherboard are dead.

 

[joeschmuck]

So just a few things I found out reading the user manual (while at work on my lunch break) for the Gigabyte B550 Aorus Pro:
1) If you use the M2B_SB connector, you loose the PCIEX4 connector, leaving you with one less full length slot. (something @Evertb1 already has said)
2) If you use SATA3 connectors 4 or 5, you loose PCIEX2 connector, leaving you with one less full length slot.
3) I like the Fan Speed Control in BIOS, hopefully it works well. I typically run my case fans at 7VDC to provide a constant slow moving flow of air.
4) The board has two connection points for Thermal Sensors but the box does not come with thermal sensors. Where are those items procured?

Overall it looks like a good motherboard. I have owned several Gigabyte motherboards and I can't think of a single one that gave me trouble. Now to figure out a proper CPU (with or without graphics) and the RAM, then price it. It's still a dream but as I get closer, I'll have a good idea what I want.

 

[Yorick]

64GB of Kingston 3200MHz Non-ECC RAM (only because there was no tested 16GB sticks of ECC RAM so if I do go the ECC route, I'd have to do more research)

On Asrock, you can use really any DDR4 ECC UDIMM. ECC memory is conservative with timings and speed. We built on a B450M with "whatever" ECC we could find (might have been Crucial, I'd have to check), and it's working great. Crucial (Micron), Samsung, Kingston, will all work.

Whether ECC is "required" is less a technical question and more a quality of life one. Personally, I have decided it's required for all future desktop builds, only because when I did have to troubleshoot memory, it was invariably a major pain. Only happened two or three times in the past 20+ years, and still. I value my time more than I used to, and I don't want to spend it tracking down memory errors that I could, instead, just get alerted on.

That plus the fact that ECC costs a negligible amount more, and it's almost a no-brainer. It does lock me into Asrock for Ryzen because their ECC support is the most "solid" across all the board manufacturers, just by virtue of the fact that they have it on every Ryzen board, and on Asrock Rack, and thus have more experience with it than other manufacturers.

 

[Evertb1]

2) If you use SATA3 connectors 4 or 5, you loose PCIEX2 connector, leaving you with one less full length slot.

Good catch. I recall that one from a test on a Dutch tech site but I missed it somehow while reading the manual. However it shows clearly in the block diagram. The chipset has its limitations and there are several ways the manufacturers handle those. The whole is pretty flexible though.

The board has two connection points for Thermal Sensors but the box does not come with thermal sensors. Where are those items procured?

Those thermal sensor connection points are part of the Gigabyte Smart Fan 5 program. I have seen temperature sensor cables displayed on the Gigabyte site where that package is explained. I have found some of them here but to me it's an item that should be included in the package. I haven't found any in a regular store (yet).

 

[Evertb1]

I just received word from Gigabyte that while ECC memory is supported the ECC function is not utilized (I was afraid of that). They point out that their TRX40 series do, but you won't believe the pricing of those suckers and I wont buy a Threadripper CPU anytime soon.

 

[Evertb1]

I just build my new workstation. The new components are as planned:
CPU Ryzen 7 3700x
Motherboard Gigabyte B550 Aorus Pro
Memory HyperX 32 GB DDR4-3200

The system is running just fine. Before I drop Windwos 10 on it I have installed ESXi 7 on it and that went well.

The first try the installation failed because no supported NIC was found (well I could try). So in went the Intell NIC and gone was the problem. During installation the installer reported several not supported PCI devices (no surprise with this motherboard) but that was not fatal. After installation everything seemed to run OK. And the boot was very snappy.

I had a Dell PERC H310 (in IT mode) plugged in and I was able to pass it through. The option to enable that in the BIOS of the motherboard was not hard to find (it's called vrm on this platform). After several reboots passthrough was still enabled. I had read reports that with some users it would drop back to disabled after a reboot but that did not happen.

So now I am pretty confident that my next ESXi build will be AMD based. Be it with a Asrock Rack motherboard.

Now its Windows time for my new rig.