Virtualise on i3? Overly Ambitious Beginner Build?

[DuaneDibbley]

Hi all,

I'm not sure why I need another NAS. I have a pretty happily running DS415+ with 4 of those 6TB Reds with SMR in it .
It's okay but it's pretty boring and I'd like to work towards moving my computers which run 24/7 doing various tasks into one box.

Ideally I'd like to build a system which can:
1. Act as remote storage for my wife and I's images in Lightroom - ie store but also work on originals off the NAS = so decent R/W speeds needed.
2. Act as general home bulk storage.
3. Replace my 2012ish Mac Mini server which is running 24/7 with Sonaar, SabNZBd and Plex (with videos stored on synology).
- I could mostly do this with the Synology apart from the poor performance for Plex.
4. Replace Surveillance Station on my Synology with one of the various other options most of which run on Windows like Blue Iris etc.

Initially I plan on using the old 4x 3TB reds I took out of the Synology and replaced with the 6TB ones which won't be suitable for use with ZFS it seems. If this works well I'll increase the capacity and number of drives in the FreeNAS box.

It seems that to accomplish all of that on one machine I need to look at ESXi, various VMs a HBA of some variety to pass through to FreeNAS.

Originally I thought I'd just try building a faster NAS for points 1 & 2 above but then as I looked into it feature creep seems to have really taken hold.. This was my initial "just a NAS" spec:

CPU: Intel Core i3-9100 3.6 GHz Quad-Core Processor
Motherboard: Asus WS C246 PRO ATX LGA1151 Motherboard
Memory: Crucial 16 GB (1 x 16 GB) DDR4-2666 Memory
Storage: Crucial MX500 250 GB M.2-2280 Solid State Drive (For OS or ESXI etc)
Case: Silverstone CS380 ATX Mid Tower Case
Power Supply: SeaSonic FOCUS Plus Platinum 650 W 80+ Platinum Certified Fully Modular ATX Power Supply
= ~$1300 Australian

With the addition of a HBA could I manage to proof of concept with this spec that could potentially be my main server?
I suspect the main weaknesses to be CPU and RAM? More ideally would be 32gb RAM and I'm not sure with the CPU... You gotta spend a LOT on a Xeon before it really seems to outshine the i3 it seems?

I'd probably prefer a Supermicro X11SCM-F (or similar) for the motherboard but it really seems difficult to buy one in Australia for some reason. HBA's are something I know very little about but I think I'm sort of on the right track there - again seems to be less available new and reasonably priced than regular consumer gear. Probably look second hand on ebay - how resilient is ZFS to HBA's exploding due to their owners cheaping out and buying well used?

To go Xeon/HBA/32gb I'm probably looking at around $2000 AU which is quite a lot for a bit of a play/learning experience/hopefully eventually main home NAS/Server.

Please poke any holes you want in my plan! Very keen on hearing input from those with experience and knowing what they're doing :)

 

[Yorick]

The boot drive is overkill. You could do an NVMe m.2 at 120gb for something like usd 40 (hp 900 or similar), or Intel 320 40gb if you can find it used, those go for usd 15 here.

You mention Plex. The i3 can Hardware accelerate transcode (1080p vc1 to 1080p h264 for example), but the board would need to support it. X11scm does not; x11sch does. Would need TrueNAS 12 to use the hardware transcode and, nothing Wrong with planning ahead a little.

HBA dying is handled gracefully by ZFS, just get another HBA.

You may be able to pass the onboard sata controller through. If you do that, you don’t need a HBA.

Around here ECC memory costs about the same as non-ECC. You do want ECC memory, if you are going out to build something like this in the first place.

32gb of ram is a necessity just because of all the VMs you want to run. Windows alone will eat 8gb.

Not worried about the CPU. Should give you plenty for what you want to do.
Edit: hold up a second, now I am worried. You are virtualizing. Which means you need vt-d. Does that cpu support it? Does the asus?
You likely don’t need frequency, you are not doing anything very compute intensive. A supermicro plus xeon e-2xxxg, one of the lower tier ones, will definitely support vt-d and support it well.

For the live editing, this will happen over a gig link? Your write speeds won’t be awesome with those SMR drives. That’s the only concern. You can always fix that down the road with a mirrored ssd pool as scratch space for editing.

Wd red SMR and ZFS: they don’t survive resilvering, that is rebuild, at least some firmware versions don’t. Wd will RMA those, if you open a case and tell them they’re in a ZFS pool and sustained write performance is bad, and you are worried about the idnf errors you read about during resilver. They’d replace them with CMR drives. Definitely see whether they’ll do that for you, you’ll have a better system.

4x means you can either build a raidz1 or a raidz2. Translated into use case that reads as:
Raidz1 - I have a backup, restoring it is no big deal. If the pool dies so what
Raidz2 - I have a backup, restoring it will be a pain because I need the data available, really. I’d vastly prefer my pool not dying

Raidz1 gives you 3 drives of 4 usable for storage; raidz2 it’s 2 out of 4.

 

[DuaneDibbley]

Thanks, Yorick good info :)

The boot drive is overkill. You could do an NVMe m.2 at 120gb for something like usd 40 (hp 900 or similar), or Intel 320 40gb if you can find it used, those go for usd 15 here.

Fair point actually. Would the aim be to have HBA passed through to the FreeNAS VM for the storage disks and use the board SATA controller to have other disks with the VMs on them or would I be running the VM's off the FreeNAS storage? I'd imagine that would be quite slow.

You mention Plex. The i3 can Hardware accelerate transcode (1080p vc1 to 1080p h264 for example), but the board would need to support it. X11scm does not; x11sch does. Would need TrueNAS 12 to use the hardware transcode and, nothing Wrong with planning ahead a little.

I'd probably have the plex server running on a Linux VM (or windows if I want to be lazy and it's my most familiar OS) rather than on FreeNAS. Given the pain in getting a Supermicro board I'll have to check that Asus supports it - thanks for pointing that out.

HBA dying is handled gracefully by ZFS, just get another HBA

Perfect.

You may be able to pass the onboard sata controller through. If you do that, you don’t need a HBA.

Depending on cost it might just be simpler (and more fun) to have the storage drives on a HBA and other drives on the onboard controller? Probably matters little in practice but it's a nice imaginary separation.

Around here ECC memory costs about the same as non-ECC. You do want ECC memory, if you are going out to build something like this in the first place.

32gb of ram is a necessity just because of all the VMs you want to run. Windows alone will eat 8gb.

Ah, yep - that is Unbuffered ECC RAM I'd specced just missed it on the typing out. 32gb it is!

Not worried about the CPU. Should give you plenty for what you want to do.

This is great! The cost I save from not needing a Xeon will almost pay for the extra 16gb of RAM.

For the live editing, this will happen over a gig link? Your write speeds won’t be awesome with those SMR drives. That’s the only concern. You can always fix that down the road with a mirrored ssd pool as scratch space for editing.

Yep - over a 1 gig network. Which sorta makes chasing 'speed' a bit of a poor value proposition. But at least as 10gig continues lowering in cost I can one day add an extra 4 port 10gig switch and add the two desktops and NAS to it. The SMR disks are currently in the Synology and I plan on leaving them there. Initially I plan on using the original non SMR drives for this build. Before I upgraded the drives in the Synology to the 6TB reds I'd tested and decided I could live with the speeds editing off it. However it's noticeably slower since the upgrade which is what gave me this FreeNAS idea.

Wd red SMR and ZFS: they don’t survive resilvering, that is rebuild, at least some firmware versions don’t. Wd will RMA those, if you open a case and tell them they’re in a ZFS pool and sustained write performance is bad, and you are worried about the idnf errors you read about during resilver. They’d replace them with CMR drives. Definitely see whether they’ll do that for you, you’ll have a better system.

Interesting! I hadn't heard of them doing this. Something to keep in mind I guess. Perhaps if I get them to do this once the FreeNAS box is running I can then add the replaced non SMR drives to the pool.

4x means you can either build a raidz1 or a raidz2. Translated into use case that reads as:
Raidz1 - I have a backup, restoring it is no big deal. If the pool dies so what
Raidz2 - I have a backup, restoring it will be a pain because I need the data available, really. I’d vastly prefer my pool not dying

Raidz1 gives you 3 drives of 4 usable for storage; raidz2 it’s 2 out of 4.

I plan on Raidz1 at this stage. I plan on backing up the image date from the FreeNAS to the Synology at a fairly regular interval. We're hobby photographers so we're not producing enormous numbers of images each week. The important stuff on the Synology I regularly backup on a large external I keep at work.

Thanks again for your reply - it was extremely helpful.

 

[Yorick]

So I looked into the i3-9100 - it does support vt-d. No idea about that ASUS board - does it, does it do it well, and so on. There's a reason those who choose to virtualize don't do it on "consumer" brands. I know ASUS wants to be the next supermicro, but they're even further from that goal than ASRock Rack.

Boot drive: Good point. You do need space for VMs. Go back to that 250GB idea :).

This will teach you all you need to know about HBAs: https://www.ixsystems.com/community...-lsi-9211-9300-9305-9311-hba-and-variants.54/. Beware Chinese knockoffs on eBay; buy from reputable "pulled from a server farm" sellers, preferably near you. You want these in IT mode; the older 2008 model is just fine. IBM M1015 or Dell H200 are both common models that can be had for 40-50 USD.

Alright. I'm going to try and talk you out of this virtualization idea. First please read this: https://www.ixsystems.com/community...nas-in-production-as-a-virtual-machine.12484/

Sonar, plex, yada yada, those all run great as jails. Nothing at all wrong with running that directly on FreeNAS.

Blue Iris is Windows, true. ZoneMinder isn't. Consider that instead.

Even if you MUST HAVE Blue Iris, TrueNAS Core 12.0 is months away from being stable, and Windows on bhyve on FreeBSD 12.1 is a night and day performance difference to Windows on bhyve on FreeBSD 11.3. I can actually see TrueNAS Core 12 as a viable virtualization platform for Windows guests - once it stabilizes and no longer crashes bhyve on Windows boot, that is.

Virtualizing FreeNAS is doable and considerably more complex than not virtualizing it.

So how about this: Assemble hardware that could virtualize if so desired. Maybe minus the HBA to start out with. Then don't virtualize. Depending on your timeline, you might be able to test TrueNAS Core 12. Test the truly difficult use case, that Windows VM. Here's a video that teaches you how to install a Windows VM on FreeNAS / TrueNAS, with VirtIO, which is really the main requirement to have it work well. https://www.youtube.com/watch?v=YQk6jGJPA_Y . Then, make a decision. If you hate life with a VM on top of FreeNAS / TrueNAS, go for that ESXi idea. Just - why add another layer of complexity if you don't have to.

This is what you need to know if you ultimately decide to forge ahead with virtualizing: https://www.ixsystems.com/community...ide-to-not-completely-losing-your-data.12714/

 

[Yorick]

According to that data sheet, the asus board does support the Intel iGPU. You have an x2 m.2 slot, which is perfect for a PCIe ssd boot drive. You can also boot off a sata connector, of course.

I210at Intel Ethernet is fine.

No idea how well the board / bios supports vt-d. Maybe find some others who’ve used this for ESXi and can comment on the hardware passthrough stability.

 

[Newfoundland.Republic]

Take a look at the FreeNAS bhyve virualization requirements: https://www.ixsystems.com/documentation/freenas/11.3-U2/virtualmachines.html#

That may provide you some guidance.

 

[DuaneDibbley]

So I looked into the i3-9100 - it does support vt-d. No idea about that ASUS board - does it, does it do it well, and so on. There's a reason those who choose to virtualize don't do it on "consumer" brands. I know ASUS wants to be the next supermicro, but they're even further from that goal than ASRock Rack.

Fair point. I think I've managed to find a Supermicro Board. Newegg in the US will send it to Australia. Price is reasonable too. Couldn't find anyone locally which is mad!

I think the X11SCH-LN4F will do the job nicely. Seems to be the same as X11SCH-F except quad lan ports. Neither here nor there feature wise for me but it's actually able to be purchased... The newegg listing has X11SCH-LN4F-0 as the model name and the naming convention doesn't mention a 6th option... so hopefully it's not a secret character that means "half a chipset" or something!

This will teach you all you need to know about HBAs: https://www.ixsystems.com/community...-lsi-9211-9300-9305-9311-hba-and-variants.54/. Beware Chinese knockoffs on eBay; buy from reputable "pulled from a server farm" sellers, preferably near you. You want these in IT mode; the older 2008 model is just fine. IBM M1015 or Dell H200 are both common models that can be had for 40-50 USD.

Thank you - it's been added to the pre-construction pile of required reading.

Alright. I'm going to try and talk you out of this virtualization idea. First please read this: https://www.ixsystems.com/community...nas-in-production-as-a-virtual-machine.12484/

Sonar, plex, yada yada, those all run great as jails. Nothing at all wrong with running that directly on FreeNAS.

Okay - this sounds like at the very least a good idea to try before going silly with virtualisation.

Blue Iris is Windows, true. ZoneMinder isn't. Consider that instead.

Even if you MUST HAVE Blue Iris, TrueNAS Core 12.0 is months away from being stable, and Windows on bhyve on FreeBSD 12.1 is a night and day performance difference to Windows on bhyve on FreeBSD 11.3. I can actually see TrueNAS Core 12 as a viable virtualization platform for Windows guests - once it stabilizes and no longer crashes bhyve on Windows boot, that is.

Blue Iris isn't a must have - just the most prominent alternative I've seen mentioned around. Happy to give ZoneMinder et al a go too.

So how about this: Assemble hardware that could virtualize if so desired. Maybe minus the HBA to start out with. Then don't virtualize. Depending on your timeline, you might be able to test TrueNAS Core 12. Test the truly difficult use case, that Windows VM. Here's a video that teaches you how to install a Windows VM on FreeNAS / TrueNAS, with VirtIO, which is really the main requirement to have it work well. https://www.youtube.com/watch?v=YQk6jGJPA_Y . Then, make a decision. If you hate life with a VM on top of FreeNAS / TrueNAS, go for that ESXi idea. Just - why add another layer of complexity if you don't have to.

This seems like quite a reasonable plan.

So plan will be to start ordering:

CPU: Intel Core i3-9100 3.6 GHz Quad-Core Processor
Motherboard: Supermicro X11SCH-LN4F
Memory: 2x Crucial 16 GB Unbuffered ECC DDR4-2666 Memory
Storage: Some kind of SSD for boot.
Case: Silverstone CS380 ATX Mid Tower Case
Power Supply: SeaSonic FOCUS Plus Platinum 650 W 80+ Platinum Certified Fully Modular ATX Power Supply
= ~$1600 Australian

Thank you again for your advice!

ake a look at the FreeNAS bhyve virualization requirements: https://www.ixsystems.com/documentation/freenas/11.3-U2/virtualmachines.html#

That may provide you some guidance.

Sweet, seems everything I've specced will be fine.

 

[Newfoundland.Republic]

Storage: Some kind of SSD for boot.

Take a look on Amazon for the Kingston AS400 SSDs. I picked up a couple (120GB - bigger than needed, but for this price... ->) for CDN$20/each and mirrored them for boot.

 

[Yorick]

Good plan, looks solid to me. -O at the end of the SKU is indeed secret code that means “being sold by a reseller, not SuperMicro directly”. That’s it. LN4 does not, as you say, do anything for you, but if that’s what you can get, then go for it.

There are articles around here somewhere on how to get ZoneMinder to run. I think “on top of a Ubuntu VM” is reasonably straightforward, while “in a jail” might take some tinkering.
When doing VMs on bhyve, a rule that will help you is: VirtIO for disk and network, always.

 

[Yorick]

2x Crucial 16 GB Unbuffered ECC DDR4-2666 Memory

If you can get a 32GB stick of ECC, maybe from supermicro directly?, go for that instead. You will have more room for growth. You could go to 64 with just one additional stick, and still have two slots free.

 

[DuaneDibbley]

Take a look on Amazon for the Kingston AS400 SSDs. I picked up a couple (120GB - bigger than needed, but for this price... ->) for CDN$20/each and mirrored them for boot.

Cheap! I'll see what we get here.

“being sold by a reseller, not SuperMicro directly”

Ah that's much less scary!

LN4 does not, as you say, do anything for you, but if that’s what you can get, then go for it.

Literally the only C246 board I can get. Seems in Aus most supermicro distribution is by server suppliers who don't sell retail and just use as OEM parts. Some board were on some stores but a weird mix of super expensive or old. So yeah - I'll jump on this LN4!

It only has VGA out - I don't think I even have a VGA monitor anymore! Better find a dongle.

There are articles around here somewhere on how to get ZoneMinder to run. I think “on top of a Ubuntu VM” is reasonably straightforward, while “in a jail” might take some tinkering.
When doing VMs on bhyve, a rule that will help you is: VirtIO for disk and network, always.

Thanks - I'll search these out.

If you can get a 32GB stick of ECC

I'll see what I can find. Not a large performance loss from single channel?

 

[Yorick]

It only has VGA out

The good news is you don’t care. Instead of hooking up VGA, hook up Ethernet to the IPMI port. DHCP out of the box. There should be a sticker with the board of what the initial password for IPMI is. Once in, set a static IP, next-hop, DNS if you want, and change that admin password. Then do all your work from the built in HTML5 KVM. You won’t need a monitor and keyboard, ever, and you can power cycle the system without ever touching it.

Not a large performance loss from single channel?

Negligible, you won’t notice. The rule of thumb for FreeNAS is to use the largest stick the board can take. More RAM down the line trumps “two channels”, every time.

 

[Newfoundland.Republic]

The good news is you don’t care. Instead of hooking up VGA, hook up Ethernet to the IPMI port. DHCP out of the box. There should be a sticker with the board of what the initial password for IPMI is. Once in, set a static IP, next-hop, DNS if you want, and change that admin password. Then do all your work from the built in HTML5 KVM.

@DuaneDibbley - IPMI is wonderful. Plus, your version using HTML5 is wonderful - you don't have to deal with java like I do :-(

 

[Yorick]

If you can get a 32GB stick of ECC

Before you pull the trigger on that, more reading. It turns out the i3-9100 only supports up to 64GB of RAM. I don’t know whether it can support that 32GB stick. That’d be good to find out.
If you are going to stick with an i3 instead of a xeon, there’s not a lot of sense planning for an amount of ram you will never be able to use with that CPU.

 

[Yorick]

How much is a xeon E-2124G or E-2224G in your parts? If you can get it for a little above the i3 cost (which is true here but may not hold true in AUS), then I argue it’s worth it: Higher clock, more room for memory, same iGPU for Plex.

 

[DuaneDibbley]

The good news is you don’t care. Instead of hooking up VGA, hook up Ethernet to the IPMI port. DHCP out of the box. There should be a sticker with the board of what the initial password for IPMI is. Once in, set a static IP, next-hop, DNS if you want, and change that admin password. Then do all your work from the built in HTML5 KVM. You won’t need a monitor and keyboard, ever, and you can power cycle the system without ever touching it.

IPMI is wonderful. Plus, your version using HTML5 is wonderful - you don't have to deal with java like I do :-(

Oh my! I didn't realise it allowed that. I thought it was just for BIOS. That's amazing :D

How much is a xeon E-2124G or E-2224G in your parts?

i3 9100 = $198AU
Xeon 2124G = $472
Xeon 2224G = I can't even find anyone selling it.

I'm tempted. But it's a huge jump.

 

[Newfoundland.Republic]

You could check on eBay for a server pull...

 

[DuaneDibbley]

You could check on eBay for a server pull...

I believe me - I've been looking. However I've got a smallish house and a 8mo old. I'm lead to believe these proper servers are loud. My rack isn't deep enough either I think.

 

[Newfoundland.Republic]

Naaa... the drone will put them to sleep. Worst thing you can do is try to be quiet - it seems to wake them up. I used to vacuum under the crib when my son was an infant.

I meant look for a CPU that was pulled from a server.

 

[Newfoundland.Republic]

Your significant other - well, that's another issue :)