Virtualise on i3? Overly Ambitious Beginner Build?

[DuaneDibbley]

Naaa... the drone will put them to sleep. Worst thing you can do is try to be quiet - it seems to wake them up. I used to vacuum under the crib when my son was an infant.

I meant look for a CPU that was pulled from a server.

Ha! My bad haha. Had a quick google - nothing available domestically on ebay used. This is crazy - I've never had to try so hard to find bits!

our significant other - well, that's another issue

I'm amazed she's remotely enthusiastic about a second NAS... Don't want to push my luck too far - even if I sell it as a baby white noise generator.

 

[DuaneDibbley]

Before you pull the trigger on that, more reading. It turns out the i3-9100 only supports up to 64GB of RAM. I don’t know whether it can support that 32GB stick. That’d be good to find out.
If you are going to stick with an i3 instead of a xeon, there’s not a lot of sense planning for an amount of ram you will never be able to use with that CPU.

Ah crap - yes. I've been looking at that Supermicro mobo spec page so long and seeing 128gb I'd forgotten that was xeon only. Thanks for the reminder. Might just do 2x 16gb I think.

 

[Newfoundland.Republic]

If you can only to to 64 GB with the i3, I'd probably go with the 16GB if it was cheaper...

 

[Yorick]

Ah crap - yes. I've been looking at that Supermicro mobo spec page so long and seeing 128gb I'd forgotten that was xeon only. Thanks for the reminder. Might just do 2x 16gb I think.

I concur that's your best bet, since Xeons are so expensive in your neck of the woods. Around here, an i3-9100 goes for USD 135, and a Xeon E-2124G for USD 230. Fair enough, that's a decent jump. Go for that i3-9100 (just not the 9100F, I think that one doesn't have an iGPU), and 2x16GB, and call it done. It's more than enough for your use case.

 

[DuaneDibbley]

If you can only to to 64 GB with the i3, I'd probably go with the 16GB if it was cheaper...

2x 16gb is the plan now.

I concur that's your best bet, since Xeons are so expensive in your neck of the woods. Around here, an i3-9100 goes for USD 135, and a Xeon E-2124G for USD 230. Fair enough, that's a decent jump. Go for that i3-9100 (just not the 9100F, I think that one doesn't have an iGPU), and 2x16GB, and call it done. It's more than enough for your use case.

Over twice the price in my use case is just a bit too much.

I'll order the mobo from Newegg now and sort the rest out later.

Thanks again for everyone's help.

 

[DuaneDibbley]

I ended up going on a buying spree last night as a few things I had on my initial purchasing plan started going into no stock available!

Ended up buying:

• - X11SCH-LN4F
• - i3 9100
• - 2x 16gb Kingston 2400Mhz DDR4 ECC Ram
  • The Crucial 2666Mhz part number didn't seem to exist anywhere officially on Crucial or Micron's websites and the vendor I had found it on had it listed as CL17 not CL19 which the few other places it seemed to exist had it listed at. I decided since the i3 is 2400Mhz only I'd play it safe. Might be an issue if I upgrade to Xeon in future but that's a problem for then...
• - Seasonic Focus Plus Platinum 650W
• - 256gb Silicon Power nvme SSD
  • Cheapest and smallest nvme drive I could find. Decided to keep all 8 SATA ports for storage drives. I couldn't see it noted anywhere that using a nvme drive would disable a SATA port so win! No DRAM cache but I figure it won't matter.
• Fractal Design R5 as the Silverstone one I had initially chosen went out of stock.

 

[Yorick]

Merry winter shopping spree. Wife still on board with all this? :)

 

[Newfoundland.Republic]

Wife still on board with all this?

...and then there was the sound of silence :)

 

[mpeck]

Maybe add the mention "WAF" in the FreeNAS prerequisites. :)
In any case, I will add it on my checklist for my future build.

 

[DuaneDibbley]

Merry winter shopping spree. Wife still on board with all this?

...and then there was the sound of silence

hahah! The wife is actually pretty into this actually! I kinda thought she was going to be less than enthused but hey!

Maybe add the mention "WAF" in the FreeNAS prerequisites.

WAF?

 

[Yorick]

WAF

Interweb-ese for "Wife Approval Factor". Which, as so many things on the interwebs, is mildly misogynistic. It assumes men are into technical stuff and women are not. Maybe "SAF", "Spouse Approval Factor" would be better.

 

[Newfoundland.Republic]

Maybe to be even more inclusive: SOAF - Significant Other Approval Factor

 

[DuaneDibbley]

Maybe to be even more inclusive: SOAF - Significant Other Approval Factor

The SOAF/WAF is high!

I had all the bits arrive yesterday - crazy fast shipping from US Newegg for the mobo!

It's all installed - that IPMI and iKVM is amazing! Seriously cool stuff!

I kinda just clicked buttons and stuck with defaults (incl encryption) to make a pool out of my (in the end only 3) 3tb drives. RAIDZ1. Set up a dataset and made a SMB share.

If I transfer my test ~20gb RAR to it it saturates the connection. Pegged on 113MB/s in Windows file transfer. However if I use an app I bought called LAN Speed Test set to packet sizes of 1MB - 30MB (to simulate writing/updating RAW files for photo stuff) I get a write speed of 169mbps max vs my old synology with the SMR reds in it getting 493mbps same test.

I definitely don't know enough yet to work out what's causing this. I've not run iperf3 but given then 20gb file saturates my 1gig link I'd say it's not a connectivity issue.

It's quite late now so I'll probably stop fiddling for now - any suggestions as to what areas to look for as a cause?

Big Edit: In case that app is rubbish I grabbed my Mass Effect 3 game install folder and copied it to the FreeNas and the Synology and timed them. I figured a game install folder will have a large mix of file sizes and was ~10gb in total.

FreeNAS completed in 3min37s
Synology completed in 5min25s

So yeah... in non synthetic testing the FreeNAS is spanking the old Synology! Perhaps not an issue after all.

 

[Yorick]

As in, you enabled encryption? Probably goes without saying and: Make triple backups of those encryption keys.

Encryption with FreeNAS is an odd beast. Each drive is encrypted separately, then unlocked and ZFS runs on top of it all, none the wiser.

This usually works well, and has in some cases led to catastrophic data loss, see the 11.3U2 bug. That’s a very specific set of steps and, still. It’s not the first time, either.

TrueNAS 12 will bring ZFS native encryption on a per-dataset level. While key management is still as necessary as with any encryption, I am inclined to trust this more than the encryption ixSystems has in place now, with all respect for ixSystems engineers.

 

[DuaneDibbley]

Yeah, encryption is on. This is a test pool pretty much for tesing and fiddling for the moment. I'll likely crash it entirely before recreating and moving some stuff across off the old NAS. I don't really store anything that would benefit from encryption so it'll quite likely find it self disabled in future :)

 

[Yorick]

So, couple things that aren’t default that you’ll want to get a handle on:

- Config backup. There already is a daily one but it’s not terribly visible. Consider adding your own and syncing it to cloud. I have scripts for that on github.
- Email notification from both IPMI and FreeNAS. Keep in mind you will likely need an app password for this if your email provider does any kind of 2FA / MFA.
- set up regular scrubs on a schedule
- set up short and long smart tests on a schedule

Anything major I’m missing? Basically, make sure the box squawks when something goes wrong with the storage or the other hardware.

 

[DuaneDibbley]

So, couple things that aren’t default that you’ll want to get a handle on:

- Config backup. There already is a daily one but it’s not terribly visible. Consider adding your own and syncing it to cloud. I have scripts for that on github.
- Email notification from both IPMI and FreeNAS. Keep in mind you will likely need an app password for this if your email provider does any kind of 2FA / MFA.
- set up regular scrubs on a schedule
- set up short and long smart tests on a schedule

Thank you again. I'll do some reading on each of these soon.

I am already thinking of what I can do next... Initial impressions seem to be that the 256gb SSD is a waste as boot drive on straight FreeNAS system. It was the smallest I could get though. Perhaps I shall plug a USB drive into the USB port on the motherboard and run freenas from there and use the SSD for something else like separate drive for Jails or VM if such a thing is possible. I now feel like my 1gig home network sucks lol.

 

[Yorick]

Perhaps I shall plug a USB drive into the USB port on the motherboard

It'll die on you by overheating. Boot from SSD. True that 250GB is overkill for this build, but as I recall, the idea was to discover whether it was adequate to your needs and start over on VMWare if needed.

You have options for smaller SSDs, including USB to SATA M.2 2242 adapters. For example https://www.aliexpress.com/item/32793846090.html with https://www.amazon.com/Transcend-128GB-MTS430S-Solid-TS128GMTS430S/dp/B07KG2G152/?ozlp-20

else like separate drive for Jails or VM

Possible, but necessary? One, no redundancy, so that's not great. And then, your actual files shouldn't be inside the VM or the jail anyway, they should live on your pool and be mounted in (jail) or shared in (VM). The latter has to do with how ZFS handles block storage. Keeping, say, video files inside the VM is sub-optimal, and I'd argue the same for a jail, though for different reasons such as "separate snapshots are good".

You can add the SSD as Cache (L2ARC), but unless you discover that in your specific use case, ARC is not sufficient, it'd actually make things a tiny bit worse, rather than better.

One use case we're exploring elseThread is a system with 1.7 million files, where metadata won't fit into 32GiB and RAM can't be expanded. A "metadata only" small L2ARC can make sense there.