Esxi 6.5 + FreeNAS

[markymark832]

Hello people.
Probably the 1000th time new people like myself ask similar after similar question regarding a setup configs.
but here i am doing the same. SO any advice appreciated pls... :D
Background
I'm looking to migrate off my 6yr old Synology box ( starting to get smart errors) onto a new NAS
Originally i pimped my Synology up with various apps (sickbeard/transmission/plex/ccam server/vpn etc etc) which over time i have migrate these off into my VM estate and their own dedicated machines. so now my nas is just a nas.. nfs/smb file server and a bit of music streaming etc.
i currently run two esxi hosts which i am looking to remove one ( power hungry HP DL385 G6) but also looking to get a new NAS up and running,
I stayed away from FreeNAS for a while due to the fact that everyone who's everyone was saying it doesn't play to nice virtualization (think cyberjock was one of the main adversaries ;)) . Recently on reading a few newer posts /guides it looks like there's more appetite for this kind of config.
Requirement
SO what i am looking for is a box that i can
a) replace my second esxi host for something a little more 24/7 power friendly, this will host a few machines.. nothing meaty
b) run a Virtualised FreeNAS that has dedicated pass through to a disk controller /disks..

the link below shows someone who has achieved this, what i am wondering is are there any issues with what he's setup?
https://guyrobottv.wordpress.com/2017/01/03/virtualised-2017-home-nas-build-freenas-9-and-esxi-6-5/

i put together a rough list of parts i'm looking to buy to achieve this
i'm just unsure on disks ( for the array) looking at WD green 6tb ?

Code:

CPU: AMD FX-6300 3.5GHz 6-Core Processor  (£81.60 @ Aria PC)
CPU Cooler: Noctua NH-L9a 33.8 CFM CPU Cooler  (£31.30 @ Kustom PCs)
Motherboard: Gigabyte GA-990FXA-UD3 Ultra ATX AM3+ Motherboard
Storage: Samsung 850 EVO-Series 500GB 2.5" Solid State Drive  (£134.99 @ Aria PC)
Video Card: MSI GeForce GT 710 1GB Video Card  (£23.20 @ Aria PC)
Case: Aerocool DS 200 ATX Mid Tower Case  (£52.81 @ CCL Computers)
Power Supply: Corsair RMx 750W 80+ Gold Certified Fully-Modular ATX Power Supply  (£114.05 @ CCL Computers)
Other: Crucial 48GB (16GBx3) DDR3-1866 RDIMM Memory Kit  (£308.99 @ Ebuyer)
Other: LSI SAS2008 8-Port SAS HBA
Other: Quad Network Adapter
Total: £746.94
Prices include shipping, taxes, and discounts when available

any pointers wanted thanks :)

Mod note - removed PCPartPicker links

 

[m0nkey_]

First off, AMD. FreeNAS is known to have issues running on systems with AMD CPUs. You should look into an Intel based system.

The motherboard does not support ECC and it has an integrated Realtek NIC, which will cause performance issues down the road. It's basically a gaming motherboard.

I suggest you take a look at the hardware recommendation guide: https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

As for following random blog posts from the Internet, this should be avoided at all costs. If you want to run virtualized FreeNAS and you're injecting NIC drivers into the ESXi install image, you're doing it wrong. Most of the recommendations is to run FreeNAS on bare metal.

 

[melloa]

First off, AMD. FreeNAS is known to have issues running on systems with AMD CPUs. You should look into an Intel based system.

2nd that.

Also recommended by @Spearfoot: https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

 

[Ericloewe]

which will cause performance issues down the road.

Or outright not work. Gotta love the RTL8111 series.

 

[AdrianB1]

As there good people wrote before me, do not go that route. I did and I know: my main computer has a FX 6300 CPU and a Realtek network chip, the FreeNAS box next to it is also using an AMD CPU and Realtek network. While I am not into an Intel/AMD debate (my next CPU will be an AMD, but Ryzen), the FX 6300 is an old an inefficient design, my electricity bill confirms it. As for the network, I added Intel NIC's to both computers and it was worth doing it.

How did you got into that strange memory configuration of 3 DIMMs, RDIMM? It does not make sense, it is not an i7 with 3 memory channels, AMD's have 2 channels so you put the memory in single DIMMs or in pairs, but not 3 at a time. Also you need unbuffered, not registered DIMMs.

A 750W power supply seems to be an overkill for such a computer; you need about 120W for the motheboard, CPU and RAM (peak) and another tens for the disks and controller. I am not suggesting you buy a 200W power supply, but anything that delivers real 250W or more is enough. Go to Skylake or Kaby Lake, you shave another 20-30W or more - some from the CPU, about 10W from the motherboard chipset. I don't recommend going to Ryzen sooner than a few months, I will not do it (yet) so I am not telling others to take that risk: it is a nice CPU + chipset on paper, but untested, especially with FreeNAS.

What storage do you want to use? I see a SSD for boot, what else? You put on the list a LSI controller, but do you have any disks with it?

 

[markymark832]

Thanks for all the replies. Saved me from a few mistakes appreciate it
so updated list

Code:

CPU:  Integrated with Motherboard
Motherboard: Supermicro X10SDV-4C-TLN2F Mini ITX Xeon D-1521 Motherboard  (£496.85) 
Memory: Corsair Vengeance LED 64GB (4 x 16GB) DDR4-3000 Memory  (£369.63 @ CCL Computers) 
Storage: Western Digital Red 6TB 3.5" 5400RPM Internal Hard Drive  (£216.56 @ More Computers) 
Storage: Western Digital Red 6TB 3.5" 5400RPM Internal Hard Drive  (£216.56 @ More Computers) 
Storage: Western Digital Red 6TB 3.5" 5400RPM Internal Hard Drive  (£216.56 @ More Computers) 
Case: Fractal Design Node 304 Mini ITX Tower Case  (£69.03 @ Ebuyer) 
Power Supply: SeaSonic 450W 80+ Gold Certified Semi-Modular ATX Power Supply  (£73.90 @ Amazon UK) 
Case Fan: Noctua NF-A6x25 17.2 CFM  60mm Fan  (£11.63 @ Ebuyer) 
Total: £1670.72

I have a m2 SSD 250gb hdd so i will use that and a usb for ESXI boot, and Core VM's
because this boards supports vt-d then it should allow me to passthrough the sata controller,

All i will need to do other than the ESXI and freenas install is change the timing of the disk needle parking.( wd3idle or what ever it's called.

Any other thoughts.. before i pull a very expensive trigger..?

thanks for all replies :)
Mark

 

[m0nkey_]

Memory: Corsair Vengeance LED 64GB (4 x 16GB) DDR4-3000 Memory

Why that memory? It doesn't have ECC. Choose RAM that supports ECC.

because this boards supports vt-d then it should allow me to passthrough the sata controller

No, it will not. Integrated components cannot be passed through to guests. You will need to get yourself a HBA to pass through. You can usually find the IBM M1015 on eBay rather cheaply. Just avoid anyone selling from China or Hong Kong as they are often counterfeit.

All i will need to do other than the ESXI and freenas install is change the timing of the disk needle parking.

The more recent WD REDs do not need such a treatment.

Edit: The motherboard you selected is designed to be in a 1U rack mount case with sufficient air flow and to be in a cool environment, such as a server room. This motherboard would not be suitable for the case you have selected.

Maybe you should consider the ASRock Rack E3C236D2I and if get a E3-Xeon if you still want to go the Xeon route. Although an Core i3 would be more cost effective. Both CPUs support that board and ECC RAM.

 

[snaptec]

Only 3 6tb disks?
Raidz1 is not a good option with such large disks


Gesendet von iPhone mit Tapatalk

 

[markymark832]

Thanks for the replies

m0nkey,
the reason for the board as it supports more than 32gb of ram, yes it's not EEC ram, but that's only really needed if there's power issues no? i will have a ups
and yes it is a Server board. hence the additional fan for cpu heatsink
+ it has been recommend earlier in this thread @Spearfoot: https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

thanks for the heads up with the onboard not being able to pass through.. ( i will look for the HBA card now)

snaptec
Again i have little knowledge right now about this, but raidz1 = raid 5 yeah? why would this not be a good option? 1 disk loss for 2/3 the total storage.. open to suggestions though.. i currently have about 6tb of data that will be copied across once built.. i can mirror them to start with unless i purchase more? I just thought raid 5 (raidz1) was the best price / data integrity trade off

 

[m0nkey_]

it has been recommend earlier in this thread

But you appear to have ignored even what was in that random Internet blog. Even in that blog they use ECC and a LSI HBA.

If memory is a concern, look at the Micro ATX or full ATX motherboards than can take 64GB+ RAM.

Since you're willing to spend a lot of money, you need to make sure you get this right the first time.

 

[markymark832]

true M0nkey
list edited with EEC and the LBA included... i'll have a poke around for alternatives in the MB section...
thanks

Code:

CPU:  Integrated with Motherboard
Motherboard: Supermicro X10SDV-4C-TLN2F Mini ITX Xeon D-1521 Motherboard  (£496.85)
Memory: Kingston ValueRAM 16GB (1 x 16GB) DDR4-2133 Memory  (£137.59 @ More Computers)
Memory: Kingston ValueRAM 16GB (1 x 16GB) DDR4-2133 Memory  (£137.59 @ More Computers)
Memory: Kingston ValueRAM 16GB (1 x 16GB) DDR4-2133 Memory  (£137.59 @ More Computers)
Memory: Kingston ValueRAM 16GB (1 x 16GB) DDR4-2133 Memory  (£137.59 @ More Computers)
Storage: Western Digital Red 6TB 3.5" 5400RPM Internal Hard Drive  (£216.56 @ More Computers)
Storage: Western Digital Red 6TB 3.5" 5400RPM Internal Hard Drive  (£216.56 @ More Computers)
Storage: Western Digital Red 6TB 3.5" 5400RPM Internal Hard Drive  (£216.56 @ More Computers)
Case: Fractal Design Node 304 Mini ITX Tower Case  (£69.03 @ Ebuyer)
Power Supply: SeaSonic 450W 80+ Gold Certified Semi-Modular ATX Power Supply  (£73.90 @ Amazon UK)
Case Fan: Noctua NF-A6x25 17.2 CFM  60mm Fan  (£11.63 @ Ebuyer)
Other: sourcingmap® Mini 4Pin PC Computer Case Mainboard Buzzer Speaker Beep BIOS Alarm  (£1.72 @ Amazon UK)
Other: M1015-SATA-SAS-HBA-Controller-RAID-6Gbps-PCIe-x8-wie-9220-8i (£115.21)
Total: £1968.38
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2017-03-01 21:45 GMT+0000

 

[snaptec]

raidz1 is not recommended for drives larger than 1 TB.
If 1 disk fails the risk that another fail during resilver is very high.
Recommended is raidz2.

 

[du5tin]

Pretty sure if you are using large hard drives (larger than 3TB) the recommend config is RAID 10 or the equivalent mirrored drives/striped VDevs on ZFS. So four drives needed. Lots of information online about why this is the case (darn irrecoverable read errors!). Also I have seen lots of recommendations to stripe a couple VDevs instead of using raidz to increase your IOPS for iSCSI/VMware type use-cases.

I was running a test FreeNAS environment inside VMware ESX 6.0 on a Dell C1100. I was able to pass the built-in LBA SATA controller to the FreeNAS instance without issues. The VM was configured with 32GB RAM (out of 72GB on the box) and 4x 3TB WD Red hard drives. It never felt very snappy, but it was good enough for my purposes at the time. It was a bit of a pain to run VMs on the FreeNAS virtual storage running on the same VMware host. The iSCSI takes a while to pick up on a restart (usually had to refresh it manually) so you have to be careful about what order you shut things down or started them up. Once it was in a steady state with the FreeNAS instance running it was ok.

I found I wasn't using the machine that much for VMs so I recently rebuilt the C1100 as a stand alone FreeNAS box and it seems to run better now that FreeNAS is on bare metal. All I have is anecdotal evidence but I would recommend installing FreeNAS on bare metal and doing all your VM stuff on a different machine.

I realize you are getting out of a HP G6 rack server but there might be some refurb'd server deals in your budget that would also work for your use-case. I get most of my gear through 'donations' from friends at IT companies or online through classified ads. It is fun to build a box from scratch though. :)

 

[wblock]

yes it's not EEC ram, but that's only really needed if there's power issues no?

No. RAM can have bit errors in routine operation even when everything is fine, and that is what ECC corrects.

 

[markymark832]

Thanks for all replies... i am learning as i go.

"du5tin,I realize you are getting out of a HP G6 rack server but there might be some refurb'd server deals in your budget that would also work for your use-case. I get most of my gear through 'donations' from friends at IT companies or online through classified ads. It is fun to build a box from scratch though. :)

very valid point, but for me low power usage is also up there on list of "has to be" reused server hw like my g6 server, seemed a great idea at the time.. i just see it gobble pennies of elec.
I'm already running a dell t5600 24x7 with esxi 6.5 pimped up with memory and ssd etc for data stores

ok new list below taking into account running raidz2, EEC and sizing for airflow, the motherboard is now a flex ATX with LSI 2116 HBA controller on, which can provide 16 sata connections ( which has a 7 year product life) 2 expansions for future HBA if needed
i have a m2 ssd already that i will add for data stores along with the msata i will be buying, + a usb for esxi boot

Code:

CPU:  Integrated with Motherboard
Motherboard: Supermicro X10SDV-4C-7TP4F flex ATX Xeon D-1518 Motherboard  (£619.14)
https://www.supermicro.com/products/motherboard/Xeon/D/X10SDV-4C-7TP4F.cfm

Memory: Kingston 32GB (1 x 32GB) Registered DDR4-2133 Memory  (£192.65 @ BT Shop)
Memory: Kingston 32GB (1 x 32GB) Registered DDR4-2133 Memory  (£192.65 @ BT Shop)

Storage: Samsung 850 EVO 250GB mSATA Solid State Drive  (£90.96 @ BT Shop)
Storage: Western Digital Red 4TB 3.5" 5400RPM Internal Hard Drive  (£139.82 @ More Computers)
Storage: Western Digital Red 4TB 3.5" 5400RPM Internal Hard Drive  (£139.82 @ More Computers)
Storage: Western Digital Red 4TB 3.5" 5400RPM Internal Hard Drive  (£139.82 @ More Computers)
Storage: Western Digital Red 4TB 3.5" 5400RPM Internal Hard Drive  (£139.82 @ More Computers)
Storage: Western Digital Red 4TB 3.5" 5400RPM Internal Hard Drive  (£139.82 @ More Computers)

Case: Fractal Design Node 804 MicroATX Mid Tower Case  (£86.99 @ Ebuyer)
Power Supply: SeaSonic 450W 80+ Gold Certified Semi-Modular ATX Power Supply  (£73.90 @ Amazon UK)
Case Fan: Noctua NF-A6x25 17.2 CFM  60mm Fan  (£11.63 @ Ebuyer)
Other: sourcingmap® Mini 4Pin PC Computer Case Mainboard Buzzer Speaker Beep BIOS Alarm  (£1.72 @ Amazon UK)

really appreciate everyone's feedback on this thread... made me think/change direction allot.
regards
Mark

 

[Sir.Robin]

Running FreeNAS on ESXi 6.0 U2 myself. Going 6.5 when driver rollup for 6.5 is released. Not a problem, but do it correctly and read the guides here on the forum. Keywords, ECC, VTd and VM reservation on the CPU and RAM. :)

 

[markymark832]

update for anyone interested: I took the plunge and i am happy to say i have esxi 6.5 running with 64gb ecc, with the HBA passed through to my Production FreeNAS Corral and also running a Dev FreeNAS Corral off the same box, using a vxd hdd as a temp data storage.. all purring like a kitten. Very happy indeed
thanks for all comments and advice.:):):):)

 

[freakinlaser]

No, it will not. Integrated components cannot be passed through to guests. You will need to get yourself a HBA to pass through. You can usually find the IBM M1015 on eBay rather cheaply. Just avoid anyone selling from China or Hong Kong as they are often counterfeit.

Are you sure that's right? Pretty sure that with VT-d you can pass an on-board (chipset) SATA controllers to guest just fine - seems to be working just fine on my ESXi 6.5 setup.

 

[Ericloewe]

Are you sure that's right? Pretty sure that with VT-d you can pass an on-board (chipset) SATA controllers to guest just fine - seems to be working just fine on my ESXi 6.5 setup.

Nope, it doesn't work in about 50% of cases.

 

[Frznfngrs]

Are you sure that's right? Pretty sure that with VT-d you can pass an on-board (chipset) SATA controllers to guest just fine - seems to be working just fine on my ESXi 6.5 setup.

Nope, it doesn't work in about 50% of cases.

Wrong. You can pass through integrated components. I do it every day. The only way you will know if it works is to try it. I pass through controllers of all types, and integrated NICs, M2 slots, etc. VT-D allows for it.