BUILD Hardware recommendations for FreeNAS on VMware ESXi

[JoeS]

Hi,

I'm new to FreeNAS and have had a tiny amount of experience with ESXi, after Christmas I'm going to build a white box ESXi server and (amongst other things) install FreeNAS on it. I'd like some suggestions on my hardware choices and upgrade strategy please.

In my desktop I currently have 4x1.5TB drives in a RAID 5 array and 2x1TB in a mirror. I've also bought a Antec 1200 case. This is my rough specification but I am completely open to suggestions and I don't want to buy myself in to a corner and find that my system has issues with ESXi/ZFS.

• AM3+ motherboard that supports 32GB ECC RAM, something like an Asus M5A97 R2. EDIT: M5A99X EVO R2 has confirmed support for IOMMU.
• Piledriver based 8 core CPU - FX 8320 or 8350
• Kingston Value RAM - 2x8GB ECC, I'll buy another 16GB when RAM prices become a bit more sensible.
• ~500W 80+ PSU
• M1015 SATA controller in IT mode
• 4x WD Red 3TB
• 256GB SSD - to run VMs off

Can anyone spot any issues with this hardware or can anyone recommend any better alternatives?
I'm going to create a RAIDZ2 VDEV with the 3TB WD REDs, what might happen is I run out of money and decide to only buy three hard drives and then buy the fourth a month or two later. If this happens then what I was thinking of doing is creating the RAIDZ2 in a degraded state (with only one parity drive) and then adding the fourth afterwards. Is this possible/advisable? What disadvantages does this hold over RAIDZ1? (I assume that there must be some or everyone would do it.)
After I fill this pool then I would like to add some more WD RED 3TB drives, I'm not sure if this is possible but what I was hoping to do is:

1. Buy four more 3TB drives.
2. Add one parity drive from the existing VDEV.
3. Add two 3TB virtual hard drives in ESXi (using the 4x1.5TB drives mentioned earlier.)
4. Create an 8 drive RAIDZ2 VDEV (With one drive missing)
5. Transfer the data across.
6. Add the missing 8th physical drive.
7. Use dd to image the two virtual drives on to the two remaining physical ones.

Is this possible? If so then would this mean that I had at least some degree of protection throughout the whole process? If not then it just means that I'll have to buy a few more drives which I'm sure I can find a use for.
Any advice would be great.
Thanks,
Joe

 

[JoeS]

Bump!

 

[cyberjock]

I gave up on replying when:

1. I saw your parts list and saw you chose a board with Realtek NICs despite the warnings.
2. I saw you are buying Asus which is far from server grade.
3. You are buying AMD despite the overwhelming number of people that have had problems with AMD boards.
4. You also didn't read my presentation where I explain(and the manual explains) that all disks in a vdev must be available and unformatted during vdev creation.
5. You started talking about adding virtual disks. If you read other stickies you'd see that "virtual disks" are just stupid to use for FreeNAS.

I just took the stance that if you aren't getting those right with the overwhelming body of information provided then you are probably looking for someone to validate your build. I don't validate builds. Nothing I'd recommend(or not recommend) can't be found in stickies or 30 minutes of searching the forums.

 

[Sir.Robin]

I have used two Asus AMD AM3 boards with FreeNAS since version 8.x and have had NO issues with them. Even the realtec nic's worked. :)

 

[cyberjock]

I have used two Asus AMD AM3 boards with FreeNAS since version 8.x and have had NO issues with them. Even the realtec nic's worked. :)

So? The problem isn't that Realteks don't work. It's that when you compare Realteks to Intel their pattern of problems and performance is vastly different.

Realtek complaints people see regularly:

-They randomly stop working
-They perform slowly
-They don't work at all
-They cause random unrelated errors

Intel complaints people see regularly:

-They don't work because of a driver bug
-They don't work because the Intel NIC is so new the driver isn't included with FreeNAS

Do some Realteks work fine for people? Sure.

But if you look at the problems Intel cards have those are easily fixed with an update. If you have a crap hardware NIC there is no driver that is going to fix that. You are just flat out screwed.

So why do we recommend Intel? Because for the meager cost of getting an Intel NIC you can rule out an extremely large set of problems instantly and forever.

So take the risk and see if it works out for you with Realteks, or spend $20 on an Intel NIC and have no doubt your NIC is not a source of problems or poor performance.

Speaking of which.. I just replied to http://forums.freenas.org/threads/install-of-freenas-9-2-0-stuck-in-reboot-loop.17144/ and I'm willing to bet his previously working NIC now doesn't work with the update for FreeNAS. Still think it's smart to put money on a Realtek?

 

[jgreco]

I have used two Asus AMD AM3 boards with FreeNAS since version 8.x and have had NO issues with them. Even the realtec nic's worked. :)

Irrelevant. His FreeNAS VM will undoubtedly have very nice virtual Intel ethernets. It is very unlikely to have virtual Realteks. :) The fun will be loading ESXi on here and the potential frustration involved in getting ESXi to recognize the hardware.

There's at least some overlap in what makes a board good for ESXi and what makes a board good for FreeNAS; you can often use other things and make them work, but why fight that on top of all the other complexity involved?

Maybe he will get lucky and find it works with little effort. I could see doing that for a $50 motherboard and $50 CPU. I'm pretty sure the 8111F will involve driver unpleasantness on ESXi though. And the components suggested are very close in cost to the commonly used Xeon/X9Sxx; you can easily pick much better options than some pricey AMD desktop board.

 

[Sir.Robin]

Well... where to start... maybe it was a bit offtopic.

I am not recommending Realtec. I am using Intel NIC's myself.
But i tested Realtec with no issues on a couple of boards.

My main point is you can't say it's all crap just because it's AMD based.

You can run FreeNAS just fine on AMD boards, with Intel NIC's if you wan't.

I've run AMD since the year 2000 on my main puter and with several different CPU's/boards through the years. Never had a "stability" issue.
Currently a Gigabyte FXAsomething on a 790 chipset with a PhenomII X6.

I have some AMD based HP DL585 at work running ESXi from 2.5 to 4.0 for several years. No stability issues.
I exchanged these with Cisco UCS wich is intel based though...

If you want to you can run FreeNAS on AMD systems just as fine as with Intel.
AMD/Asus gives you ECC support on the desktop models too.
Although the implementation won't be "server grade" it's better than nonECC and a good enough solution for home use.

If you wan't ESXi, guess what, you can have server grade mobos too from AMD. Maybe the cost won't be that much lower than Intel though...

With that said, i really like my X10SL7 (intel chipset/CPU) too :)

And nomatter what system you run, you'll need a backup solution.

 

[jgreco]

Yes, we've got some HP DL365's etc as well. This isn't an AMD vs Intel thing so much as it is a failure to identify appropriate hardware, especially when appropriate boards are similarly priced.

ESXi is finicky at best about hardware. The inclusion of a desktop grade board with Realtek is basically just askin' for trouble. I do not require anyone to take my advice or even listen to it, but I reserve the right to offer my criticism and opinions when asked, and the right to sigh and say "told ya so" ..

 

[cyberjock]

Exactly. Don't expect me to give you advice or even read your thread if you start using stuff we already have listed as not recommended stuff. I have better things to do than spend 20 minute explaining it to you when its already been explained to death dozens of times and in stickies! I'll tell you where to stick it and close the brower tab.

AMD seems to work fine for some people. But by far most people have problems. Not saying its an AMD problem, but by far more AMD systems have a problem than Intel. My guess(and I've said THIS before dozens of times) is that AMD simply provides a CPU. It's up to the motherboard manufacturer to make the board. Intel is more of a "system" with their CPU and chipsets. Intel also provides FAR more support for non-Windows OSes than AMD. Like it or not, but AMD laid off 1/2 of their programmers this summer that were involved in non-windows work. So you can't argue that AMD can still provide "excellent" support with those kinds of across the board cuts.

Like it or not, you are more likely to have a fully functioning system with Intel components than AMD. You are also more likely to have good support in the future with Intel than AMD.

I don't want AMD to go bankrupt, but I don't consider them a good choice for people that are trying to learn FreeNAS and want a no-hassle setup.

 

[jgreco]

I managed to largely avoid Intel thru much of the '90's and some of the 2000's... including such beauty as the AMD 486dx/4 100 in the ADZ fanless variant, and suckiness such as Cyrix. AMD's designs are generally superior, but Betamax lost despite technical superiority.

I would like to support AMD but won't be able to justify the risk of several more failed boards at $500 a shot, it gets really expensive to run staff out to do an onsite system board swapout, total cost per failure runs into several thousand dollars. I am not willing to underwrite Tyan's QC or support AMD for that sort of premium.

 

[Sir.Robin]

Exactly. Don't expect me to give you advice or even read your thread if you start using stuff we already have listed as not recommended stuff. I have better things to do than spend 20 minute explaining it to you when its already been explained to death dozens of times and in stickies! I'll tell you where to stick it and close the brower tab.

AMD seems to work fine for some people. But by far most people have problems. Not saying its an AMD problem, but by far more AMD systems have a problem than Intel. My guess(and I've said THIS before dozens of times) is that AMD simply provides a CPU. It's up to the motherboard manufacturer to make the board. Intel is more of a "system" with their CPU and chipsets. Intel also provides FAR more support for non-Windows OSes than AMD. Like it or not, but AMD laid off 1/2 of their programmers this summer that were involved in non-windows work. So you can't argue that AMD can still provide "excellent" support with those kinds of across the board cuts.

Like it or not, you are more likely to have a fully functioning system with Intel components than AMD. You are also more likely to have good support in the future with Intel than AMD.

I don't want AMD to go bankrupt, but I don't consider them a good choice for people that are trying to learn FreeNAS and want a no-hassle setup.

Right. I'm not disagreeing completely either.

And "back in the day" you are right about AMD just providing you with a CPU. The best ones were VIA made. Suckers bought SIS (wasn't it?).
But thats long ago. They have been providing their own chipsets for years. Quite good ones too.

They provided 6x SATA 6Gbps for several years before Intel finally this year dropped the 2x6Gbit + 4x 3Gbit SATA ports and got 6x 6Gbps in place for example.

Again, just saying that AMD isn't that bad. Filled with problems.

 

[cyberjock]

Again, just saying that AMD isn't that bad. Filled with problems.

Empirical evidence on this forum disagees with you. That's fine, we can agree to disagree. I still read every thread and by far people have more problems with AMD setups than Intel when it comes to compatibility. Just as a simple off-the-top-of-my-head example, look at how many people are genuinely PISSED at AMD because they can't do CPU temperature monitoring. I haven't seen that with a single Intel yet.

 

[hoboville]

Yes, we've got some HP DL365's etc as well. This isn't an AMD vs Intel thing so much as it is a failure to identify appropriate hardware, especially when appropriate boards are similarly priced.

ESXi is finicky at best about hardware. The inclusion of a desktop grade board with Realtek is basically just askin' for trouble. I do not require anyone to take my advice or even listen to it, but I reserve the right to offer my criticism and opinions when asked, and the right to sigh and say "told ya so" ..

I've tried ESXi on AMD platforms, and it purple-screened on me, wouldn't install right. The hardware worked fine, but when I checked HW compatibility on VMware website, well, my HW wasn't even on there. Oh well, I like Intel better anyway, their virtualization features have never failed me...whilst AMD....

To OP: Sure, AMD is cheap for lots of cores, but Bulldozer / Piledriver are optimized for integer workloads, which is fine...to a point. Frankly, the 4 core HT Xeons will be just as fast, if not faster if you opt for the newer Haswell Xeons.

 

[cyberjock]

Personally, my main reason for never(and I mean never) being satisfied with AMD is their lack of support with stuff that people take for granted(for example, CPU temp). Every AMD video card, every AMD laptop, every AMD desktop I have ever built or bought has performed horribly with no problem in sight. As soon as I go to Intel, the problem goes away. For that reason I haven't spent money on AMD in more than 7 years, and I have no intention of changing that. Every time I've tried AMD it has been after a 4-5 year hiatus from even trying AMD, and every time a friend tells me "AMD is so different now.. blah blah blah.. give them a chance". And every single time I throw my freakin' money down the toilet and want to punch a child in the face because I got sold crap yet again.

I don't provide advice on AMD because I have no idea what I'm talking about if I start talking AMD. But just from observing other users, their complaints on AMD hardware are quite widespread while Intel's are usually things such as failed components or insufficient processing power. Neither of which is usually an "Intel problem".

 

[Sir.Robin]

Allright… so I finally made some time for a test J
I am not trying to make a flame war or say one is better than the other one.
But you can save some money on going AMD.

As I said before, I have used AMD on my personal PC for over a decade.
Duron, Athlon, AthlonXP, Athlon 64, Athlon 64 X2, Opteron, AthlonII X4 and PhenomII X6 (current).
I use my PC for different stuff, but mainly office, VMware Workstation, surfing and gaming.
This has worked just fine for me and I never had any hardware problems with these setups.

Now the case here was to recommend a system for ESXi/FreeNAS use and JoeS are looking at a AM3+ setup.
Will ESXi install on this and let you run VM’s? Sure. I’m running vSphere 5.1 on my M5A88-M as I’m writing this.
Will IOMMU work? Not sure. Maybe on he’s mobo choice, but it does not seem to be supported on my M5A88-M board. Passthrough is not available. So that suck for FreeNAS as a VM usage.

My M5A78L-M (with ECC support) board + FX-4100 costed me about the same as a i3 CPU.
So theres definetly some money to save, but offcourse you won’t have all the bells and whistles of a SP X9/X10 board.

Also, if your buying a full setup with drives, casing, PSU and maybe a HBA… the savings obviously aren’t that much.

 

[raidflex]

To me my data is very important. When I am building a system to house important data and server it up I want hardware that just "works" and is reliable. I do not want to be worrying about NIC issues, reliable temperature reporting, motherboard reliability and compatibility. If the underlying hardware for Freenas is not sound then to me all the benefits to ZFS and Freenas will not matter.

Now when adding the complexity of ESXi I would really want to make sure I do no skimp on the hardware being used.

Really cost is the only factor I see with going to AMD instead of Intel and even then we are not talking about many hundreds or thousands more usually about $150-200 more will get you a server board and at least a low end Intel part.