Kabini - next step up from n40l microserver?

[Middling]

I've been using my HP n40l microserver with 8GiB ECC and 5x 2TB drives in RAIDZ on FreeNAS for the past two and a half years. It's performed great as a fileserver (and also, for the past several months, as a personal web and CUPS server using jails) but i'm running out of space and also want to switch to RAIDZ2.

I did have a whole new build planned out using the AMD FX-6300, an IOMMU-compatible motherboard, 32GiB ECC and 8x 3TB drives. It would have been a bit of a power-hog but i'd been thinking about also using it for virtualisation (by switching away from FreeNAS).

I saw an article on Kabini yesterday and it got me thinking. For the same price as the FX-6300 and motherboard i could get an Athlon 5350, motherboard and IBM M1015. Sure the performance will be a lot lower but so will the power consumption. And the Athlon 5350 seems to have at least twice the performance of the n40l, which has never been a problem for me.

I could use it as a pure FreeNAS storage box and repurpose the microserver for basic virtualisation experimentation, upgrading next year should i need to. Even with both machines running i'll probably still save power over the FX-6300.

Looking for comments on this idea, any cons i might have overlooked, etc.

In particular i'd like to know if anyone is currently running FreeNAS on a Kabini system and what the performance is like. I'd also like confirmation that ECC RAM is correctly utilised. The APU supports ECC but none of motherboards have explicitly stated support.

 

[cyberjock]

Confirming if ECC is being utlized for a given set of hardware for AMD is near impossible. Nobody I can find anywhere has a foolproof way to validate how AMDs work. Intel is documented in their specs and there are many ways to validate it. If you are hellbent on ECC (which in my opinion you should be) then you shouldn't go AMD without the ability to validate this yourself. Over on the hardforums some people were flat out lied to by a support person and someone found out the hard way that they were lied to.

Anyway, enough about that. Do you know that FreeNAS can run virtualbox in a jail? If not check it out. Quite a few people are using it and it works well for them. Performance isn't as high as going with an ESXi server or Xenserver, but it does get the job done. Just figured I'd mention this in case you didn't know.

 

[AlainD]

Confirming if ECC is being utlized for a given set of hardware for AMD is near impossible. Nobody I can find anywhere has a foolproof way to validate how AMDs work. Intel is documented in their specs and there are many ways to validate it. If you are hellbent on ECC (which in my opinion you should be) then you shouldn't go AMD without the ability to validate this yourself. Over on the hardforums some people were flat out lied to by a support person and someone found out the hard way that they were lied to.

Anyway, enough about that. Do you know that FreeNAS can run virtualbox in a jail? If not check it out. Quite a few people are using it and it works well for them. Performance isn't as high as going with an ESXi server or Xenserver, but it does get the job done. Just figured I'd mention this in case you didn't know.

Hi

I always thought that ECC checking was extremely difficult. How are they validated on Intel is there software for?

 

[Middling]

I'm not averse to using Intel, it's just the cost of doing so. The closest Intel solution would be something based on the new c2000 Atoms. They have more features and can access more RAM but the cheapest i've found such a solution is around £230 as opposed to the £65 for the Kabini package.

I did consider the C2750D4I when i initially started choosing the parts for my new system. In the end i decided that the power savings over the FX-6300 system weren't worth the additional costs of the Intel hardware.

I think i will pull the trigger on the Kabini. It's only £65 and if it doesn't support ECC i can always use it for another project (maybe an OpenELEC box, or OpenStack compute node). I'll report back on whether ECC is is utilised or not.

I'm aware FreeNAS now supports Virtualbox but i've used that on the desktop for years and am familiar enough with it already. I'm more interested in other virtualisation technologies. Maybe ESXi, but i'm more interested in KVM-based stuff (oVirt, OpenStack, etc) as i think this is the future of virtualisation.

 

[Middling]

Ordered an Athlon 5350 and Gigabyte's microATX motherboard (along with 4u case and PSU). Should be with me mid-week. I'll test with ECC and report back.

 

[Ericloewe]

How do you plan to figure out if ECC is actually working?

 

[cyberjock]

How do you plan to figure out if ECC is actually working?

Exactly.. he hasn't planned that far ahead. ;)

 

[AlainD]

Like all memory go either trust it or test it thoroughly.

Having ECC options in BIOS is the first way to check it.

To be sure you need ecc-ram with the option of generating single bit errors and a lot of time and extended test facilities (emc tests, load tests, temperature tests,..).
--> This is limited to a few big reputable server makers and those will sell complete servers including their selected memory.

According a research paper about Google servers 8% off their ECC ram sticks correct a single bit error yearly, to give you an impression of the odds.

 

[Middling]

How do you plan to figure out if ECC is actually working?

With dmidecode.

The Kabini data sheet states that its integrated memory controller supports ECC and traditionally AMD motherboards have supported it if the memory controller does, so i see no reason it shouldn't work. I just wondered if anyone here had tested it out. Since no one has i'm prepared to be the Guinea-pig and report back.

Not everyone can afford Supermicros and over-priced Intel chips, as people using FreeNAS on old hardware can attest. If the Kabini platform proves capable of using ECC then it could be a way for those using ancient hardware to move to a more modern platform without taking out a second mortgage for the SM/Intel solution.

The motherboard and CPU together were £63. £63! I'm not expecting miracles from it, but if it proves viable then it represents amazing value for money. If it doesn't then i'm only out £63 and then only if i can't find another purpose for it (which is unlikely in my household).

Or maybe i'll just go ahead and run Linux MDADM + XFS on it. The data's non-critical (media files. A pain to redownload, but not devastating if lost), with backups elsewhere of the important stuff.

 

[cyberjock]

Now for the killer... dmidecode pulls memory locations for Intel chips. AMDs documentation is virtually nonexistent, so even if you pull dmidecode values, the 0s and 1s don't necessarily mean what you think they mean. In fact, dmidecode only works on certain Intel chipsets.

That little tidbit is why ericloewe said that. Your answer is a common misconception because the short reality is that you have NO way to prove AMD systems use ECC except to buy the AMD server boards from reputable companies. Ironically, they won't be cheap either. Even on Intel boards, only certain chipsets work with some of the tricks like dmidecode and ecc_check. So those overpriced boards do have advantages that you just can't beat at times.

So yeah, he said that because you have some problems to work out. You can either take the manufacturer at their word (which has been documented to be lies by more than 1 manufacturer before since they consider ECC RAM to "function" as ECC being used when its not), you can hope for the best, or you can instead go with hardware that is certain to work and can be tested with real-world applications.

To be honest, that linux with MDADM is probably looking pretty good for you. ;)

 

[AlainD]

Hi

dmidecode and other testers only do wat a decent BIOS would do. If the bios options for ECC are visible and enabled, it will give you as much information as those "testers". They don't test the ECC system itself.
Like I wrote testing an ECC-memory system is hard and I doubt that much ECC-ram tests are available. Unfortunally for testing, a RAM error is rare and thus expensive to test. Live systems can run for years before there are effected.
In the past most if not all companies at a certain point screwed up, so trust is not easy.

It's important for ZFS (read FreeNas), because it has no protection for memory errors itself and a 1 bit error can destroy you're complete pool. As I understand it even you're ZFS replications on other hardware. Lot's of meta data structures are kept in memory and are written back to disk if needed.

 

[Ericloewe]

I'm sorry, but saying that a Pentium or i3 is overpriced is pure FUD. A Pentium that's good enough for file sharing costs less than 50 bucks. Needless to say, it supports ECC.

The problem comes from the unreasonable expectations so many have. For my previous servers (WHS and WHS 2011), I did not use server-grade hardware. Yet, to name an example, the motherboard for my WHS 2011 (the ever popular P8Z68V-PRO) easily cost 200 bucks. Why so expensive? Because the cheap crap is just that - grade A crap. A 50 buck motherboard is universally bad (unless we're talking about a very basic Atom board) - any issues are simply masked away under Windows (and maybe Linux, if the manufacturer went through the trouble) in software through hacky drivers.

Basic server-grade hardware (the kind most of us run) is comparable in price with decent quality desktop stuff - the only things that necessarily carry a premium are ECC RAM (still smaller than it should be, theoretically) and i5/i7 class processors that support ECC (they call them Xeons).

Of course, once pricing turns into a race to the bottom, it's quite hard to compete on that factor alone. Nobody ever wins from races to the bottom, though.

 

[cyberjock]

Testing ECC is easy... there's a dmidecode command for that that will simulate a single-bit error. Oh, but it works on Intel chips only.

Sorry, you have NO clue what you are arguing about there AlainD. And at this point there is no point in me continuing this conversation. You clearly do not understand how all this stuff works.

Good luck to you!

 

[AlainD]

Testing ECC is easy... there's a dmidecode command for that that will simulate a single-bit error. Oh, but it works on Intel chips only.

Sorry, you have NO clue what you are arguing about there AlainD. And at this point there is no point in me continuing this conversation. You clearly do not understand how all this stuff works.

Good luck to you!

It's simulating, like you wrote, not testing the complete ECC chain. In the best case it's testing a part of the chain, but that cannot be verified.
It's clearly useful for testing the ECC reporting chain.

Unfortunately a problem in memory (whether it's ECC or not) can destroy a pool on ZFS, because ZFS does no in memory checks.

 

[AlainD]

...
The problem comes from the unreasonable expectations so many have. For my previous servers (WHS and WHS 2011), I did not use server-grade hardware. Yet, to name an example, the motherboard for my WHS 2011 (the ever popular P8Z68V-PRO) easily cost 200 bucks. Why so expensive? Because the cheap crap is just that - grade A crap. A 50 buck motherboard is universally bad (unless we're talking about a very basic Atom board) - any issues are simply masked away under Windows (and maybe Linux, if the manufacturer went through the trouble) in software through hacky drivers.

Basic server-grade hardware (the kind most of us run) is comparable in price with decent quality desktop stuff - the only things that necessarily carry a premium are ECC RAM (still smaller than it should be, theoretically) and i5/i7 class processors that support ECC (they call them Xeons).

Of course, once pricing turns into a race to the bottom, it's quite hard to compete on that factor alone. Nobody ever wins from races to the bottom, though.

When I last really checked desktop motherboards the price difference was almost complete by features and brand, unfortunately. It may have changed in the last two years.

 

[Pharfar]

My home system is an AsRock AM1B-ITX + Athlon 5350. If any1 has ECC ram in my vicinity (Vejle / Denmark), i'll gladly offer coffee while tests are run

 

[cyberjock]

lolz....

 

[rustyfork]

Hi everyone! Someone redirected me to this thread from a post I made earlier. Glad this discussion is taking place.

A few questions:

Does memtest86 use dmidecode when it does ECC error injection tests? What about this won't work for AMD ECC-checking verification and only works for Intel? Is it a matter of support or shoddy AMD implementation?

Also, some resources:

AM3+ Mobos Supporting ECC:
- GIGABYTE GA78LMT-USB3
- ASUS M5A78L-M/USB3

AM1 Mobos Supporting ECC
- ASUS AM1M-A

Maybe we should make a more exhaustive list? Anyone?

Also, if you want to do a test, I'd suggest testing the ASUS AM1I-A - the companion board to the verified AM1M-A - for ECC compatibility.

 

[cyberjock]

That's a question for the memtest guys. I know that 4 or 5 (the previous version to what they have right now) had a more robust mechanism for detecting ECC. The latest version is actually incorrectly reporting ECC RAM for my system as well as 2 friends with AMDs that we're 99% sure supports ECC.

The problem with a list is that it can say it supports it all day. "Supporting" it and "using" it are NOT the same thing to many manufacturers. That little detail is why AMDs turn into a mess. How do you actually validate that ECC is working? Many AMD boards (and Intel) will let you install ECC RAM and it *will* boot and it *will* function properly. But we know that in many cases for 100% certainty that the ECC feature is NOT used and cannot possibly function.

So I take the conservative stance that unless you can validate for yourself for darn near 100% certainty that ECC RAM function does work via dmidecode or ecc_check or something like that or the manufacturer is well know for server-grade products and a reputation for excellence then you should be asking yourself if you are truly building a system with ECC RAM supported and functioning.

 

[Middling]

Just thought i'd update this thread.

Received the case, PSU, motherboard and CPU on Thursday and put it all together. I can confirm that the Gigabyte GA-AM1M-S2H motherboard does NOT support ECC, at least in the F1 and (current) F2 BIOSes.

This is not such a big deal to me since i was contemplating a move away from FreeNAS/ZFS on my new server anyway, but if ECC is a deal breaker for anyone else then you should look elsewhere (there are a few suggestions that the Asus AM1 motherboards do support ECC).