Kabini - next step up from n40l microserver?

[rustyfork]

How do you actually validate that ECC is working?
...
So I take the conservative stance that unless you can validate for yourself for darn near 100% certainty that ECC RAM function does work via dmidecode or ecc_check or something like that...

I hate to sound redundant, but I was wondering if you could take a look at this page where some guys from Passmark run tests on ECC memory with an FX-8150:

http://www.passmark.com/forum/showthread.php?4470-MemTest86-ECC-RAM-error-reporting-status

Is there any reason to doubt the veracity of the ECC tests in this particular instance? If so, could you please explain a bit more as to why? Time and patience permitting, of course :)

 

[cyberjock]

Please read what you linked...

During 2013 for V5 we have been working to bring back support for reporting of ECC errors, for at least the popular current platforms, if not for some of the older platforms as well. It turns out that this is not a trivial exercise. Different code is required for different chipsets and the mechanisms for reporting errors in UEFI BIOS are poorly documented, with some of the documents not even being available to the public. (Note that V5 of MemTest86 will only support UEFI based hardware).

See the underline. David (the poster you linked) is saying the exact same thing I've been telling you in this thread. Intel does things different for different chips, it's documented, but hard as heck to actually identify it.

Also, if you read what he says two things that should be "a-ha" moments...

Since 2004 this code to read ECC errors was never maintained.

Hey! Maybe I shouldn't trust Memtest for ECC validation. Not surprisingly, I've thought this the whole time based on watching people on the forum. But as I don't read MemTest's code I can't exactly point you to the code. Even if I could, you and I could never actually validate it because of the whole chipsets, mechanism chagnes, and lack of good documentation. So we'd have some code and have no way whatsoever to validate if it works or not.

During 2013 for V5 we have been working to bring back support for reporting of ECC errors

This effectively validates that they don't do any kind of "ECC tests" like you claim. In fact, if you know how ECC works you'd realize that what you said is nonsensical. :P

All that Memtest (as well as other programs out there) do is report the errors that the hardware reports. The *whole* problem with validating ECC stems from two parts:

1. ECC is handled entirely in the hardware and is not designed to allow you to manipulate it in software.
2. The way in which you validate ECC is functional varies widely and is often totally undocumented.

It's literally a part of the hardware that does its magic behinds the scenes and if all is well you have no way to actually prove it in software across many platforms. On the flipside, if things are going wrong but not being reported (regardless of the reason for them not being reported such as ECC not working, broken ECC implementation, etc.) you are still left in the dark.

Now you get to ask yourself a question. Since Intel is the only one with any kind of documentation (despite the documentation being scant), are you willing to put FreeNAS on that AMD machine and trust that FreeNAS (and FreeBSD) will know about those ECC errors and report them to you appropriately? Sure, it can correct the single-bit errors as it goes, but don't you want some validation that you'll actually get an email from your FN box when errors are corrected? How are you going to do that?

See, the deeper down the rabbit hole you go, the more uncertainty there is. How much uncertainty are you wanting to risk? Even if you can prove for 100% certainty that it works in FreeNAS 9.2.1.x, what about FreeNAS 9.3? What about FreeNAS 10? What happens when someone choose to yank out that code that allowed your AMD board to work because it conflicted with too many other boards? Are you going to be aware of that?

If you were a master programmer you could probably find the code in Memtest and make sure that something equivalent exists in FreeBSD and then in FreeNAS. Then you could be relatively sure that things will be "ok" so long as a bug doesn't break that particular part of the code. But if you aren't that master programmer, you have to decide how much risk.

I can't (and nobody in this forum can) quantify how much risk it is. It might be 15% or it might be 90%. But, when you also throw in the fact that AMD hardware has typically had a hard time running FreeNAS along with this ECC problem and *all* the other threads out there with errors from AMD hardware that's never been seen on the hardware we recommend (which happens to be Intel too) which would you choose to endorse as a forum? To tell the world to go AMD despite all of these challenges or you spend a little more and go Intel because "it just works"? I'd feel pretty crappy if I was telling people to go AMD and all these people are dumping wads of cash into a box that won't even boot FreeNAS because of some weird unexplainable problem that only exists to AMD.

I choose to "stick to what works".

 

[rustyfork]

Thanks again for your insight, Cyberjock. I want you to know that I really do appreciate the time and effort you put into your responses and that I try not to ask questions arbitrarily. It's a lot to take in considering I knew next to nothing about the subject a few days ago.

What are your thoughts on this special stick of memory they used for the tests? Supposedly the button on the DIMM would produce an error/bitflip on a hardware level. If pressing the button results in the software detecting the ECC activity, would that not be an indication that the software works? If nothing else, at least in that particular instance?

 

[cyberjock]

What are your thoughts on this special stick of memory they used for the tests? Supposedly the button on the DIMM would produce an error/bitflip on a hardware level. If pressing the button results in the software detecting the ECC activity, would that not be an indication that the software works? If nothing else, at least in that particular instance?

My take? I want one! Badly! It could make for some interesting tests in FreeNAS. I'd love to have a bad stick of ECC RAM that has single bit errors but works properly after ECC has corrected the errors.

I agree it's an indication that the software works. But, it's more complex than that. If you read that whole thread you get to post 10 and if you take the modified memory stick and put it in any slot except slot 1 then no errors are reported. WHOOPS! LOL. That to me tells me that something is potentially "not quite right" with their ECC detection code. Yet another potential example of things "appearing" to work, but not actually working. That kind of scenario is exactly what scares me with AMD.

One thing I *really* wish is that Intel (or AMD) would have a definitive way to inject deliberate errors into RAM (both single-bit and multi-bit) so that as a user you could validate this for certain. Unfortunately I find this to be something that is very unlikely. The reason I say that is that the servers that Intel and AMD aim for are not DIY servers. They are the full fledged Supermicros, Dells, HPs, etc. Those had *better* support ECC fully and without problems. If they don't you can bet that if word got out that a family of, say, Dell computers didn't do ECC correctly Dell would be in for some serious lawsuits, refunds, etc. It could cripple Dell and people might never trust them again.

Despite everything else, when you look at AMD versus Intel in the server world, Intel is dominating. If I remember correctly Intel owns something like 96% of the server market (in their hay-day of 2005 and 2006 they peaked at almost 25%). That's really bad news for AMD. Then you look at stuff like the first chart at http://www.fool.com/investing/general/2013/12/30/can-amd-make-gains-against-intel-in-2014.aspx and see that AMD's R&D has decreased while Intel's is up about 75%. Not to mention Intel's R&D being like 9x that of AMD and you've got a bad place for AMD to be in. I know I've been accused of being an Intel fanboy. I don't really care. Accusations really don't bother me with crap like that. To me, all these deep facts that people ignore are very important and part of my decision making. (you don't even want to know how hard it is for me to decide what brand of vehicle to buy!)

Anyway, the whole argument between Intel and AMD (especially with regards to FreeNAS boxes) to me is pretty easily summed up with a very short statement:

Intel has a vested interest in and is capable of providing the R&D both currently and for the forseeable future to keep their hardware compatible with open source OSes like FreeBSD. AMD has none of those things.

So which would you use when you boil all this information down to that short statement? I'd never spend my money on AMD with that kind of night and day difference.

 

[rustyfork]

Very interesting, indeed. I'm surprised with all the cashflow in their R&D departments, they haven't come up with some kind of validation tool for ECC yet - especially considering that large corporations with mission-critical data rely on it. Of course, I'm sure they've got their bases covered with various other contingency strategies such as parallel servers, off-site backups, and cloud solutions. Here's hoping they make ECC sanity checks more viable for the rest of us mere mortals, though.

Thanks again, Cyberjock.

 

[AlainD]

...

Unfortunately I find this to be something that is very unlikely. The reason I say that is that the servers that Intel and AMD aim for are not DIY servers. They are the full fledged Supermicros, Dells, HPs, etc. Those had *better* support ECC fully and without problems. If they don't you can bet that if word got out that a family of, say, Dell computers didn't do ECC correctly Dell would be in for some serious lawsuits, refunds, etc. It could cripple Dell and people might never trust them again.
...

Something to think about when DIYing a server for critical stuff. (I don't like the situation, but there's not much to do about.)

 

[Middling]

Made an interesting discovery today that's completely changed my plans.

It seems the i3 CPU in my desktop supports ECC but the motherboard doesn't. Conversely the motherboard supports VT-d and the CPU doesn't.

So my new plan is to pick up a Supermicro board and put the i3 in, and get a VT-d capable i5 for my desktop board. Stick them both in 4u cases and use the Supermicro for storage and the ex-desktop system for virtualisation.

After a game of musical chairs with the motherboards and cases the Kabini system will end up as a backup desktop incase of problems with my main virtualised desktop (which i'm hoping will be Mac OS X with VGA passthrough).

 

[messerchmidt]

did anyone test the Asus AM1M-A with ecc?

http://www.overclock.net/t/1495837/ecc-works-on-am1

i wish amd would release the actual opteron x2150 as that one defiantly has ecc support

 

[eire1274]

Late response, but the AM1M-A does indeed support ECC unbuffered. For the record, I am using 2 Corsair CT102472BA160B (8GB DDR3 PC3-12800) with an Athlon 5350.

Running fine as a multilane file server, FTP & HTTP webserver, and Plex Media server plugin.

 

[Ericloewe]

Late response, but the AM1M-A does indeed support ECC unbuffered. For the record, I am using 2 Corsair CT102472BA160B (8GB DDR3 PC3-12800) with an Athlon 5350.

Running fine as a multilane file server, FTP & HTTP webserver, and Plex Media server plugin.

You mean Crucial ;)

I had a "Wha?" moment when I read that: "But Corsair stopped making ECC RAM in the DDR2 days!"

 

[eire1274]

Ugh, duh... sorry, trying to make the motherboard able to boot via GPT, so my brain is scattered. Yes, you are correct, Crucial. I have a Corsair logo staring me in the face and I typed it by reflex.

Thanks!

 

[eire1274]

Well, while I am still running 9.2.1.9 slick as snot on my Kabini 5350, I am not getting good news from either iX Systems or Asus.

An iX admin was so bold as to say that they are developing away from AMD entirely (which makes no sense when you are building a product on an open operating system [FreeBSD loves AMD hardware]) which makes no sense, as that's still 20% of the PC base with intent to start increasing again in 2015, thanks to AMD having taken over game consoles and renewing their public interest. I'm not an AMD fanboy by any means, but when it comes down to it, AMD still makes more bang for the buck, and as I said on the other post, I'd rather cheap-out on the processor and focus myself on better quality drives and memory, which is what I did with this last upgrade. If this was a more server-active operating system, this design requirement might be better explained, but the fact is I have yet to push my Kabini over 25% CPU usage, so I see zero benefit in buying more expensive Intel hardware.

In the old days, we were given information before releases that prepared us for these changes, but GPT issues still aren't listed on the 9.3 FAQ.

Asus, on the other hand, has yet to respond to my demand for an explanation for the removal of GPT compliant booting from the AM1M-A. The simple fact is that GUID Partition Tables (GPT) are a standard of EFI, and to boast that this motherboard is UEFI compliant and Windows 8.x compliant, while trimming this feature set out, is an absolute outrage.

I'm not a hardcore system builder anymore, nor am I really interested at all in programming, which is why I have put my heart and soul into FreeNAS since 0.5 days, when it was a flaky system, but when you got it working right, it just ran, even on ancient hardware. Now, you're expected to be running a Xeon with 32Gb of memory it seems. I'm thinking that this "free" operating system has lived it's life, unless something extraordinary happens soon.

 

[cyberjock]

You need to understand the situation a little more. Call me an Intel shill if you want, but here's some cold hard facts...

1. AMD is not overly compatible with FreeBSD. It's never been a great match with FreeBSD at all from what I've heard. Yes, it has worked. It has never been a flawless experience across AMD's product line because AMD doesn't have the revenue and income to pay devs to write code to make FreeBSD work on AMD builds. Just to compare, Intel's monthly R&D budget exceeds AMD's yearly budget by 5 fold or so.
2. Back when AMD was in its hayday (2004-2007 or so) AMD used to participate more in FreeBSD. AMD had paid developers (just like Intel has paid developers). Since AMD has basically slowed their support of FreeBSD (and all server-side stuff) naturally you can expect that over time FreeBSD will be less and less compatible with AMD systems.
3. Intel sells their hardware as a platform. 99% of the decisions are made by Intel, so Intel provides a pretty common experience on a given CPU. AMD doesn't provide the same kind of integration. AMD systems are usually an AMD CPU with one of a dozen or more chipsets that have been designed to work on AMD. Those chipsets are made by companies that do what they do while cutting costs everywhere they can. Even Intel based systems with VIA chipsets are a horrible choice for AMD, so this isn't "just an AMD" problem. For Intel though, non-Intel chipset based systems are in the extreme minority and mostly are the lowest end purpose-build hardware. AMD's entire product line is mostly things that are similar to VIA chipsets.
4. Intel provides most of the engineering specifications AND implements them via the BIOS. AMD's systems, by comparison are a situation where AMD provides options and the motherboard designer choose what they want to use or not. If you want a more feature-packed and more compatible system, you're going to have to spend more money. PERIOD. Of course, people buying AMD aren't looking to spend more money, so they are often left in the cold.
5. I don't know where you get your stats for AMD usage in servers, but they've NEVER been a "significant market share". Check this out... http://www.businessweek.com/articles/2014-11-06/amd-falls-further-behind-chip-rivals-intel-qualcomm AMD had yet another round of layoffs just 60 days ago and fiancially they are VERY bad off. They may not exist by this time next year. Can you really be upset that support is poor for a company that *need* to provide support for their hardware. Unlike what most people thing, x86 is not x86. CPUs are far to complex to simplify things down to that. Having thorough engineering teams, coders that provide code, and proper implementation of features from inception to execution is vital. None of those 3 things are in AMDs favor, not even slightly.

Quite literally, when you buy AMD you are doing nothing more than supporting the underdog. Unfortunately the underdog is so sick right now you
shouldn't expect them to support you back. If this is unpalatable I'm sorry. That's the harsh reality. There's a reason why I've been labeled an Intel shill. The objective evidence suggests that AMD is an extremely poor choice for an obscure OS like FreeBSD. On Windows AMD might fair better. But this forum doesn't deal with Windows except on the client side, so that argument is totally invalid for the scope of this discussion.

So you can say that it sounds like iXsystems is developing away from AMD. It's more appropriate to say that AMD isn't really designed for FreeBSD. I've been saying this for more than 2 years. FreeBSD is basically continuing development, completely exclusive of AMD since AMD isn't participating. You could say that FreeBSD is outgrowing AMD and AMD doesn't have the friendly relationship with FreeBSD they used to.... mostly because of costs. Nobody would argue that FreeBSD should just stop innovating since AMD doesn't want to play. They just aren't enough of a marketshare to even consider making that argument.

At the end of the day, argue all you want. AMD has not historically been and likely will never been a major player in FreeBSD development and support. Don't like that reality either don't use FreeBSD or don't use AMD. I'll let you decide which you want. But trying to argue that AMD has *ever* had a great relationship with FreeBSD is a laugh and a half for the rest of us. It's not even remotely true.

When I buy a desktop I care about things like chipset specs, on-board components (both type and quality), and actual hardware quality (such as manufacturing process). When buying server-grade, you are looking for a totally different set of variables. Unless you've been building servers for years, you probably aren't overly privvy to this info and it seems very non-intuitive. My first FreeNAS box (built solely for testing and never had real data) was only using recommended hardware because I happened to have an X58 based system collecting dust. It's circa 2008 and boots FreeNAS 9.3 without fail.

Now, changing subjects a little...

gpt issues aren't in the FAQ because the bottom line is that gpt support is very much a BIOS thing (as you've pointed out). So if it doesn't work, it doesn't work. There's not much option. I will say that Intel systems that meet our general recommended hardware work just fine with gpt. Why? Because that's well supported and "just works" as designed and documented. If it's not working for you then you're generally looking at something specific to your motherboard. There's just no way we can have support for all motherboards out there, nor are we interested in it. I know that the recommended hardware thread that I wrote and is maintained works just fine. If yours doesn't work then all I can really recommend is you stay on the more beaten path. That, by definition, means Supermicro stuff on an Intel-based system. If you were a FreeBSD wizard you might know how to troubleshoot the problem. But as your hardware isn't working and virtually nobody here even has your hardware, we don't really want to spend stupendous amounts of time on hardware that really isn't a particularly great choice, even if we *could* get it to work. When 9.3 was in beta testing quite a few AMD systems were found to not work. The answer was "talk to your motherboard manufacturer to troubleshoot the issue". Now a company that is selling their systems at cutthroat prices isn't going to be ready to throw tons of time and resources at a problem experienced by a small minority. In comparison if the X10 series (Intel-based) suddenly didn't work with FreeBSD 9.3/FreenAS 9.3, Supermicro (and Intel) would be VERY interested and willing to spend resources to troubleshoot the issue. AMD (and companies like MSI and such) don't make lots of money from your one board and are not overly motivated due to market forces to dig into troubleshooting the problem, nor fixing it. All that being said, feel free to troubleshoot your problem on your own. FreeBSD/FreeNAS is open source and code that makes your hardware work is gladly accepted. Being that you aren't a developer and may not have the experience to troubleshoot the problem, you're left with only one choice.. buy what you KNOW will work. That means AMD should be avoided.

Lastly, you need to understand that FreeNAS prior to 8.o was developed by a different group of people, had different engineering considerations, and is not related at all to the pre-8.0 except by name. PERIOD. So your argument about the FreeNAS working previously on "ancient hardware" isn't valid. The reason is that we do not and haven't targeted that market in more than 3 years. If you are yearning for the "old" FreeNAS that was designed for ancient hardware you should use NAS4Free. That is the true renamed project that continues fro the pre-8.0 code.

In conclusion (and I apologize for the wall of text, trying to explain things to you) I think your expectations of what FreeNAS is compared to what it was 4 years ago is outdated and the development of FreeBSD to work with AMD hardware is in AMD's ballpark and not FreeBSD. FreeBSD gladly excepts code from anyone. AMD just isn't a major player and not overly involved with FreeBSD. Even some of my friends that use linux heavily have said that AMD's support for linux has seriously waned in recent years, and there's serious concerns that linux may not be AMD friendly in the future.

At this point you should probably stop and ask yourself if AMD really is the way to go considering all of this information. I've never recommended them for servers in my 3 years here and right now I don't expect that to change. AMD is going to way of ARM for their future, so I expect AMD support for server OSes (and especially FreeBSD/FreeNAS) to die a slow painful death while people hold onto the past where AMD may have worked (for various definitions of "worked"). Just like 9.3 has been somewhat hostile to AMD because AMD hasn't added to FreeBSD development, I expect FreeNAS 10.x to be even more hostile. Unless AMD pulls it around (not likely considering the markets) I expect that soon no AMD hardware will work with the, then-current versions of FreeBSD in the future.

 

[eire1274]

Lastly, you need to understand that FreeNAS prior to 8.o was developed by a different group of people, had different engineering considerations, and is not related at all to the pre-8.0 except by name. PERIOD. So your argument about the FreeNAS working previously on "ancient hardware" isn't valid. The reason is that we do not and haven't targeted that market in more than 3 years. If you are yearning for the "old" FreeNAS that was designed for ancient hardware you should use NAS4Free. That is the true renamed project that continues fro the pre-8.0 code.

I think this is exactly my point. FreeNAS, pre iX Systems, was a very different creature. With the takeover, it would have been worthwhile to have let us know that the purpose of FreeNAS had altered and the specs were expected to run into the true server-grade realm.

The only thing that I can say is that FreeBSD still documents compatibility with AMD hardware, and for an IT engineer who is now disabled (that's me), if AMD is cheaper and still capable to do the work I need it to do, compared to more expensive options, well, then, you see my motivation. If I only have $1000 left over after my children, the house, utilities, well, you had damn sure realize that things are not going to end up with a Supermicro running a Xeon.

As time goes on, I'm seeing more and more resistance, and it seems like my (new) hardware set is a thing of the past for FreeNAS, well, it's probably time for me to look at competing packages that do support my hardware. I am NOT EXPRESSING ANTI-FREENAS SENTIMENT HERE, as you still produce a fantastic operating system that I will continue to recommend, but now with a certain caveat for server-grade components.

Oh, and I'm going to say it again: 9.3 runs FINE on Kabini hardware, once you bypass the bad UEFI. As much as you want to deny it, it's quite happy on my hardware.

 

[cyberjock]

I think this is exactly my point. FreeNAS, pre iX Systems, was a very different creature. With the takeover, it would have been worthwhile to have let us know that the purpose of FreeNAS had altered and the specs were expected to run into the true server-grade realm.

I'm not sure why you are saying this... it's never been a secret. Even if you look at our recommended hardware thread from 2013 https://forums.freenas.org/index.php?threads/so-you-want-some-hardware-suggestions.12276/ it's pretty clear you need server-grade. I'd find the old one but I can't even find it anymore. But even when I started using FreeNAS in Feb/March 2012 (when I opened my account on this forum) it was very clear that server-grade was required. Which is why I had server-grade parts 4-5 months later. I think it's more correct to say that YOU haven't done the research to know this and are simply denying the information exists when it clearly does.

The only thing that I can say is that FreeBSD still documents compatibility with AMD hardware, and for an IT engineer who is now disabled (that's me), if AMD is cheaper and still capable to do the work I need it to do, compared to more expensive options, well, then, you see my motivation. If I only have $1000 left over after my children, the house, utilities, well, you had damn sure realize that things are not going to end up with a Supermicro running a Xeon.

I understand your motivation. I'm a disabled veteran myself and very privileged to also be able to work from home, so I do have a good source of income. But when I bought my server parts that I mentioned above, I had no job and I had no prospect of a job. You have to prioritize where your money goes. You've made your choices, sir. For better or worse. I made my choice to invest in truly supported hardware and have had great success and no problems.

As time goes on, I'm seeing more and more resistance, and it seems like my (new) hardware set is a thing of the past for FreeNAS, well, it's probably time for me to look at competing packages that do support my hardware. I am NOT EXPRESSING ANTI-FREENAS SENTIMENT HERE, as you still produce a fantastic operating system that I will continue to recommend, but now with a certain caveat for server-grade components.

To be honest, that caveat has existed since before 2012. So you are arguing something that was long dismissed years ago. Not sure you have a leg to stand on by arguing points that are more than 3 years ago.

Oh, and I'm going to say it again: 9.3 runs FINE on Kabini hardware, once you bypass the bad UEFI. As much as you want to deny it, it's quite happy on my hardware.

Now you're just being silly. Did you even read what I wrote? I didn't say it wouldn't work, and I've never denied that it *can* work.

You need to understand the situation a little more. Call me an Intel shill if you want, but here's some cold hard facts...

1. AMD is not overly compatible with FreeBSD. It's never been a great match with FreeBSD at all from what I've heard. Yes, it has worked.

So either you chose to not read what I wrote, or you're just trying to start a fight. In either case, I'm going to lock this thread as there is nothing more to be said and I'm not going to wait for 10 other people to come in here and start yet another flamewar.

The facts stand for themselves.

Good luck to you.