ECC-enabled build recommendation

[petr]

Hi,

at this moment, I am running an Intel i5 build 12GB of RAM and 6HDDs providing 16TB of combined storage. All is working well and I've just upgraded to the 9.1 release. However, reading through the forums, I am starting to get a bit worried about not having ECC ram installed in the system.

So, as I've got almost everything up and running, what combination of ECC-enabled motherboard+CPU+RAM would you recommend to get as safe as I can possibly be? Dual ethernet would be a plus and performance-wise I would like to opt for at least quad-core CPU. Power consumption is not much of an issue.

What would be the current state-of-the art recommendation? I assume that usb3/thunderbolt are not extras worth paying for..

 

[cyberjock]

Here's what I have...


32GB ECC RAM
Supermicro X9SCM-F motherboard(with IPMI and dual Intel NIC)
e3-1230v2 Intel Xeon CPU.

It's like this. If you go with non-ECC and your RAM has problems, you are pretty much completely screwed. Only 1 person was able to save any data that I can think of after identifying bad RAM. But, how often have you had RAM fail? I've had like 8 sticks in about 20 years of dealing with hardware. And I'm not talking "i think its bad so I'm gonna RMA it"(which seems to be a pretty rampant problem). I'm talking doesn't work in any computer at all or flat out failed RAM tests.

If you want to gamble, stick with non-ECC. Of course, if your RAM goes bad it will trash your original copy thereby also trashing any backups made after that point in time.

So weigh the potential consequences with the chances of failure and make your own call. I'll never build a server without ECC again though.

 

[petr]

Thank you very much! I am happy to invest into my data integrity. At this very moment, by backup strategy is to rsync the critical data to simple linux based NAS but I feel that if something goes wrong with the RAM, then the error may propagate to the backups. The board you are suggesting looks promising, I think

And to answer RAM failure question - I've had some trouble a while ago when dual-channel memory started to appear and some no-name memory sticks claiming to be dual-channel capable simply did not work its way through the testing but nothing recently. However, question is - how would I know? Maybe the occasional kernel panics I've seen in my day were indeed due to bad RAM sticks.

So I think I will go for the ECC option.

By the way, I've never used IPMI - what use-case do you have for it?

 

[cyberjock]

Oh, the errors will absolutely propagate to the backups. In fact, one person got very lucky. He setup a FreeNAS server, did an rsync of a 20GB file from one server to another, and happened to do a sha256 sum of the file on both ends. They didn't match. He was confused because rsync was supposed to prevent corruption so he posted to the forum. It turns out that the new server he had just built had bad RAM and it was causing rsync to silently allow corruption. :P

 

[cyberjock]

IPMI is the shiznit. Its like doing remote desktop to your server, but you can do poweron/poweroff, access the BIOS, boot from ISOs on remote machines without a cd-rom on the actual server. It's just amazing.

 

[petr]

That does indeed sound pretty cool! I guess I will be placing an order very soon :)

 

[Whattteva]

I can't access the BIOS or boot from ISO's, but I can poweron/poweroff without the IPMI though. All you need is for the network card and BIOS to support WOL, which most somewhat recent computers should easily support.

 

[cyberjock]

I can't access the BIOS or boot from ISO's, but I can poweron/poweroff without the IPMI though. All you need is for the network card and BIOS to support WOL, which most somewhat recent computers should easily support.

You also need for the NIC driver to support WOL. That's been a problem for many NICs in FreeBSD.

I'm not sure which ones support it and have been tested to actually work since most users aren't powering off their server regularly(if you are you probably shouldn't be since it seems to wear out the hard drives faster than letting them run).

 

[tonyr]

WOL does not support power off. If your system needs a power cycle and you cannot log in over the network, you cannot power cycle. Also most WOL works on the connected subnet and does not work over the internet atleast not without some help.

 

[Whattteva]

Uh yeah... why would you want WOL to EVER work over the internet, lol. That's what VPN or SSH is for.

I generally only use WOL to wake it up. Once it's on, then I can log on to it using other normal means (SSH, remote desktop, etc.) and then power it off using that same method.
WOL is more than enough for my use case, at least.

 

[tonyr]

1) VPN is only useful in specific cases

2) SSH is only helpful if Unix is up and responsive

It is very useful if you have a colo or rented server.

 

[petr]

One related thought crossed my mind today. I was going through iXsystem's offering - FreeNAS mini and I've noticed that the RAM is not ECC-enabled.... odd from the system creators?

 

[pbucher]

One related thought crossed my mind today. I was going through iXsystem's offering - FreeNAS mini and I've noticed that the RAM is not ECC-enabled.... odd from the system creators?

That's because ECC isn't a drop dead requirement. Yes bad ram can nuke your files, file system, & your life....but you can responsibility expect quality non-ECC ram that is burned in & tested to be the least of your data integrity problems. Statistically hard drive blocks going bad will be the death of your data. I've got like 10 bad hard drives for every bad memory stick I've had.

The thing is RAM rarely goes bad in time(unless you have temperature issues and you bake your RAM) after an initial use period(hence the reason to burn it in) but hard drives tend to work initially and then go bad with age. I had 70% of the drives fail in a enterprise disk array over the course of 3 years(but it worked great the first 8 months).

 

[survive]

Hi petr,

The mini doesn't offer ECC because there aren't many (any?) boards that have ECC in a form factor that would fit in the case. That's just one of several compromises you need to make if you want an itty-bitty NAS box.

-Will

 

[Yatti420]

That's because ECC isn't a drop dead requirement. Yes bad ram can nuke your files, file system, & your life....but you can responsibility expect quality non-ECC ram that is burned in & tested to be the least of your data integrity problems. Statistically hard drive blocks going bad will be the death of your data. I've got like 10 bad hard drives for every bad memory stick I've had.

The thing is RAM rarely goes bad in time(unless you have temperature issues and you bake your RAM) after an initial use period(hence the reason to burn it in) but hard drives tend to work initially and then go bad with age. I had 70% of the drives fail in a enterprise disk array over the course of 3 years(but it worked great the first 8 months).​

​

I always stress/burn my ram and hdds before putting them into production and while I agree ECC isn't drop dead.. It's certainly a nice addition with ZFS for absolute protection against spontaneous corruption.. It's not like an ECC stick can't go bad either..​

I assume ECC sticks however wont pass along garbage to the pools in the case of corruption or failure.. That alone is worth the extra money.. ​

 

[cyberjock]

From what I've seen if ecc can't correct ram errors it usually halts the system. That protects the pool from damage.

I've never had a bad stick of ram out of the box, but I've had ram fail suddenly after months or years of use. In every case for me we hadn't done any server maintenance that might contribute to the failures. /shrug

 

[Mr_N]

whats the easiest way to stress test both ram and the created zpool (hdd's) in a freenas server before copying data to it?

 

[pbucher]

whats the easiest way to stress test both ram and the created zpool (hdd's) in a freenas server before copying data to it?

Use Memtest86 and let it run for a few days. Google it and read up on it. Asimple zpool test is to run out a good # of GBs using the dd command.

 

[Yatti420]

While I'm not against testing within FreeNAS I'd rather run my drives outside of FreeNAS (before I even begin to use them). I usually run HDAT2 or something similiar for a few days to avoid any early death scenarios and verify for sure the drive read/writes correctly..

Edit: Warning - HDAT2 is an advanced tool that can destroy your data.. Be-careful what you select..

 

[Mr_N]

the HDAT2 looks ok and wont need to worry about destroying data as there wont be any on the pool when testing, but it says many different hdd tests, yet i cant find any info on what they are or how to use them :P