ECC requirements for a "multitier" system

[beyondevil]

I've been tasked to help a non-profit set up a backup system.

After some discussions, the decision was made that we would build our own system and use FreeNAS (instead of buying something ready-made like Qnap).

Being a non-profit, money is a concern. With that said, they understand that spending some money up front can save the some headaches down the road.

The data that will be stored is mostly going to be content for their website (pictures and texts, video will be hosted on Vimeo). Initially the size needed is roughly 1TB and the 10 year estimate is around 8TB. So we're not talking large quantities of data here. The pictures and texts will mostly be user added (scanned copies of real photographs etc.). So somewhat sensitive, but not impossible to restore in case of corruption.

So the backup solution we've discussed is basically two NAS:es (master and slave, more on that later) with an additional cold storage (probably Backblaze B2). The two NAS:es will not be co-located.

None of the NAS:es will serve the content to the website, that's handled by a separate file server. So what I'm building is a pure backup system. The only time we'll ready from any backup is if the data on the file server would be corrupted in some way.

So my question is, given the details above, is ECC memory even required at all?

If yes, would it suffice to have ECC memory on the master NAS? Since syncing won't be two way. The master NAS will sync one-way to the slave and to the cold storage.

 

[Holt Andrei Tiberiu]

Ecc is a must. If founds are low, go with some used hardware from ebay or other. Like a dell r510 8 to 12 bay model. But wach the specs, to have de h200 controller, not the 6i, 6i is limited to 2 tb hdd.
I recomend a 5 x 3tb hdd setup in raid z2
32 gb ram.

 

[kdragon75]

I would honestly buy a lowend qnap or similar with large mirrored drives. If FreeNAS is only being used for backup, there's no reason to build a system. Unless you using low end second hand parts (Pentium class). Perhaps if you outlined the system you are contemplating we can make some suggestions? In general ECC does not cost much more if you plan carefully. I mean like $50... The amount you would spend on power with an older 1366 based system. Also 10 years is a rather long life span. You will need to budget as lease a 50% drive replacement cost.

 

[danb35]

is ECC memory even required at all?

ECC is never required; it's just strongly recommended for any system where data integrity is important, regardless of the filesystem--there's nothing about ZFS or FreeNAS that makes ECC any more important than it is with any other OS or filesystem. So the question is, how much does data integrity matter to you/your organization?

Other than used gear (which can be a great bargain, but often isn't very energy-efficient), the most cost-effective way to get suitable hardware is to catch a sale on a small purpose-built server like the Dell Poweredge T30/Lenovo TS140/HPE Proliant ML10. The T30 was selling for $300 with 8 GB (ECC) RAM and a Xeon CPU on Cyber Monday, for example.

 

[Holt Andrei Tiberiu]

ECC is never required;

Well, if scrub start's and the memory gives some error, won't the data on the drive's/pool's be corrupted?
That is why i would stick to ECC memory.

 

[danb35]

Well, if scrub start's and the memory gives some error, won't the data on the drive's/pool's be corrupted?

No, it won't*. See below for a more detailed explanation:
http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

* OK, yes, it can, but the necessary confluence of events to make this happen would be astronomically unlikely.

 

[Chris Moore]

Well, if scrub start's and the memory gives some error, won't the data on the drive's/pool's be corrupted?
That is why i would stick to ECC memory.

This is a concept called the 'scrub of death' and it is a theoretical that was discussed. Better safe than sorry.

 

[Holt Andrei Tiberiu]

I first started with a AMD Phenom x4 system with 16 gb ddr3 ram and 4 2TB drives in 2 x mirror which i had laying around in the office.
After a few month's, when i started to see how well freenas was working, i started to worry for may data, and quickly switched one of my hyper-v server's into freenas ( HP DL 160 G6 ), and never looked back.

 

[Chris Moore]

I've been tasked to help a non-profit set up a backup system.

Just because it is a "Non Profit" does not mean they don't have money. They may not want to spend it, but there needs to be a budget of some sort. How much storage are they looking for and what is the amount they are willing to spend? You need a starting point, not just the cheapest thing that will do the job for the moment and they need to enter this with a plan to upgrade / replace the system on some kind of schedule. I have seen too many places that buy it and forget about it until it 'suddenly' stops working seven years later but they never did any preventive maintenance between. Not even monitoring the system for error messages.
It is not a furnace that can sit in the back room and be ignored.

 

[Chris Moore]

So my question is, given the details above, is ECC memory even required at all?

If yes, would it suffice to have ECC memory on the master NAS? Since syncing won't be two way. The master NAS will sync one-way to the slave and to the cold storage.

For reliable function of the operating system of a server, regardless of the function the server is fulfilling, it is always a best practice to have ECC memory. We even put ECC memory in our workstations. If you need it to work reliably, all day, every day, ECC is the way to go. Systems without ECC memory are much more likely to have random crashes that are never explained. Random crashes are always bad and extra bad for a server.

 

[Holt Andrei Tiberiu]

Yapp, Chris is wright, I've seen the same.
If you cannot agree on a budget for hardware for freenas, better of with some qnap with 4 x 10tb drives.
When the software on those will give an error or not start ( as I have seen, qnap, iomega, WD and others ) or when you give a software update and data will magically disappear, then they will find the right funds for a decent setup.
Or you cold do this: explain that a decent entry model Netapp or DELL/EMC is arround 15.000 us dollars, and your setup is ~3000 us dollas.

 

[Chris Moore]

you give a software update and data will magically disappear,

We like to avoid that. If we can.

 

[Holt Andrei Tiberiu]

We like to avoid that. If we can.

I happened to me twice if, one time with some IOMEGA 4 bay and one time with a WD 2 bay model.
Never with FreeNAS. Even when both of my USB boot drives failed.
Fresh install on new boot drive, import config and all was upp and running.
Never lost my data even when my server had 2 memory dim's in error.

 

[microserf]

So my question is, given the details above, is ECC memory even required at all?

The job of any file system boils down to this: when asked to read a block, it should return the same data that was written to that block. If it can't do that -- because the disk is offline or the data has been damaged or tampered with -- it should detect this and return an error.
-- Jeff Bonwick

It will use ECC RAM or I'm not going to be responsible for it.
-- Anyone with a ZFS filesystem that has been in a rig with faulting RAM

 

[beyondevil]

Wow! Amazing responses everyone! Thank you, I really appreciate it. :)

Some quick answers to some of the responses/questions. (I won't waste my time nor yours by explaining decisions that have been made, unless explicitly asked to. Simply because it's probably not relevant.)

- Can't buy used/refurbished parts or systems.
- No QNAP (or other "single purpose" systems). Will build from parts.
- Budget is 10.000 SEK (approx. $1000) which is not particularly transferable since pricing sometimes differs significantly.
- I won't (at least not initially) use mirroring or raid. Master and slave will each have a 4TB WD Red disk for storage. OS will be run off of a SSD or USB stick.

In general ECC does not cost much more if you plan carefully. I mean like $50... The amount you would spend on power with an older 1366 based system.

Agreed. However, the problem, at least here, is that it severely limits the options available for mobo, cpu and ram. Especially in the lower end of pricing.

Also 10 years is a rather long life span. You will need to budget as lease a 50% drive replacement cost.

10 years was the estimate for storage requirements. Not the lifespan of the system.

So the question is, how much does data integrity matter to you/your organization?

All the data will initially come from hard copies. So in lieu of any natural disasters or fires or the like destroying the hard copies, it's more of an inconvenience should the data be corrupted.

Just because it is a "Non Profit" does not mean they don't have money. They may not want to spend it, but there needs to be a budget of some sort. How much storage are they looking for and what is the amount they are willing to spend? You need a starting point, not just the cheapest thing that will do the job for the moment and they need to enter this with a plan to upgrade / replace the system on some kind of schedule. I have seen too many places that buy it and forget about it until it 'suddenly' stops working seven years later but they never did any preventive maintenance between. Not even monitoring the system for error messages.
It is not a furnace that can sit in the back room and be ignored.

You missed the latter part of that paragraph: "With that said, they understand that spending some money up front can save them some headaches down the road."

Also, re. storage: "Initially the size needed is roughly 1TB and the 10 year estimate is around 8TB."

As for the starting point. This is one of the reasons why we're going with a custom built system (and not QNAP et. al.). And yes, we're starting with the cheapest (within reason) possible system. The reason being that the requirements and usage patterns are unclear at the moment. As they become more clear it'll be easier to configure something more "real" (if necessary).

Initially we're talking about maybe 2-3 users who will be uploading content to the master and it won't happen on a daily basis. So these systems will see very little usage. They'll be more or less cold storage (again, initially).

I'll be doing monitoring and maintenance after the systems are brought online.

So why FreeNAS? I need something with an intuitive GUI for the users - since they are not techies by a long shot (think grandma and grandpa).


OK, so first question answered: ECC - get it (got it, good).

But I don't think I got a clear answer to my second question:

Will the slave need ECC? Let me illustrate with an example.

Master (w/ ECC) has file A (which is in a good state). It now syncs that file to Slave (w/o ECC) and the file is for whatever reason corrupted.

Since we're only syncing one-way (Master -> Slave), next time the scheduled syncing happens - will Master see that file A is corrupt on Slave and rewrite/resync file A?

If so, there's really no need for ECC on the Slave, correct? Or is my assumption incorrect or am I missing something else?


Someone asked, so here's an example system: (Ignore the prices, this system would be approx. 600 USD for me in Sweden)

Case:
https://www.amazon.com/Fractal-Desi...35&sr=8-1&keywords=fractal+design+arc+mini+r2

Mobo:
https://www.amazon.com/Supermicro-M...=1546537857&sr=8-1&keywords=Supermicro+X11SSL

CPU (boxed)
https://www.amazon.com/Intel-Pentiu...546537905&sr=8-1&keywords=Intel+Pentium+G4560

DDR4 RAM ECC
https://www.amazon.com/Crucial-DDR4...8&qid=1546538035&sr=8-1&keywords=CT8G4RFD8266

Disk
https://www.amazon.com/Red-4TB-NAS-...TF8&qid=1546538077&sr=8-2&keywords=wd+red+4tb

PSU
https://www.amazon.com/Corsair-VS45...6538117&sr=8-2-fkmr1&keywords=crucial+450+psu

 

[Chris Moore]

Master and slave will each have a 4TB WD Red disk for storage.

You should have a mirrored pair of drives in at least one of these systems. If you don't, any data error is uncomfortable. ZFS stores checksum data, so it will be able to tell you that you have data corruption, but it will not be able to correct (as it normally would) because there is no parity data to draw the correction from. Disk redundancy is not just about protecting from disk failure, it is about guarding from data corruption.

However, the problem, at least here,

If you would share your location, other than, "From Planet Earth," it might be possible for our advice to take into account your location.

I'll be doing monitoring and maintenance after the systems are brought online.

This might help:

Building, Burn-In, and Testing your FreeNAS system
https://forums.freenas.org/index.php?resources/building-burn-in-and-testing-your-freenas-system.38/

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

Will the slave need ECC? Let me illustrate with an example.

I have a two FreeNAS system at home. Both use ECC for the same reason. Reliability. In my previous configuration, the primary had 32GB of RAM where the backup only had 16GB, because the backup system was doing significantly less. I had them configured for hourly, one way, updates.

Since we're only syncing one-way (Master -> Slave), next time the scheduled syncing happens - will Master see that file A is corrupt on Slave and rewrite/resync file A?

If you are using rsync, it will see that the two files are not the same and transfer the file again. This brings me back to the point of needing a mirrored pair of drives on the primary system so that it can check and correct the data on disk. Bits can flip on disk and ZFS will check for that, but it can only correct if there is redundancy.

If so, there's really no need for ECC on the Slave, correct? Or is my assumption incorrect or am I missing something else?

This does not mean that there is no need for ECC. ECC is always the correct answer.

DDR4 RAM ECC
https://www.amazon.com/Crucial-DDR4...8&qid=1546538035&sr=8-1&keywords=CT8G4RFD8266

No. This is Registered (RDIMM) ECC memory, you need Unbuffered (UDIMM) ECC memory.
This memory will not work with the system board you have selected.

 

[danb35]

If you would share your location, other than, "From Planet Earth," it might be possible for our advice to take into account your location.

The currency he uses would suggest Sweden, but as you say, we can only guess otherwise.

 

[microserf]

I'll be doing monitoring and maintenance after the systems are brought online.

So why FreeNAS? I need something with an intuitive GUI for the users - since they are not techies by a long shot (think grandma and grandpa).

What do you envision users doing in the FreeNAS GUI?

 

[Chris Moore]

What do you envision users doing in the FreeNAS GUI?

I was curious about that myself. For my part, I manage storage at my work and at my home and in neither of those environments would I want anyone other than me messing with the GUI. There are too many things that can be broken by making a setting change.

 

[SweetAndLow]

Wow! Amazing responses everyone! Thank you, I really appreciate it. :)

Some quick answers to some of the responses/questions. (I won't waste my time nor yours by explaining decisions that have been made, unless explicitly asked to. Simply because it's probably not relevant.)

- Can't buy used/refurbished parts or systems.
- No QNAP (or other "single purpose" systems). Will build from parts.
- Budget is 10.000 SEK (approx. $1000) which is not particularly transferable since pricing sometimes differs significantly.
- I won't (at least not initially) use mirroring or raid. Master and slave will each have a 4TB WD Red disk for storage. OS will be run off of a SSD or USB stick.



Agreed. However, the problem, at least here, is that it severely limits the options available for mobo, cpu and ram. Especially in the lower end of pricing.




10 years was the estimate for storage requirements. Not the lifespan of the system.




All the data will initially come from hard copies. So in lieu of any natural disasters or fires or the like destroying the hard copies, it's more of an inconvenience should the data be corrupted.




You missed the latter part of that paragraph: "With that said, they understand that spending some money up front can save them some headaches down the road."

Also, re. storage: "Initially the size needed is roughly 1TB and the 10 year estimate is around 8TB."

As for the starting point. This is one of the reasons why we're going with a custom built system (and not QNAP et. al.). And yes, we're starting with the cheapest (within reason) possible system. The reason being that the requirements and usage patterns are unclear at the moment. As they become more clear it'll be easier to configure something more "real" (if necessary).

Initially we're talking about maybe 2-3 users who will be uploading content to the master and it won't happen on a daily basis. So these systems will see very little usage. They'll be more or less cold storage (again, initially).

I'll be doing monitoring and maintenance after the systems are brought online.

So why FreeNAS? I need something with an intuitive GUI for the users - since they are not techies by a long shot (think grandma and grandpa).


OK, so first question answered: ECC - get it (got it, good).

But I don't think I got a clear answer to my second question:

Will the slave need ECC? Let me illustrate with an example.

Master (w/ ECC) has file A (which is in a good state). It now syncs that file to Slave (w/o ECC) and the file is for whatever reason corrupted.

Since we're only syncing one-way (Master -> Slave), next time the scheduled syncing happens - will Master see that file A is corrupt on Slave and rewrite/resync file A?

If so, there's really no need for ECC on the Slave, correct? Or is my assumption incorrect or am I missing something else?


Someone asked, so here's an example system: (Ignore the prices, this system would be approx. 600 USD for me in Sweden)

Case:
https://www.amazon.com/Fractal-Desi...35&sr=8-1&keywords=fractal+design+arc+mini+r2

Mobo:
https://www.amazon.com/Supermicro-M...=1546537857&sr=8-1&keywords=Supermicro+X11SSL

CPU (boxed)
https://www.amazon.com/Intel-Pentiu...546537905&sr=8-1&keywords=Intel+Pentium+G4560

DDR4 RAM ECC
https://www.amazon.com/Crucial-DDR4...8&qid=1546538035&sr=8-1&keywords=CT8G4RFD8266

Disk
https://www.amazon.com/Red-4TB-NAS-...TF8&qid=1546538077&sr=8-2&keywords=wd+red+4tb

PSU
https://www.amazon.com/Corsair-VS45...6538117&sr=8-2-fkmr1&keywords=crucial+450+psu

You're going to have a much better system if you buy one of those Lenovo, hp or Dell for like $150 USD. Much cheaper, has ecc and perfect for a simple Nas solution.