New storage server build - will it FreeNAS?

Joined
Oct 18, 2018
Messages
969
But then, why bother with ZFS, sync writes, and so on? Just start running massive RAID-0 arrays.
Am I wrong that non-ECC mirrors or RAIDZ1|2|3 is safer than non-ECC RAID-0? It seems to me like you choose ZFS w/Mirrored or RAIDZ vdevs to provide protection against corruption arising from storage media issues and use ECC memory to provide protection against corruption as a result of bad memory.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Yes, it is a matter of degrees. But if you want to rely on the data staying good, it has to be ECC memory, sync writes, and so on. That to me is the main reason to go to ZFS in the first place. Writing bad data with perfect fidelity is no better than writing good data to an imperfect storage device. Both result in long term data loss because chances are that you won't detect the issue before the backups have been over-written with poisoned data.
 

Bozon

Contributor
Joined
Dec 5, 2018
Messages
154
Definitely agree that memtest is important in a build without ECC memory, or really any build. Here is a nice study attempting to understand the actual affects of memory corruption and ZFS. An important takeaway here



You can take precautions if you choose to go without ECC memory. If your backup scheme involves running memtest on some interval (say every 3 months) and you keep backups until the next successful memtest run you'll have reason to be confident that your recent backups were not corrupted from bad memory. If a memtest run fails you can be confident that the backup from just prior to your last successful memtest run is safe. From there it is up to you how you choose to inspect new and modified files to confirm integrity. Of course, those that are corrupted may remain so if you don't have a non-corrupt backup. This scheme is not as good as ECC but is better than nothing, I think.

This reminds me of an article I read, about incorporating memory hardiness in ZFS. I can't remember the article now, but it seemed to be from an authoritative source (I am thinking it was one of the original ZFS designers.) The article mentioned it had been planned for a future version of ZFS (or it was basically incorporated into ZFS, but wasn't enabled), and it was a matter of flipping a flag. With this flag, ZFS would also double check memory data structures to make sure that they were still valid. My curiosity, is that I wonder if it is time to start adding this to the list of important things to implement, and test. It was probably originally left out because of a performance, or memory limitation issue, which may not exist for all users of ZFS now.

Anyone know anything more about this? I'll try to find the article when I have some time on my hands.

Memtest, can't really take the place of ECC because ECC is happening 24/7 in real time for every memory access.
Of course, ECC, is best at correcting 1 bit spurious errors or flagging 2 bit or more spurious errors, but even that isn't perfect because a 2 bit or more error could look like a valid memory state, if particular bits are flipped. Of course, the algorithm is optimized to make this case less likely, but of course, it can't make that 0%.

https://en.wikipedia.org/wiki/ECC_memory
 
Joined
Oct 18, 2018
Messages
969
Writing bad data with perfect fidelity is no better than writing good data to an imperfect storage device. Both result in long term data loss because chances are that you won't detect the issue before the backups have been over-written with poisoned data.
Absolutely agree, well-thought-out array architecture will not protect you from writing bad data in the first place. Data integrity it isn't black and white; it is more a matter of probabilities.

That to me is the main reason to go to ZFS in the first place.
I think this is spot on. System design is a personal choice everyone must make. I think a huge benefit of these forums is to provide FreeNAS users with the knowledge to understand the risks they take on with their given design so that they can develop a system that fits their risk tolerance, budget, data, etc.

Memtest, can't really take the place of ECC because ECC is happening 24/7 in real time for every memory access.
Totally agree. I meant my advice to be read only as if you choose to take the risk of ECC there are steps you can take to mitigate those risks. My personal choice is in line with @Constantin's, spend the few extra bucks and use ECC.
 

Bozon

Contributor
Joined
Dec 5, 2018
Messages
154
Am I wrong that non-ECC mirrors or RAIDZ1|2|3 is safer than non-ECC RAID-0? It seems to me like you choose ZFS w/Mirrored or RAIDZ vdevs to provide protection against corruption arising from storage media issues and use ECC memory to provide protection against corruption as a result of bad memory.

At some point, all data that is written to disk exists in memory. If it gets corrupted while in memory, it doesn't really matter how much redundancy you have on disk. In that sense, ZFS is fully GIGO compliant.
 

Bozon

Contributor
Joined
Dec 5, 2018
Messages
154
Absolutely agree, well-thought-out array architecture will not protect you from writing bad data in the first place. Data integrity it isn't black and white; it is more a matter of probabilities.


I think this is spot on. System design is a personal choice everyone must make. I think a huge benefit of these forums is to provide FreeNAS users with the knowledge to understand the risks they take on with their given design so that they can develop a system that fits their risk tolerance, budget, data, etc.


Totally agree. I meant my advice to be read only as if you choose to take the risk of ECC there are steps you can take to mitigate those risks. My personal choice is in line with @Constantin's, spend the few extra bucks and use ECC.

Yes, I just wanted to make that clear, because some people read what they want to read, just as some people hear what they want to hear. I wanted to make sure that someone didn't come away thinking: "Hey if I don't buy ECC ram, so and so, said I would be fine if I ran memtest x times a month, so to be really safe, I'll run it 2X times a month."
 
Joined
Oct 18, 2018
Messages
969
Yes, I just wanted to make that clear, because some people read what they want to read, just as some people hear what they want to hear. I wanted to make sure that someone didn't come away thinking: "Hey if I don't buy ECC ram, so and so, said I would be fine if I ran memtest x times a month, so to be really safe, I'll run it 2X times a month."
Totally agree there. It is good to make this very clear to anyone reading this in the future.

I will say though that I don't agree that just because someone uses non-ECC memory that they don't gain any benefits from disk parity or mirrors. All data written and backed up prior to the memory module going bad are not corrupted due to bad memory. It is a complicated issue as I think this conversation rightly reflects. Hopefully anyone reading this in future will see that complexity and will think very hard about their design choices especially as ECC memory and mirrors/parity are involved.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Also consider how a random bit flip due to a cosmic ray or whatever may not be persistent enough to be caught by memtest. Yet, the file might still get corrupted. I'm not saying this happens all the time (it doesn't) but like Bozon, I prefer a built-in watchdog that looks for bit flips as opposed to running the system on a open-loop basis. But that's the value I place on my data, others may not feel as strongly about it.
 

Mattias Hedman

Dabbler
Joined
Sep 5, 2016
Messages
21
I have been in your place -joined this forum almost a year ago and I have been going back and forth on whether I want to implement FreeNAS or just go with something else and "keep it simple"
Sorry I havn't answered until now, but some of the comments made never get back here, but talking to friends that really recommend freenas I am back again. As of now, I am running Ubuntu Server with a storage zfs array of 8 NAS-grade/Enterprise-grade SATA disks.
Working perfectly.
 
Joined
Oct 18, 2018
Messages
969
As of now, I am running Ubuntu Server with a storage zfs array of 8 NAS-grade/Enterprise-grade SATA disks.
Working perfectly.
If you're using ZFS anyway you might find FreeNAS easier than rolling your own using Ubuntu Server. All of the hardware recommendations in this forum have more to do with ZFS than with FreeNAS and would apply equally well to ZFS on another OS. The exception being recommended hardware that might not be well supported by FreeBSD such as certain NICs etc.
 

IQless

Contributor
Joined
Feb 13, 2017
Messages
142
Do be careful with any type of cloud storage, I had to recover my files from Jottacloud a few years ago, and when all files were downloaded I found out multiple files were corrupted and broken. The file transfer (at least one they had a few years ago) did not to my knowledge checksum the files transferred, so the files could have been corrupted already on arrival.
I were lucky, turned out I had managed to have multiple backups (by accident), so I got back almost all of my files. Some were lost forever, but that was some mp3's or something, no big loss.
 

xcqn

Cadet
Joined
Mar 19, 2017
Messages
9
Sorry I havn't answered until now, but some of the comments made never get back here, but talking to friends that really recommend freenas I am back again. As of now, I am running Ubuntu Server with a storage zfs array of 8 NAS-grade/Enterprise-grade SATA disks.
Working perfectly.
If you really want to make the jump to FreeNAS may i suggest the following tweaks to your system.

I don’t know if you are aware but AMD ”silently” supports ECC on at least some processors in the Ryzen lineup. There is a possibility you could switch out that RAM for ECC modules. If possible, just do it. ZFS is ment to run on ECC Ram.

If your NIC turns out to be Realtek i sugest you drop-in a PCI-E intel NIC instead. It will be better supported in UNIX-like systems and perform better too.

Regarding the Delock SATA add-in you are using. I personally tried that at some point. I ran into problems after a while (random checksum errors showing up during scrubs) i wouldn’t trust it. They are cheap and nasty. Get a proper LSI SAS add-in and flash to IT-mode. You can find used for about 50$

If you do these things you should be able to sit back relax and enjoy FreeNAS.
 
Joined
Oct 18, 2018
Messages
969
Sorry I havn't answered until now, but some of the comments made never get back here, but talking to friends that really recommend freenas I am back again.
Welcome back. What hardware did you end up going with?
 
Top