data-rot and filesystems

[johnblanker]

Is data-rot all that common? I am relatively new to NAS and server systems. It seems there are 3 common filesystems that have protection for data rot, ZFS, BtrFS, and ReFS (Windows). Are all these created equal?

I am in the stages of researching which NAS OS is right for me. All I need is to serve movie ISOs on my network. Just simple SMB or NFS shares. The data rot thing does have me concerned though.

Why is it all these NAS units (Synology, Qnap, etc) do not have support for dealing with data-rot? None of these users care? I see the Drobo units have data rot support but I have been reading a ton of bad things about Drobo units.

I am currently looking at unRAID with BtrFS as a possible NAS OS. One member on the unRAID forum suggested going with XFS over BtrFS and that there are checksums that can be used for data rot detection with XFS. Does that just check for errors or heal them?

Just wanted to get some ideas for you power users.

Thanks.

 

[mattbbpl]

I don't have all the answers to this question, but to chime in on a few pieces:

Checksums detect data rot errors, and parity is used to correct them. It is my understanding that both must be present in order to correct bit rot issues. (How can checksums repair the data if it doesn't have a verifiably correct copy of the data to pull from? How can parity tell which copy of the data is correct if there isn't a checksum to verify that?)

"Why is it all these NAS units (Synology, Qnap, etc) do not have support for dealing with data-rot? None of these users care?"

In response to the above, I'd wager that most of the users either don't know, or don't want to deal with the hassle of learning how to protect against it and actually doing it. We live in a world where most consumers can't even be bothered to defrag a hard drive. Do we really expect them to set up jobs to run SMART tests and scrubs? Those units are meant for the lowest common denominator that think they might like a NAS but don't want to get too involved in managing their data.

 

[Mirfster]

I can't speak on the other units, but avoid Drobo at all cost. This is from someone who has had multiple Drobo units and "been around the block" with them multiple times. I have returned no less than 3 units. When they die, they just flat-out die. That is one of the main reasons I found my way to FreeNas.

 

[jgreco]

Is data-rot all that common? I am relatively new to NAS and server systems. It seems there are 3 common filesystems that have protection for data rot, ZFS, BtrFS, and ReFS (Windows). Are all these created equal?

Bit rot is rather unusual in well-designed, well-maintained server systems. It can and does happen, often because it is difficult to determine a failure from a success for an operation such as a read retry.

The larger problem is that a lot of people think they are server geniuses and know all about how to make a good file server. I think everyone had a great laugh at the LinusTechTips thing. Unfortunately it hits a little too close to home. Some of us have spent decades learning the trade and we still learn new stuff, often daily.

I am in the stages of researching which NAS OS is right for me. All I need is to serve movie ISOs on my network. Just simple SMB or NFS shares. The data rot thing does have me concerned though.

Don't freak out about it. You can always go back to the source for an ISO.

Why is it all these NAS units (Synology, Qnap, etc) do not have support for dealing with data-rot? None of these users care?

Why did people drive around before the 1980's without fastening their seatbelts? They didn't care. Individually, nobody ever got into an accident. Why worry about those pesky inconvenient seat belt things?

Dealing with bitrot is expensive. It means you need an infrastructure to do it. It means checksumming your data. If you're Synology or Qnap, and you can build a $400 NAS that has a cheap SoC and some SATA ports, or you can build a $500 NAS that has a decent CPU and some SATA ports, with enough CPU to do the checksumming, what do you think sells?

Seat belts only ever got sold to the general population because they were mandated AND THEN also because police gave tickets for failure to wear them.

You think users care about bit rot? They don't even think it's real. And for most of them, it will not affect them.

I see the Drobo units have data rot support but I have been reading a ton of bad things about Drobo units.

I am currently looking at unRAID with BtrFS as a possible NAS OS. One member on the unRAID forum suggested going with XFS over BtrFS and that there are checksums that can be used for data rot detection with XFS. Does that just check for errors or heal them?

No clue. We do ZFS here. ZFS will detect and repair bitrot, assuming it is possible to recover the data from redundancy.

 

[johnblanker]

It took a lot of time to rip 10TB. Going back to the disk is really not a viable option. Speaking of the Linus thing, that was more relating to failure without having a backup employed. As far as FreeNAS, does it automatically detect and fix errors on the fly, or do users need to manually run the checksum or other process periodically?

It seems $500 is the entry level for FreeNAS. This option might be out of my $ range at the current time.

"You think users care about bit rot? They don't even think it's real. And for most of them, it will not affect them."
I don't understand, why will it not affect them?

Thanks for the help.

 

[mattbbpl]

If the estimated $500 price is too high, some users have had good luck getting under that (without disks, of course) by picking up a used TS140 or T20 server.

 

[jgreco]

If one file goes corrupt, you do not need to re-rip all 10TB. You need to re-rip a single disc. 4GB. Or 8GB. Or 50GB. Whatever.

"You think users care about bit rot? They don't even think it's real. And for most of them, it will not affect them."
I don't understand, why will it not affect them?

They won't notice it. Maybe they never access the data again. Or they'll notice but they'll dismiss it because they don't realize what they're seeing. Or they'll dismiss it as a bug in the program. Or maybe they're lucky and nothing ever rots on them.

Looking at it from the point of ISO's, maybe you just attribute that video glitch when you're playing it back to a scratch on the original media.

Who the hell knows. The wonderful thing is that most people float on through their lives blissfully ignorant of things like this. If and when it happens... they write it off.

See, now think about the whole seat belt thing again. Everyone used to ride around in cars without seat belts. Ignorance is bliss. The fact that you're wearing a seatbelt today doesn't mean that you are going to get into a car crash. It's just there to help increase your odds if you do happen to get into a car crash.

 

[rsquared]

As far as FreeNAS, does it automatically detect and fix errors on the fly, or do users need to manually run the checksum or other process periodically?

Yes, and yes... ZFS will validate against the checksum each time you read a file, but you should also schedule a regular scrub to check the full drive.

 

[johnblanker]

haha, your cynicism is making me laugh. I get what you are saying though. I understand the need for businesses and the need to protect crucial data. I would like that too, but like you said, for movies, I'm not sure the trade-off of buying the stuff and then learning a great deal is worth it. Perhaps in the future when I have more time.

They say that copying data back and forth between the array and the backup will prevent bit rot, I guess to keep it fresh and preventing it from "rotting". I'm thinking of cabbage or other types of vegetables here.

Is there any way to check for data rot on Windows' systems? NTFS file systems?

 

[johnblanker]

Yes, and yes... ZFS will validate against the checksum each time you read a file, but you should also schedule a regular scrub to check the full drive.

I see. thank you.

I get the feeling that most of the people on this forum are big-time server gurus that spend most of the time working for Google in their server rooms. Is learning FreeNAS as hard as it seems? My experience for the last 15 years has been exclusively building Windows machines and some "light" networking. i.e., checksums and scrubs are a foreign language.

 

[johnblanker]

If the estimated $500 price is too high, some users have had good luck getting under that (without disks, of course) by picking up a used TS140 or T20 server.

Amazon has some of these for $250. Is the Xeon cpu really worth it? I might have to make a choice betweeen Xeon with 16GB, or i3 with 32GB.

 

[mattbbpl]

Amazon has some of these for $250. Is the Xeon cpu really worth it? I might have to make a choice betweeen Xeon with 16GB, or i3 with 32GB.

The Xeon CPU is really only worth it if you intend to do CPU heavy tasks such as transcoding or other "extras" via jails. An i3 would suffice for a FreeNAS file server. Just make sure it supports ECC RAM.

 

[mattbbpl]

I see. thank you.

I get the feeling that most of the people on this forum are big-time server gurus that spend most of the time working for Google in their server rooms. Is learning FreeNAS as hard as it seems? My experience for the last 15 years has been exclusively building Windows machines and some "light" networking. i.e., checksums and scrubs are a foreign language.

Is learning FreeNAS hard? I would say learning what you need to be safely functional in it (ie. build and run your own server) would probably take about a month if you have no FreeBSD background to start with. You don't need to know WHY you need ECC RAM, why you need to run the scrubs, etc. However, you need to know that you SHOULD use ECC RAM, you SHOULD run Scrubs every couple weeks, and you need to know enough to set up your storage pool and not fill it completely. Just some basics akin to knowing that you need to change the oil in your car every 3 months.

 

[johnblanker]

The Xeon CPU is really only worth it if you intend to do CPU heavy tasks such as transcoding or other "extras" via jails. An i3 would suffice for a FreeNAS file server. Just make sure it supports ECC RAM.

Thanks. No, I just plan to serve up files. No transcoding. My Dune does it all.;) It seems all of Amazons pre built systems all come with a form of ecc memory. I would prob go with the lowest amount and just get my own 16GB since I can get the brand I want. I suspect you guys like crucial and kingston?

 

[mattbbpl]

Thanks. No, I just plan to serve up files. No transcoding. My Dune does it all.;) It seems all of Amazons pre built systems all come with a form of ecc memory. I would prob go with the lowest amount and just get my own 16GB since I can get the brand I want. I suspect you guys like crucial and kingston?

That's good. Try to get close to the 1 GB RAM per 1 TB storage rule.
We like Crucial. Guys around here don't like Kingston because they've not been honest with their part numbers in the past. But really, as long as you get a RAM model that is verified to work with your motherboard (this is important), then you should be fine. Part of the benefit of getting the amount of RAM you want up front in the pre-built system you order is that you don't have to worry about it.

 

[jgreco]

They say that copying data back and forth between the array and the backup will prevent bit rot, I guess to keep it fresh and preventing it from "rotting". I'm thinking of cabbage or other types of vegetables here.

No, actually handling the data increases risk. Every time you move the data, a plethora of things must happen perfectly in order for your data to remain intact. There are a wide variety of risks, each of which is fairly unlikely on its own. It's certainly possible that data might "fade" on disk, which seems to be the way you're thinking of bit rot. But if you copy the data, the data has to go through a disk controller, be read into memory, then has to be written back out through another controller, and what if something happens to one poor little bit along the way? Your average desktop system doesn't sport ECC memory or other mitigating technologies to protect the base system.

Is there any way to check for data rot on Windows' systems? NTFS file systems?

Sure, run a periodic checksum of all your files and then compare it against the known-good checksum list created at an earlier point. That sucks and so you don't do it. Very few people do. That's why it's better to just have the filesystem grow a pair and take responsibility for the overall problem. As users, we inherently believe that a filesystem is supposed to just magically store things and retrieve them later reliably. Most other filesystems are reliant on the underlying technologies to all be working correctly, and then usually that pans out. However, it's not a guarantee, so ZFS is proactive. ZFS actually makes *sure*. Every access is verified and (if need be) reconstructed from redundancies.

 

[johnblanker]

Thanks. It seems the Lenovo ts140 would be better since it offers 5 sata ports over the dell which only has 4. It seems like ECC is the most important thing when it comes to data integrity, whereas the mobo and cpu are more related to performance, an area where I might be able to cheat a little since I only plan to serve.

 

[Zorin1]

Amazon has some of these for $250. Is the Xeon cpu really worth it? I might have to make a choice betweeen Xeon with 16GB, or i3 with 32GB.

I went and bought a barebone poweredge T20 from Dell, around 17?.00. This gave me a powersupply, motherboard, case and CPU. Not the fastest CPU but good enough. I then bought 32 gigs of ECC memory. This cost me over $240. I'm kind of regretting buying ECC memory now. The same issue that you would have with a NAS you would have with any PC. If memory gets corrupt and written you could corrupt a file. But really, how often does this happen? Might have been better for me to buy a UPS. Most problems are going to happen when power to the server gets interrupted during a write. For me though, the NAS is my backup. I say this because other machines have a copy of the data on them. I only put them on the NAS as a backup. I don't use the NAS for everyday use. I plan on putting backups of my computer on them. But if those get lost no big deal. I'm not looking to be 100% bullet proof here. That would cost a lot of money.

By the way Newegg has a sale on the Western digital RED 4TB drives for the next 48 hours or so. Saved me a total of $60 when I bought four of them.

 

[mattbbpl]

Why would you regret buying ECC RAM? The price difference between ECC and non-ECC is very minor.

 

[Mirfster]

I'm kind of regretting buying ECC memory now

Where/Who did you buy the memory from? $240.00 sounds a bit high, but either way it is cheaper than losing data.