RAM memory scrub option?

[Knowltey]

OK, that's fine if you want to make that claim, but you need to provide proof as to why it would not decrease it or increase it.

Because right now it's the source code itself plus also the main ZFS developer himself claiming that it should reduce it vs your word.

Can you explain why checksumming the data at rest and then verifying it a second time before committing it to disk is more likely or equally likely to cause an error than checksumming it as it passes through memory and not verifying it a second time?

Look you aren't understanding us. You are claiming this it does x. We are saying the we don't know if it does x or y or z. and there isn't proof because nobody has actually fucked around with it enough to know what exactly it does in your hypothetical situation.

But as cyberjock said, even IF it is the case that it is emulating ECC, it won't likely work because bitflips on non-ECC RAM tend to cause large chains of bitflips, and ECC can't correct more than one bitflip, so an emulation of ECC won't do shit for it either.

 

[SirMaster]

The point of enabling the flag is that it adds an extra verification step that checks the data again and if it doesn't match with the first time it checked it, it wont commit the data to the disk and thus wont corrupt that part of your pool at that point. Sure it cant always work if the corruption is bad enough, but it's better than not double checking, and I can't possibly see how double checking could be worse. It only commits if the verification matches twice.

 

[SirMaster]

What are you talking about nobody has messed around with it? How can you possibly know that or even make that claim?

I already covered this. The guy who added the feature himself has used it extensively and I have had conversations with him about it's purpose and to clarify how it works and if you understand logical statistics then you will be able to see why it can reduce the probability that a corrupted block in memory is written to disk at least occasionally which is more often than none times when the code doesn't run because the flag is disabled.

It's obviously meant for debugging (hence "unsupported") and so one can do additional checks on the data coming out of the ARC, but it has the nice side effect of possibly preventing bad data from being actually committed to disk.

It could actually even prevent correct data from being committed to disk, but then that is still better than possibly writing a corrupt block.

If you are not willing to look at what the piece of code does and look at the math behind it's implications then you really have no place in this discussion... I'm sorry, but you can't just make claims without actually looking at the facts available to you.

 

[Knowltey]

What are you talking about nobody has messed around with it? How can you possibly know that or even make that claim?

Well maybe someone has, but they surely haven't publicized their research on the matter that I'm aware of. If you know of some be our guest.

works and if you understand logical statistics then you will be able to see why it can reduce the probability

Yes, and if you understand how ZFS works you'll understand that reading bad blocks from memory is a very bad thing to be doing. Forcing the system to do it where it normally wouldn't is even worse.

you can't just make claims without actually looking at the facts available to you.

What? Are you even reading what we're saying? That's exactly what we are trying to say to you! We aren't making a claim either. We are explicitly saying that a claim can't be made to how that function works. You are saying that it reduces the chances. We are telling you that you can't be sure of that.

 

[SirMaster]

Again, from looking at the code, in the best case it can prevent bad data from being committed if it does not match the second verification.

And in the worst case it certain can still let the corruption slip by as if the flag were off.

 

[Knowltey]

And in the worst case it certain can still let the corruption slip by as if the flag were off.

That is exactly what we've been trying to tell you this whole time!

 

[SirMaster]

Right, but you seemed like you were ignoring the best case which can occasionally happen. And the best case with the flag on is better than the best case with the flag off because the second verification has a greater than 0 chance to catch and prevent a bad commit where there is simply no second verification when the flag is off so it's always a 0 chance that it catches anything because in that case it doesn't even exist.

 

[Knowltey]

Right, but you seemed like you were ignoring the best case which can occasionally happen. And the best case with the flag on is better than the best case with the flag off because the second verification has a greater than 0 chance to catch and prevent a bad commit where there is simply no second verification when the flag is off so it's always a 0 chance that it catches anything because in that case it doesn't even exist.

Right, but how can you know that the best case scenario even works, or is even possible with the way that regular RAM usually fails? How do you know that it doesn't cause some other unforseen bug that can destroy data in some other strange way?

How do you know there isn't some huge performance hit for turning this flag on that wouldn't be worth the small reduction in probability if that even exists?

 

[SirMaster]

Well I am quite sure that the flag wouldn't cause some other unforeseen bug that would affect data in some other way because the code just doesn't do all that much nor does it interact at all with any other part of the data on disk (compared to having the flag off) The execution can't just magically jump the call stack unless we have very serious system problems already haha. So I'm assuming it will only ever execute the very deterministic lines of code that the flag enables.

The start of the code is when reading a block from ARC, and the end of the code is at committing to disk. In essence, the code in question is simply a new extra check that computes a checksum a second time on the same block. And then a new extra check that compares the 2 checksums and decides whether or not to even commit anything to disk if they match or don't match.

If the memory corruption is minor enough the catch should see the problem and trigger a non-commit. From memory errors I have seen in the past memory errors this minor can and do happen so my hypothesis is that if one of these minor errors that do happen does happen, then the check is capable of catching it at the right time and place.

But then yes of course the corruption can be a lot worse and it won't do any additional good.

It really will come down to how bad things are and how bad your RAM is behaving.

But if we agree that non-ECC RAM can have single bit errors, then I see no reason why the code is not capable of catching these at least occasionally and preventing bad commits.

 

[Knowltey]

Where is it comparing the checksum to though? If it's comparing it to the one stored on disk then you're going to see a HUGE performance loss when the system is accessing the disk everytime it needs to read from RAM.

 

[SirMaster]

Right, this is why it's meant for debugging and why it's unsupported. But speed was never a part of any claim I had made. I'd bet some would be willing to trade speed for any extra reliability given the choice in many scenarios. Typically speed just has to be "fast enough" for a given user's needs and some users don't actually need much for their use cases and workloads especially in a home environment (media servers and things) where you tend to find non-ECC RAM IMO.

It will also cause some strange things to applications using the zpool like data loss when data in the arc isn't getting committed to disk. But hey at least it's then corrupting less data permenantly on the pool. Lesser of 2 evils?

I didn't ever mean it was some sort of amazing feature or anything. Just thought that it was relevant and sort of similar to a discussion on "scrubbing ram" because it kind of performs an extra check on the RAM.

 

[Knowltey]

Right, this is why it's meant for debugging and why it's unsupported. But speed was never a part of any claim I had made. I'd bet some would be willing to trade speed for any extra reliability given the choice in many scenarios. Typically speed just has to be "fast enough"

I highly doubt you are going to get anything resembling good enough speeds when everything from RAM has to also be read from the disk and verified. I mean it's going to be a hell of a lot slower, not just a little.

 

[jgreco]

The correct fix for bad memory is to fix the memory, not try to halfarse around it in code. I'd be all for checksumming stuff in core if errors resulted in a red screen of death that said "Your NAS has bad RAM and is corrupting stuff. Your NAS has been halted to prevent pool damage. Have a nice day."

There's an admittedly too-sparse sticky on hardware burn-in in the hardware forum.

 

[AlainD]

The correct fix for bad memory is to fix the memory, not try to halfarse around it in code. I'd be all for checksumming stuff in core if errors resulted in a red screen of death that said "Your NAS has bad RAM and is corrupting stuff. Your NAS has been halted to prevent pool damage. Have a nice day."

There's an admittedly too-sparse sticky on hardware burn-in in the hardware forum.

Basicly that's why I suggested the option. Most of us do a burn-in (I hope and expect) and then the box keeps on being used for a long time without any further tests. I doubt that none enterprise scale users will check the ECC-error logs on the hardware.

After reading :

http://research.cs.wisc.edu/adsl/Publications/zfs-corruption-fast10.pdf

It became very clear to me that there's a hole with the memory protection inside ZFS. Some of it could be found with an extra software scrub (which can be done without impacting a lot on the core code), fixing it on the meta data would probably made it incompatible with existing pools (and it seems an awfull lot harder to me).

ECC-memory will lower the odds for problems a lot, but not fix it. Although the odds will be low enough for most of us to get problems.

 

[jgreco]

It's a tradeoff. The intended target for ZFS was a Sun server, which was going to be something that had ECC and proper burn-in testing as part of the design. ZFS was designed assuming that the underlying server platform was reasonably trust-able.

Taking ZFS out of that environment essentially comes with the possibility of invalidating some of those assumptions; putting ZFS on a box without ECC (for example) is risky. Around here we argue, essentially, that people should build the PC equivalent of Sun servers - properly designed and tested quality server platforms. The people who do this do not seem to have any problems with corruption due to memory issues.

Basically computers frequently remind me of cars. If you've got a Ford Pinto, I suggest that you not take it offroad and be mindful that the gas tanks are known to explode. You might be able to haul some lumber with your ordinary car;

but quite simply just because it has four wheels, that does not make it a delivery truck. So rather than claiming the car was designed wrong and that you could make this change and that change to make it better, the correct solution is to actually get the vehicle that is appropriate to the job.

 

[AlainD]

It's a tradeoff. The intended target for ZFS was a Sun server, which was going to be something that had ECC and proper burn-in testing as part of the design. ZFS was designed assuming that the underlying server platform was reasonably trust-able.

Taking ZFS out of that environment essentially comes with the possibility of invalidating some of those assumptions; putting ZFS on a box without ECC (for example) is risky. Around here we argue, essentially, that people should build the PC equivalent of Sun servers - properly designed and tested quality server platforms. The people who do this do not seem to have any problems with corruption due to memory issues.

Basically computers frequently remind me of cars. If you've got a Ford Pinto, I suggest that you not take it offroad and be mindful that the gas tanks are known to explode. You might be able to haul some lumber with your ordinary car;

but quite simply just because it has four wheels, that does not make it a delivery truck. So rather than claiming the car was designed wrong and that you could make this change and that change to make it better, the correct solution is to actually get the vehicle that is appropriate to the job.

But are servers like the Sun servers (with mostly system admins watching after them) the target for freenas?
I 100% follow ECC-ram, but I suspect that almost no users look at their ecc-error logs, if they even know where to look.
I also suspect that most freenas servers aren't in climate controlled computer rooms.

Maybe ecc-ram after burn in is secure enough, I don't now, but the situation keeps me "unsecure".

 

[cyberjock]

But are servers like the Sun servers (with mostly system admins watching after them) the target for freenas?

If you've setup your server properly you shouldn't have to watch over them at all. They'll tell you when stuff hits the fan... example: http://forums.freenas.org/index.php?threads/how-freenas-responds-to-a-failed-disk.22289/

I 100% follow ECC-ram, but I suspect that almost no users look at their ecc-error logs, if they even know where to look.

If you've done your job and bought quality hardware you should get an email from FreeNAS about single-bit RAM errors and the system should halt on multi-bit RAM errors. So either it's a situation where your data wasn't corrupted because it was corrected or it could have been corrupted but the system was halted. In either case the outcome is exactly what you want. ZFS isn't corrupted.

I also suspect that most FreeNAS servers aren't in climate controlled computer rooms.

Mine is not in a climate controlled room. But it stays reasonably cool and has worked magnificently for me.

 

[AlainD]

...

If you've done your job and bought quality hardware you should get an email from FreeNAS about single-bit RAM errors and the system should halt on multi-bit RAM errors. So either it's a situation where your data wasn't corrupted because it was corrected or it could have been corrupted but the system was halted. In either case the outcome is exactly what you want. ZFS isn't corrupted.

...

Hi Do understand it correctly that freenas sends an e-mail (like with the smart service)? Or does you're motherboard sends an e-mail?

I didn't find any info on how to set upt sending email from freenas on ECC errors.

 

[scotch_tape]

I look at threads like this and I wonder if they are some social engineering designed to wear down the FreeNAS staff so that they unwittingly clobber the latest newbie that asks a stupid question. Cyberjock, why don't you create a sticky of "shit we are not even going to debate with you" and lock every stupid thread that comes along and point them at the sticky?

The FreeNAS "experts" only have a limited number of cycles and I'd rather them expend it on something productive and not something worthless.....

 

[cyberjock]

I look at threads like this and I wonder if they are some social engineering designed to wear down the FreeNAS staff so that they unwittingly clobber the latest newbie that asks a stupid question. Cyberjock, why don't you create a sticky of "shit we are not even going to debate with you" and lock every stupid thread that comes along and point them at the sticky?

The FreeNAS "experts" only have a limited number of cycles and I'd rather them expend it on something productive and not something worthless.....

That's what I wanted to do... and the forum maintainer didn't agree with me. Feel free to let her know... she's in IRC most of the time with the username dlavigne.