Scrub over 100%

[seppok]

Code:

[root@freenas1] ~# zpool status tank
  pool: tank
 state: ONLINE
  scan: scrub in progress since Sun Jan  4 00:00:12 2015
  32.7T scanned out of 1.63T at 83.3M/s, (scan is slow, no estimated time)
  0 repaired, 1998.73% done
config:

  NAME  STATE  READ WRITE CKSUM
  tank  ONLINE  0  0  0
  raidz1-0  ONLINE  0  0  0
  gptid/ed85192c-fb57-11e0-b89c-e0699562c744  ONLINE  0  0  0
  gptid/ee431903-fb57-11e0-b89c-e0699562c744  ONLINE  0  0  0
  gptid/eefc6b1a-fb57-11e0-b89c-e0699562c744  ONLINE  0  0  0

errors: No known data errors

If I try to do zpool scrub -s tank the machine reboots. I tried to detach and import the volume but the scrub continues.

Any ideas?

 

[enemy85]

That's really weird...i guess you are not using disks bigger than 10TB, so there's for sure a problem. What's are your hw specs?

 

[enemy85]

Plus, how does the problem started? Something in particular happened to your server pr it was just a routine scrub?

 

[seppok]

I only have 3 2TB disks on a dual core Atom system and I know it is a bit low spec for Freenas. I recently did the upgrade to Freenas 9.3 and did the zfs update it was asking. This is maybe the first scrub after that. At the same time I also accidentally disconnected one sata cable but I noticed it soon and Freenas was happy after resilvering.

 

[enemy85]

How much ram? It could be that it's not enough to let the system work properly or it could be that the zfs upgrade with the new flags made something else. Let's wait for someone else more knowledged to chime in.

 

[seppok]

Only 4GB RAM.

 

[Ericloewe]

Only 4GB RAM.

Yeah, that's known to cause trouble.

 

[seppok]

The motherboard supports only 4GB but it has worked fine for a long time. Is there anything I could try? It's strange that scrub -s causes immediately a lot of text to the console and a reboot.

 

[9C1 Newbee]

Better hardware is what I would do.
Going back to an older version of FreeNAS that might work better is also out of the question since you upgraded the pool.

 

[seppok]

Ok, but is there any way to make it usable before I can get the new hardware?

 

[cyberjock]

No. But in the past when we've seen people with >100% scrubs, their pools ended up being corrupted in a way that it got stuck in a loop it never got out of. So if you don't have a backup, now is the time to make one.

 

[seppok]

Yeah, only problem is that the machine seems to restart after some time. Maybe it does the same as when I try to end the scrub. Maybe I should check all cables etc. but I would be happy to try if you have any suggestions.

 

[Starpulkka]

No one has sayed memtest yet. Funny thing . Are you sure you have identical big hardrives? You can find few same cases what you have, if you try put more effort on duckduckgo. If you have lost 1 drive and have much more memory errors while using degraded state and you put back 3rd drive you then dont have enought parity to complete scrub and also you might have memory erros in same time. So you should now understand The scrub loop you have. But thank you for humor 1998% scrub was funny =) (im sorry for this, i should not feel humored on anothers down)

 

[cyberjock]

Yeah, only problem is that the machine seems to restart after some time.

That would be your indicator that your scrub probably is corrupt. The other users had their boxes crash and reboot because of ZFS. So yeah, bad pool is likely and you should backup while you can.

Make sure you upgrade to 8GB of RAM to prevent corruption and such. There's a reason why we keep harping on people about 8GB of RAM minimum.

 

[seppok]

I think I'm going to try the pool on new hardware and hopefully get all my data. I'm not going to let it continue the scrub on my old hardware...

The only thing I cannot understand is how can the pool go corrupt with 4 GB RAM? I understand that it can be a lot slower and all (using swap) but how badly can the filesystem or Freenas be designed if not enough RAM causes corruption? How can I be sure that 8 GB is enough? Over 4 GB enables prefetch by default and that already uses a chunk of the memory?

edit: And yes, all my 3 hard disks are identical. I didn't have to replace any drives, one drive was just disconnected for a couple of hours accidentally as a sata cable came loose when I installed the case cover. Resilvering was quickly completed and I thought everything is fine. I think it was after that when I upgraded the pool.

 

[cyberjock]

The only thing I cannot understand is how can the pool go corrupt with 4 GB RAM? I understand that it can be a lot slower and all (using swap) but how badly can the filesystem or Freenas be designed if not enough RAM causes corruption? How can I be sure that 8 GB is enough? Over 4 GB enables prefetch by default and that already uses a chunk of the memory?

Well, considering we've probably seen 50+ cases of people that got corrupt pools with 4GB of RAM, pretty high.

We keep telling people 8GB of RAM. We've put it in the minimum requirements that if you don't have 8GB of RAM you should get more powerful hardware before using FreeNAS. At some point the responsibility has to be put on the server admin to do the right thing, right?

From the manual:

If you don't have at least 8GB of RAM with ZFS or 2GB of RAM with UFS, you should consider getting more powerful hardware before using FreeNAS® to store your data. Otherwise, data loss may result.

Seems fairly straightforward to me.

 

[abaumgaertner]

guess I'm lucky then - 5 older systems home multimedia storage - all with 2GB Memory only.
no data loss, no swaping and still free memory ... impossible - no

it's imparitive that you run memtest and make sure your hardware is 100% working before you contiue the scrub

also check out dmesg and run tail -f /var/log/messages while the scrub continues ... please paste the output

and there is definetly something wrong with your pool: 32.7T with 3 Drives is (at least today) impossible
Please run a zpool history and paste the output.

 

[gpsguy]

Yes, you are lucky.

Hopefully, you are running FreeNAS 0.7 or using UFS on FreeNAS 8.0.x

guess I'm lucky then - 5 older systems home multimedia storage - all with 2GB Memory only.

 

[cyberjock]

guess I'm lucky then - 5 older systems home multimedia storage - all with 2GB Memory only.
no data loss, no swaping and still free memory ... impossible - no

Memory errors aren't the problem. In fact, we aren't quite sure what the problem is when you don't have enough RAM. We've just seen it over and over again with lots (LOTS) of user. Eventually it turns into something you can't ignore and the only responsible thing is to say "hey, we got this problem, don't use less than this much RAM".

Can it work? Yes.
Should it work? I don't know.
Should your pool fail when you run out of RAM? Not according to the ZFS devs.

But we all have seen it happen so many times its hard to ignore the reality.

 

[abaumgaertner]

Thanks for your feedback - I'll keep it in mind.

have you heard of successfull zpool recoveries after such failures / corruption ?

BTW: I like your FreeNAS guide - very helpfull to noobs - using raidz2 or raidz3 is the most important point!
Maybe you could add that the pools should be sized properly (recommended less than 80% full)