Important data on RaidZ, cant mount HELP!!!

[Miodrag Kovačev]

I am using Freenas 9.2 with 4 drives:
3x 250GB RaidZ
1x 1TB ZFS

I am using FreeNAS for two years, and I never had any problem with it.
Yesterday I was moving some data from local computer to NAS when FreeNAS stopped to respond.
I found this message on screen:

Then detached RaidZ drives, then system waked up. I removed RaidZ Volume from FreeNAS, shutdown, attached 3 RaidZ drives, turn on. After this I tried to Auto Import Volume. FreeNAS sees the RaidZ volume but when I Auto Import it the System reboots. I even tried to import Volume through console, using 'zpool import RaidZ' command. After this command, FreeNAS started to read something on HDD, then unreadable text started to show (run) on screen and after few seconds system restarts.

I even tried to make new USB, new system but when I try to import volume, same thing happens.

I would appreciate if someone could help me . I'm very desperate because I used super safe RaidZ volume to store family photos, videos, and some other personal stuff.

Thanks

 

[Miodrag Kovačev]

Here is what zpool import command returns:

Code:

zpool import
  pool: RaidZ
    id: 7194990289600790111
  state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
config:
 
        RaidZ                                          ONLINE
          raidz1-0                                      ONLINE
            gptid/6c146c59-6d56-11e3-bd36-0013d4db0be5  ONLINE
            gptid/6ca97a2f-6d56-11e3-bd36-0013d4db0be5  ONLINE
            gptid/6d482399-6d56-11e3-bd36-0013d4db0be5  ONLINE

 

[Miodrag Kovačev]

Hardware specification:
RAM: 875MB
CPU: AMD Athlon(tm) 64 Processor 3000+
FreeNAS-9.2.0-RELEASE-x86 (ab098f4)
CPU load was very high when system crashed: 3-4

 

[Miodrag Kovačev]

~# zpool import
pool: RaidZ
id: 7194990289600790111
state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier, though
some features will not be available without an explicit 'zpool upgrade'.
config:
RaidZ ONLINE
raidz1-0 ONLINE
gptid/6c146c59-6d56-11e3-bd36-0013d4db0be5 ONLINE
gptid/6ca97a2f-6d56-11e3-bd36-0013d4db0be5 ONLINE
gptid/6d482399-6d56-11e3-bd36-0013d4db0be5 ONLINE

 

[cyberjock]

You have 1/8th of the minimum recommended RAM. More than likely you killed your pool from insufficient RAM. If you checked out the system requirements for FreeNAS 8 it has always said something like 6GB of RAM(or 8GB of RAM) for ZFS.

You should upgrade to at least 8GB of RAM before trying to import your pool again. Continuing to try to import a pool with that little RAM may permanently damage the pool if it is not already damaged beyond recovery.

 

[Sir.Robin]

cyberjock, could you explain exactly why (what happens) a low memory or kmem panic can destroy a pool?

 

[cyberjock]

I can't, because nobody knows what exactly goes wrong. ZFS is supposed to be impervious to partial writes and sudden loss of power, but for whatever reason it sometimes fails. Nobody here really puts much time into it because we all know better and simply follow the minimum requirements for FreeNAS.

 

[david kennedy]

If you take a look on the net you will find a number of references to these sort of issue. The "classic" reason is a pools committed storage grows beyond a "magical" limit and can no longer be mounted without more ram.

While the pools are small:

3x 250GB RaidZ
1x 1TB ZFS

So is the memory in the system.

Upgrade the ram and let us know if it mounts.

 

[cyberjock]

Not sure what problem you are referring to david, but its well known that the pool itself gets damaged, permanently. Some people have attributed it to low-RAM conditions and a code migration to FreeBSD that overlooked or created some rare bug that presents itself in low RAM conditions. Adding more RAM after you've trashed your pool won't do you any good. This "low RAM killing pools" is very random, very permanent, and very unrecoverable. At least, nobody I've seen in my 2 years here has recovered. Surely, if your data is worth the cost of a new vehicle, someone like Ontrack might be able to help.

In any case, the 8GB of RAM minimum seems to be the point at which nobody has definitively had this problem. I've run 12GB of RAM with 36TB of data and aside from extremely poor performance it did ran. I couldn't even stream 1 video, but it did run. Of course, to add to this mess the issue doesn't always present itself to everyone, and not always. We've had people that had weekly panics and hit the reset button and kept going. We've had users that had a single panic and that was the end of their pool. So my 3 weeks with 12GB of RAM isn't exactly a good indication of expected results.

 

[david kennedy]

I'm moving over to freenas from Solaris and there are reports where people have had pools run fine for a while. One day they restart and the pool wont import until they add more ram.

This might also explain his issue. He stated he was copying files and things blew up, perhaps he hit the "pool size vs available ram" limits?

What you say is interesting as on my solaris machine (the one i am considering moving to freenas) I have around the same storage as you (36TB) with 16GB and it runs fine.

I can stream to multiple clients at once (typically 5 or 6 at a time, all HD) and i run a few apps and a 2gb VM as well. Typically when i copy to/from the pool i saturate the gig network (i can do this while streaming without issue) and its been like this for around 2 years.

 

[Sir.Robin]

I can't, because nobody knows what exactly goes wrong. ZFS is supposed to be impervious to partial writes and sudden loss of power, but for whatever reason it sometimes fails. Nobody here really puts much time into it because we all know better and simply follow the minimum requirements for FreeNAS.

Allright... better safe than sorry, but it still would be good to have it explained. :)

 

[david kennedy]

Allright... better safe than sorry, but it still would be good to have it explained. :)

Are you running FreeNas under ESXi?

NAS01: FreeNAS 9.2.0 Rel/ESXi 5.5

 

[Sir.Robin]

Are you running FreeNas under ESXi?

NAS01: FreeNAS 9.2.0 Rel/ESXi 5.5

On my primary yes :)

 

[Miodrag Kovačev]

I tried to import pool in another machine which has 4GB of RAM and the result was same. I've lost family pictures I'we made ​​in last 4 months (last backup). It's not so terrible loss but it hurts. I'm angry, but first of all I'm angry to myself.

I think users like me, were thankful to the development team of FreeNAS to disable the use of ZFS file system, if your computer does not have enough memory. Why? Most of the users are not administrators of large systems, and they do not have enough experience. Most users say: "Yeah, FreeNAS, download, setup, and that's it." Believe me, I'm sure that 90% of users who use the ZFS file system is not aware that their data is at danger! I've used this system for more than 2 years. I'd never used ZFS if I knew that anytime I might lose data. RaidZ = Raid5...Yeah, right! Three discs are safer than one.

What do You think, can I try to recover data on some Ubuntu Linux? Any advice that could help? I dont have money to pay for recovery. Is there some 'fsck' from ZFS?

 

[warri]

I can understand your frustration, but this is exactly why the manual nowadays clearly states that you need 8 GB of RAM when running ZFS. I think they also plan to introduce a very clear warning message in the future when you are trying to use ZFS with too little RAM.

I wouldn't give up yet, though - can you access any other computer with more RAM? Just plug in the disks + your stick (or a fresh installation) there and try to import the pool. Sometimes it also helps to import the pool as read-only from the CLI. If you are not familiar with the CLI, do some testing in a VM first and be sure to understand what you are doing - I'd start with some other threads in this forum where people were able to import their pools after a similar situation.

Unfortunately, there is no such thing as fsck for ZFS.

 

[cyberjock]

From personal experience trying to help others, we've never gotten data back trying to mount it read-only, doing rollbacks, etc. Of course, you have nothing to lose at this point either. More RAM isn't the problem. If you start looking at transactions and stuff you'll see that stuff is actually damaged. More RAM isn't going to fix that problem unfortunately. :(

I can't say I've heard of anyone trying to recover with ZFS on linux, but you don't have much to lose at this point.

Honestly, I wrote off your data when I first read your thread. It happens fairly regularly, and there's nothing we can do to stop it if people don't read the manual. Unfortunately, I don't think I've heard a story yet where someone recovered a pool after having kernel panics on bootup with less than the minimum recommended RAM. :(

 

[Sir.Robin]

Well first of all, i feel yah. It sux loosing data. My better half would kill me if we lost our precius toddler photos/vids.

That beeing said, you should always have a backup. Nomatter what your primary system run.
And i'm sorry to say but RTFM.

If you don't feel youre technical enough to admin FreeNAS or some other DIY solution, maybe QNAP or Synology is a better choice.

Either way... you need backup.

Luckily you didn't loose everything though. "Only" the last 4 months.

 

[Miodrag Kovačev]

I installed ZFS on Ubuntu server 12.04 LTS http://dansysadm.com/blog/installing_zfs_on_ubuntu_1204_n40l.html

Code:

# zpool import
  pool: RaidZ
    id: 7194990289600790111
  state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
  see: http://zfsonlinux.org/msg/ZFS-8000-EY
config:
 
        RaidZ                                        ONLINE
          raidz1-0                                  ONLINE
            ata-WDC_WD2500JS-55MHB0_WD-WCANK1422381  ONLINE
            ata-WDC_WD2500JB-55REA0_WD-WCANKJ102003  ONLINE
            ata-WDC_WD2500JS-55MHB0_WD-WCANK1422323  ONLINE

Code:

# zpool import RaidZ
cannot import 'RaidZ': pool may be in use from other system, it was last accessed by  (hostid: 0x38c6cbd2) on Tue Dec 31 12:28:21 2013
use '-f' to import anyway
 
# zpool import -f RaidZ

This was a point where FreeNAS rebooted.

Linux kernel returns the next message:
Jan 4 00:08:00 kernel: [ 5309.300020] alloc_vmap_area: 143138 callbacks suppressed
Jan 4 00:08:00 kernel: [ 5309.300027] vmap allocation for size 4198400 failed: use vmalloc=<size> to increase size.
Jan 4 00:08:00 kernel: [ 5309.300064] vmap allocation for size 4198400 failed: use vmalloc=<size> to increase size.
Jan 4 00:08:00 kernel: [ 5309.300101] vmap allocation for size 4198400 failed: use vmalloc=<size> to increase size.
Jan 4 00:08:00 kernel: [ 5309.300137] vmap allocation for size 4198400 failed: use vmalloc=<size> to increase size.

 

[cyberjock]

Yeah.. that's a very trashed unrecoverable pool in my opinion. :(

 

[david kennedy]

Are the two even compatible (zfsonlinux and Freenas)?

Out of curiosity I tried to import a pool from freenas and it presents the following warning:

pool: z01
id: 17980638055947465005
state: UNAVAIL
status: The pool can only be accessed in read-only mode on this system. It
cannot be accessed in read-write mode because it uses the following
feature(s) not supported on this system:
com.delphix:spacemap_histogram
action: The pool cannot be imported in read-write mode. Import the pool with
"-o readonly=on", access the pool on a system that supports the
required feature(s), or recreate the pool from backup.
config:

z01 UNAVAIL unsupported feature(s)
ata-WDC_WD4000KD-00NAB0_WD-WMAMY1502434 ONLINE