ZFS Pool not realising missing device?

[brumnas]

Hello, I'm building the first time a FreeNAS, after planning/reading a lot (also here of course ). To get some touch with the ZFS I've installed FreeNAS 9.10 into into a VM and made a few tests for later troubleshooting scenarios surely waiting for me one day ;-).

So I've created a few same sized files with dd, to simulate a few disks (may be my problem is related to using files and not physical drives for vdevs?). I started with a simple mirror, like:
zpool create test /path/disk1 /path/disk2.

Then I've created a text file onto /test/xx pool. Everything ok so far. Then I dropped brutally the disk2 file:
rm /path/disk2

..and I thought ZFS will notice immediately. But it did not. So I've tried to read/write onto /test/ pool - no problem realized; ZFS happily persists to the left disk1. Also scrub and status were nicely reporting 'ONLINE' and 'no errors'. I had to "offline" disk2 (which was not there in that time of course) to have ZFS realize finally that the file is long time gone. Only after that, I could resilver etc. normally. But this would be of course not be usable, to try offline all the disks just to have ZFS realize a problem with a missing disk.

I'm sure I don't understand something here: my intuitive feeling was like 'ZFS notices everything'. Now I'm not sure, what ZFS is actually able - and when? - to notice. But to ignore a missing disk is pretty strange somehow. What I'm missing here?

Thank you very much,
Andrej

 

[dlavigne]

What virtualization software are you using? You can't use files for disks, you'll need to create virtual disks. Also, don't use the command line, use the GUI so FreeNAS is aware of what you are doing.

 

[Robert Trevellyan]

rm /path/disk2

My explanation is along the lines of, "you released one reference to the file, but ZFS was still using it, so the filesystem kept it alive."

 

[SweetAndLow]

Files don't usually get deleted until all links to them are gone. So your virtualization software still had the file open so the OS you are on didn't delete the file just removed the link you can see in the terminal.

 

[brumnas]

Thank you guys -

@dlavigne: I'm under VMWare player 7.1.3 (windoze box).
1. Why can not I use files for ZFS learning? Does it behave in usage/learning different than when used later with real disks?
2. Why can not I use it from command line? When I troubleshoot later on, I'm surely in the terminal and not in GUI. TBH, I didn't even start the GUI in this moment ;-) (have to find out, which port does it listen on etc.) I want to move all my stuff to the ZFS, that's why I'm after the FreeNAS and that's why I need to dive to ZFS before relying on it. Does the FreeNAS later on in GUI stable realize when I create/import a pool from command line? Or do I have to click-click always somewhere?

@robert: + @SweetAndLow: That's something I need to understand better: you mean that I "rm" a file and the FreeBSD under FreeNAS doesn't remove it _really_ but only some kind of "visual effect", like not showing in "ls"? I've double checked with "lsof" and can't find any reference onto that file - so I would say the file is really gone and ZFS didn't realize this. But I've never used FreeBSD before so may be I'm wrong of course.

@SweetAndLow The virtualization SW has no idea about the internals going on under the emulated OS. If FreeBSD doesn't delete the file until nobody references it, it will behave like this also on bare-metal. So the point for me is: does FreeBSD really keep that file, although I can not find it with "lsof"?

Thx,
Andrej

 

[SweetAndLow]

If you open a file with a text editor then in a different terminal delete that file your text editor will still work and have the file open. This is how all *nix based systems work.


Also freenas is meant to be used from the gui if you find yourself at the command line you better only use it for commands that are read only and don't make any modifications. Have you read the manual?

 

[brumnas]

If you open a file with a text editor then in a different terminal delete that file your text editor will still work and have the file open. This is how all *nix based systems work.

Ah, now I see what you mean :)! But these are totally different things:

1. Keeping the file itself
- This means: the file is still registered in the FS, e.g. blocks are reserved and any referencing processes listed with "lsof"
- After "rm" is successfully exited, the file is _gone_, from FS

2. Keeping a copy in memory
- This means: the file is may be removed from FS already, but the process which holds a full copy of it in RAM, can rewrite it again to the FS if needed

I guess we were talking about the 1. scenario, e.g. when I remove a file from FS, it's gone - and I guess ZFS won't recreate it just because the user trashed it from the FS. And ZFS shouldn't think "ok, the user trashed the base file, but I have a copy in RAM still, so show it ONLINE". ZFS should not behave as "vi" I hope! That's why I was wondering what is going on.

Also freenas is meant to be used from the gui if you find yourself at the command line you better only use it for commands that are read only and don't make any modifications. Have you read the manual?

TBH, I didn't read the manual step by step yet ;-). I surely find the UI settings when needed later on. But I'm reading very cautious the most critical/important stuff, which I didn't know so far, e.g. ZFS. The rest is pretty much the same everywhere/everytime and can be found when needed. The very special for me was the HW selection (was reading/planing almost 1 year now!), ZFS and all those ZFS nuances, like resilvering an encrypted pool under FreeBSD vs Solaris: those things I've read of course already! And am reading these forums, with all those insiders/troubleshooters, pointing you such deep as to the C++ sources of ZFS, which I do read as well to get better idea.

..hehe, would be wondering a bit if my command line got read-only one day :-D!

Thx,
Andrej

 

[brumnas]

I can confirm, that in a lab/test scenario ZFS ignores the underlying file removal if simulated under the OS itself (like created with 'dd') and notices its removal only after reboot/onlineOffline and similar.

But when the pools is created with virtual disks simulated as real-disks by the VM-host itself (e.g. .vmdk files), ZFS realizes it immediately as expected.

May be this helps another ZFS noob as me when trying to understand the

Bye,
Andrej

 

[SweetAndLow]

Ah, now I see what you mean :)! But these are totally different things:

1. Keeping the file itself
- This means: the file is still registered in the FS, e.g. blocks are reserved and any referencing processes listed with "lsof"
- After "rm" is successfully exited, the file is _gone_, from FS

2. Keeping a copy in memory
- This means: the file is may be removed from FS already, but the process which holds a full copy of it in RAM, can rewrite it again to the FS if needed

I guess we were talking about the 1. scenario, e.g. when I remove a file from FS, it's gone - and I guess ZFS won't recreate it just because the user trashed it from the FS. And ZFS shouldn't think "ok, the user trashed the base file, but I have a copy in RAM still, so show it ONLINE". ZFS should not behave as "vi" I hope! That's why I was wondering what is going on.



TBH, I didn't read the manual step by step yet ;-). I surely find the UI settings when needed later on. But I'm reading very cautious the most critical/important stuff, which I didn't know so far, e.g. ZFS. The rest is pretty much the same everywhere/everytime and can be found when needed. The very special for me was the HW selection (was reading/planing almost 1 year now!), ZFS and all those ZFS nuances, like resilvering an encrypted pool under FreeBSD vs Solaris: those things I've read of course already! And am reading these forums, with all those insiders/troubleshooters, pointing you such deep as to the C++ sources of ZFS, which I do read as well to get better idea.

..hehe, would be wondering a bit if my command line got read-only one day :-D!

Thx,
Andrej

I think you are going down the wrong path and making way to many assumptions. Read the manual, read the noob slideshow and use the gui.

 

[depasseg]

1. Why can not I use files for ZFS learning? Does it behave in usage/learning different than when used later with real disks?

You can, but you need to understand the limitations/differences that a virtual environment brings. Your OP brings in to question whether you have that grasp or not.

2. Why can not I use it from command line?

You can, but realize that changes made at the CLI are done behind the scenes of FreeNAS (and it will have no awareness of those changes), so you have a very high potential for breakage if you don't know what you are doing.

 

[brumnas]

You can, but realize that changes made at thte CLI are done behind the scenes of FreeNAS (and it will have no awareness of those changes), so you have a very high potential for breakage if you don't know what you are doing.

Thank you; this sounds interesting - do you know what concrete use cases is FreeNAS GUI implementing "itself", like persisting configs outside of the FreeBSD ecosystem? I don't care if jails and configs are screwed up after a troubleshooting - but I do care about ZFS with a few TB of my projects/data. I won't rely on a system which I can't drive from command line, that's for sure ;-).

 

[depasseg]

If you make any change to ZFS (datasets, pools, spares, vdevs) using the cli, freenas won't have any idea until you export and import the pool. There are other configuration areas (shares, services, startup, etc) that all come from freenas. When disks get formatted to be added to a vdev, freenas does it a certain way.

It's one thing to need to use the CLI for troubleshooting, it's another to completely ignore the GUI. I'd suggest you either try a plain FreeBSD distro, use the FreeNAS GUI or wait until FreeNAS 10 (great CLI).

 

[SweetAndLow]

Thank you; this sounds interesting - do you know what concrete use cases is FreeNAS GUI implementing "itself", like persisting configs outside of the FreeBSD ecosystem? I don't care if jails and configs are screwed up after a troubleshooting - but I do care about ZFS with a few TB of my projects/data. I won't rely on a system which I can't drive from command line, that's for sure ;-).

Why are you even bothering with freenas? Install freebsd if you don't want to use the gui.