iscsi volumes turned raw

[Lorenzo Vittorio]

Hello guys. I have a Freenas V 9.3 installed on a USB pen drive; and pendrive corrupted. It had 2 disks, zfs mirrored with 3 iscsi volumes and 2 datasets. I reinstalled on a different pendrive and imported the volumes but the iscsi volumes are all raw. Why is this and what can I do?

Why is freenas so brittle? I've only been using freenas for 1 year, before I used nas4free for nearly 3 years, and I never had such a disaster as this.

 

[mav@]

What do you mean "all raw"? There is no information critical for iSCSI data stored on boot drive, only general configuration: what zvols/files export as LUNs, what LUNs serial numbers, etc. After you imported old pool you should recreate old iSCSI extents, and everything should just work. LUNs will likely change their serial numbers and other IDs, so initiator may need some kicking to detect the change, but the data should be intact.

"brittle" here I suppose is from really crappy quality of modern consumer USB sticks. That is why FreeNAS 9.3 supports boot device mirroring. So next time take couple of USB sticks (may be even from different vendors), or even small SSD(s) and be safe. Configuration backup after all significant changes would also not hurt.

 

[Lorenzo Vittorio]

First off, I take regular config backups and my first step was to install Freenas on a new USB stick and restore config. All went well, except iSCSI volumes where raw; the dataset data remained intact - hence the word "brittle". Why should a simple error result in such a catastrophic data loss? I have lost a lot of faith in this product because I use it to prevent such disasters in the first place!

"all raw" means that when when viewing drive from windows, the drive results as unformulated and "raw"

 

[mav@]

I don't see _any_ reason for that to happen. If you imported existing FreeNAS configuration, then Windows should not see any difference at all.

 

[Lorenzo Vittorio]

I agree. I do not wish to complain - this happened to me before with NAs4free twice due to hardware failures - no data loss whatsoever. First time with Freenas. I am not a newbie, I have 22 different Freenas deployed now and if product is brittle or faulty I need to know. I swapped to freenas due to performance issues; but if loosing data is freenas's thing, I will have to find something else.

 

[mav@]

OK, so to not just a complain, lets make it more constructive and try to understand the situation deeper. I am not ready to accept any defeat without understanding why it happened.

What type of extent are you using: file or zvol? Are they still exist and occupy some expected space (for files you may use `du -h` command, for zvol -- `zfs list`)? Are there any complains on FreeNAS boot from CTL about inability to open them, or something else? Have you done anything at all on Windows side? Have you seen any issues before you replaced the stick?

 

[Lorenzo Vittorio]

they are zvols and yes they occupy the space they should be. No error show up on boot and windows connects without issues - except that according to windows they are unformatted.

UPDATE since I started this post.
I have decided to use a recovery software on volume, and files are being found though there are far less than there should be. Recovery will take all day so I do not know what these files are and if they are corrupt or not till done.

Regarding to windows; same windows has 2 other freenas server attached to it, which are operating normally. And no, there was no changes either to windows or this particular nas

 

[Darren Myers]

You didnt post any hardware specs, that could of helped as well. As far as data is concerned, when you imported your config and you checked the services, did you check to make sure the iscsi target settings? I dont think i would of jumped into recovery i might have tried to check disk the iscsi connected drive, there have been times where a check disk clears up whatever error occurred and caused it to be seen as "raw"

 

[Lorenzo Vittorio]

I didn't think the specs are or were relevant and I can upload the config of said freenas if need be. In any case specs are as follows

ASUS Motherboard, 16GB DDR3 RAM, 300W PSU, 2 xWD SE 4 TB 7200 RPM 3.5-inch SATA-600 Intel Core i3-3220

regarding chkdsk; to my knowledge, chkdsk does not work on unformatted drives. To answer your question regarding iscsi parameters; on new install I tried all different block settings (512/1024/2048/4096)
Having said that, a restore of config would set the settings exactly as they were - and I already did that - and drive still looked unformatted.

This should not have happened. And if there is an underlying instability issue people should be warned.

 

[pirateghost]

This should not have happened. And if there is an underlying instability issue people should be warned.

You're right, it should not have happened. Did you file a bug report? If you didn't, how do you expect the developers to know?

 

[Darren Myers]

So ASUS generic board, 16GB DDR3 i assume no ECC. It was checkdisk that failed me, but i was able to use some tools from EASUS, and i have used there partition master software as well. I was able to not only recover a harddrive that was RAW but it corrected whatever was corrupt the drive became functional again

 

[Lorenzo Vittorio]

So far I do not know it is a bug or not. I am waiting to see if others have had same issue.

 

[pirateghost]

So far I do not know it is a bug or not. I am waiting to see if others have had same issue.

With your many deployments, I would think you have a system or 2 around that you could test with and see if you can duplicate it.

 

[Lorenzo Vittorio]

My priority is getting the data back for now. In any case I have reported it as a bug.

 

[mav@]

Knowing how data corrupted may give clue why they were corrupted. Please report any information you may get. Because otherwise as one of the new iSCSI stack developers I have no idea what to do with this case, with bug report or without. :(

 

[Lorenzo Vittorio]

Had you hit me with a brick, you would have hurt me less. I have no idea what happened, I have always believed this was the safest way to have data stored. I was obviously wrong.

 

[mav@]

I see nothing really dramatic in my words, there are no gods in developres, only people working hard. And that is how it works: first we diagnose the problem, then we fix it. Diagnose in best case starts from the issue reproduction. But so far you are the only on my memory with such problem, so I doubt that it is easily in reproduction. If reproduction, and so debugging, is not possible, the only thing that remains is data collection, hoping that it give us some ideas.

From the data safety perspective I would recommend you to use regular snapshots on your FreeNAS systems. That is a basic feature of ZFS, it is very cheap, and in worst case of data corruption by initiator or iSCSI server you would just revert to the last working snapshot. Backups of the most critical data are also important.

 

[Lorenzo Vittorio]

As a developer myself I am quite aware of the sentiment; regarding snapshots I have held back as it is a "new" feature for me which means it is untested. I only deploy solutions after I have tried and tested them at least 12 months, which is why I am in such shock now regarding this particular disaster. regardless, i doubt the snapshot feature would have helped in this case.

In all my systems I have a redundant NAS which mirrors or backs up via OS scripts. This happens to be one of the few zvol's which is not mirrored or backed up automatically as data was slowly being archived to BD drives. Unfortunately process was still ongoing and much of the data remains unwritten.

 

[mav@]

i doubt the snapshot feature would have helped in this case.

Let me disagree. ZFS has internal checksums of everything. Unless the system has faulty (non-ECC) RAM that corrupts the data, there is almost impossible to corrupt the data in a way that would not be at least discovered by ZFS, but also quite probably fixed from redundancy. So the most probable source of data corruption there I see in incorrect behavior of something outsize ZFS, either on server or client. And from such kind of errors snapshots are the perfect protection. ZFS never overwrites previously written data that are covered by any snapshot. As I understand, your pool is healthy after FreeNAS reinstall (at least you haven't mentioned opposite). In such case, even if Windows would tell server to erase everything, you would still have full intact copy of your data from the moment of last snapshot. There are only two ways to corrupt snapshoted data: heavily corrupt pool on physical layer, or corrupt data in a way not detected until the last correct snapshot is rotated.

Once you are already doing some BD backups, it would be perfect to integrate them with snapshots, so that snapshot would be deleted only after respective data are backed up.

 

[jgreco]

Just playin' the devil's advocate here...

Over the years, we've had some users show up with similar issues, and I have actually seen this firsthand as well. It appeared to me at the time to be likely related to the serial number (etc) changing and VMware didn't seem to like that at all.

That behaviour doesn't seem to be limited to FreeNAS. Seen it with other stuff like EMC LifeLine as well.

Snapshots may not be useful in this case, because it is the client that is being obstinate, and also the client that is managing the filesystem layer.