New Install Lost Config with no Backups

[jeebs01]

Hello,

New to FreeNAS. Transitioned over from Openfiler when I read into some benefits for iSCSI sharing with the BSD platform. Using this system in a ESXi lab/home file storage. I have searched around and tried a few things, but I think I have been led down a very bad path... Please prove me wrong.

So, I was transitioning between my old Openfiler system to a newly built FreeNAS 8.0.4 p3 x64 (11703). I moved my backup drives, 2x 1TB, over to the FreeNAS box. Setup a device extent and presented drive A to ESXi and drive B to my domain controller. Both drives were device extents over iSCSI. I then vMotioned all my VM's to drive A (replaceable with many hours of effort), and all my personal data (family videos/photos........) to drive B. I run the box for several days during the slow data migration, rebooting here and there to verify stability. All looks well at this point.

Today, I moved the primary drives with their RAID controller (8x 2TB storage and 8x 500GB ESXi via a Highpoint 4320 & and Intel SAS/SATA Expander). Everything installed and detected in the POST's just fine. After verifying the RAID configuration, POST settings, etc, I couldn't boot to my 8GB USB drive FreeNAS any more. I tried a few things in BIOS (This is not my first box I have built) with no luck on the auto selection. I recalled a "upgrade" process in the installer - where it detects a system is already there, upgrades the distribution, and updates the configuration file. No luck there. I trouble shot with another bootable USB drive without success. I was finally able to boot manually with a manual device selection (likely a BIOS upgrade to the Z8PE-D12, least of my concerns at this point in the day).

When the install came up, there was no configuration. I do not have a backup of the FreeNAS configuration to restore as it was a fresh install on new hardware.

I looked around, found some information about device extents - from what I read, the process looked to present the disk to the target, which handled all formatting, etc. Made sense. So I try to create a new device extent and present to ESXi. The disk is blank. I tried passing iSCSI to my Windows 7 box to see if I can see anything. The disk is requesting to be initialized. As fyi, I never formatted or initialized.

I tried looking into the zpool import -D command. No Luck. Nothing found.

At this point, I am about sick. Is there anyone out there that can lend a helping hand here? Sorry for the length, just trying to provide details.

Thanks,

- Stressed over family photos/videos

 

[ProtoSD]

I can't help you at all with the iSCSI stuff, but if you haven't reformatted the original USB disk, there is a chance you could boot from a FreeBSD rescue CD or even another FreeNAS disk and grab the /data/freenas-v1.db file which contains all of your settings, and/or any other .db files in that folder. This might be a place to start. I understand the stress you're feeling, but take things cautiously one step at a time and don't be afraid to ask for help.

I'm away on vacation, but hopefully that'll get you started and someone else can jump in. I'll keep my fingers crossed.

 

[paleoN]

When the install came up, there was no configuration. I do not have a backup of the FreeNAS configuration to restore as it was a fresh install on new hardware.

I looked around, found some information about device extents - from what I read, the process looked to present the disk to the target, which handled all formatting, etc. Made sense. So I try to create a new device extent and present to ESXi. The disk is blank. I tried passing iSCSI to my Windows 7 box to see if I can see anything. The disk is requesting to be initialized. As fyi, I never formatted or initialized.

I tried looking into the zpool import -D command. No Luck. Nothing found.

I haven't messed with iSCSI that much myself, but zvols are saved with your pool configuration. ZFS device extents are not using a physical disk as they are above the ZFS layer & not below it.

Have you tried to auto-import your pool? Your pool was never destroyed and so it wouldn't be listed with zpool import -D.

- Stressed over family photos/videos

Isn't this all on the primary drives that go with their RAID controller? Wouldn't you simply need to move everything back?

 

[jeebs01]

Thanks for the replies.

I did try an auto-import without success. It did not detect any volumes. Is the auto import for device extents too or just if I was to create a NFS/CIFS/etc share?

As for moving the primary disks back to the previous system, I did attempt that briefly last night with the bigger disks, however OpenFiler was not seeing the previous information (issued a blkid and lvscan). While I have not completely explored this path, at first glance it does look like Openfiler is not seeing the volume. As I am typing this, I recalled that I did not verify the drive was mounted via the CLI. With a clear head, I plan to open the two systems again today to attempt again.

I have pulled the original thumb drive out and will also see if I can recover the database in the meantime.

Other suggestions are welcome. Clearly I took a wrong turn somewhere along the way.

Thanks again.

 

[jeebs01]

Hello,

Just to follow up, I was looking within the import and saw the existing volumes located there. I am certainly treading carefully here due to the situation, but if I were to import, would that erase the data contents if I where to choose the incorrect filesystem?

Please recall that both my 1TB drives have different formatting done (vmfs on one and ntfs on the other) due to the device extent. I am not sure which drive has which formatting.



I also will be working on some tests in VMware Workstation to test theories out as well. I just wanted to keep the thread up to date with my thought processes.

Thanks.

 

[jeebs01]

So, after a little break and some family time, I was able to put some more thoughts to the issue.

I have loaded a FreeNAS installation in VMware Workstation (Ubuntu Host) and configured two 20GB data disks. The OS was put an 8GB disk. I shared them via iSCSI as a device extent over to a Windows 7 VM on the same box. I was able to create everything successfully and copied a few junk Cisco PDF Guides to put some data on the disks.

I then shutdown and deleted the 8GB disk, simulating a failure without a backup. A new disk (10GB just to verify I was working of a new disk) was created and a fresh install was completed.

Upon startup, I reconfigured one of the disks as a device extent and shared it via iSCSI. The Windows 7 VM refreshed the iSCSI target and pulled the share in successfully. It was determined by windows to be an uninitialized disk. At this point, this test has behaved the same way as my production data. I initialized the disk for the heck of it, just to confirm that it would indeed present a blank disk. It did. After initalization/reformatting, I of course had a blank disk to work with.

The second disk I attempted to import. There was nothing selected as an Auto Import, so I attempted a manual import. I gave it a generic name and selected import. There was an error message "An error occurred while labeling the disk" and the device did not import. On the console, an error message was generated "GEOM: label/extent_da1: partition 1 does not start on a track boundary." There was a similar message that stated that the partition 1 does not end on a track boundary. Nothing was imported.

I found this link (http://forums.freenas.org/archive/index.php/t-1353.html ) suggesting to import via cli. Even though it was for an earlier version, I gave it a try. It errored out and requested an attempt as RO. While the read only command was successful, when I browsed to the mount point there was no data to be seen.

I plan to keep working on this, however I am starting to loose hope. This test has generated to two specific question though:

- is anyone able to confirm if pre-existing data is retained when a device extent is created? It would appear that it is not from my test and production examples, but I do not see anything in documentation stating otherwise. The entire idea of the device extent is to present the disk in a raw format (i.e. SAN-like experience) via the iSCSI protocol. In that case, the initiator handles the formatting and partitioning of the disk. If it does erase the contents, why is there not red warning text on the button like in other locations of the FreeNAS OS?

- What would be the proper way to recover a device extent? I would think you just need to recreate the device extent, point it to the correct client and you are in business. i.e. SAN-like. Again, I was not able to find anything one way or another in the lines of documentation. (maybe my google-fu is bad today...)

Thanks to anyone who has been following along and to anyone who can offer suggestions.

 

[jeebs01]

So, had an idea to test the pre-existing data theory. Within my test VM, both test disks started with a NTFS partition. I took both of the disks and added them as as individual device extents. One disk had a NTFS re-created on it, one was left as it was. I fired up gparted v0.12.0 and was able to see the NTFS partition on one disk and nothing on the other. I attempted a recovery without luck. It does appear that adding the device extent does blank the disk. I just wanted to add the update.

Thanks.

- Feeling hopeless

 

[survive]

Hi jeebs01,

At this point I would stop poking at things and wait for one of the devs to respond. Take some notes and join the #freenas irc channel and ask there tomorrow when there is a better likelihood of one of the pro being around.

It seems we all will need confirmation about how iscsi device extents work when this sort of failure occurs.

-Will

 

[jeebs01]

Hello,

Thanks for the suggestion - I certainly agree about treading carefully with the production disks. I have been playing within the test VM environment and have been able to successfully rebuild the ntfs drives twice (tested it a second time/screen recorded just to be sure it wasn't random!).

Within the GParted live cd is a utility called TestDisk that will successfully detect existing partitions when the MBR/partition tables are cleared. I did successfully follow the steps I took in the two tests and recovered my NTFS partition. I deleted some of my unnessecary work VMware Workstation backups within my "work" backup disk to make room for another copy of my family videos. 160GB is copying as I type this. I was able to successfully open a few images on the location. (Hoping this is able to continue!)

For folks looking at this thread in the future, my test environment worked great to provide a proof of concept. I was able to use the GParted live CD v0.12.0.5 to test a recovery process. From there, I moved to my production disk and was able to run the TestDisk utility there. There was one hang-up with the production disks. I attempted to plug it into a Windows 7 laptop to read the disk. It saw there was a partition on the disk, but wanted still to format it. I plugged the disk into my Ubuntu laptop and was able to read the disc successfully.

As for the next step, I plan on running the same process against my 2nd 1TB drive that houses my 30 or so lab VM's. From research online, it appears that the TestDisk utility has worked for VMware folks, but it takes a decent amount of time to scan (couple hours for my size of disk). I will certainly follow up with the outcome later.

I do look forward to hearing from some of the experts the details around the device extent and will stay subscribed to the thread.

Thanks for all the reads thusfar!

-About done with this emotional roller-coaster

 

[paleoN]

Upon startup, I reconfigured one of the disks as a device extent and shared it via iSCSI. The Windows 7 VM refreshed the iSCSI target and pulled the share in successfully. It was determined by windows to be an uninitialized disk. At this point, this test has behaved the same way as my production data. I initialized the disk for the heck of it, just to confirm that it would indeed present a blank disk. It did. After initalization/reformatting, I of course had a blank disk to work with.

You are not understanding how FN8 is using device extents on ZFS. It does not use the drive directly. If you connect the drive directly you will not see any data. A device extent, zvol, on ZFS is a virtual device that uses a ZFS record size of 8k on it's own separate ZFS dataset.

The second disk I attempted to import. There was nothing selected as an Auto Import, so I attempted a manual import.

You need to Auto Import the entire zpool not just one of the drives.

The entire idea of the device extent is to present the disk in a raw format (i.e. SAN-like experience) via the iSCSI protocol.

Wrong with ZFS see above.

What would be the proper way to recover a device extent? I would think you just need to recreate the device extent, point it to the correct client and you are in business. i.e. SAN-like.

No, do not recreate the device extent. You need to get the zpool imported properly. The device extents are stored in the zpool on top of ZFS.

 

[jeebs01]

You are not understanding how FN8 is using device extents on ZFS.

I agree 100%. In my searches, I wasn't able to find anything descriptive. I will attempt again this evening as I still want to use FN8 for my environment.

You need to Auto Import the entire zpool not just one of the drives.

I will be using my proof of concept environment to get a better handle on a process. There was no selection available within the Auto Import. I will also plan to look into some zpool documentation.

As far as presenting the disk in a raw format vs. some of the ZFS techs (Please keep in mind I still have to do some reading...), something didn't quite add up in my proof of concept. To explain, I was able to create a device extent and share it to iSCSI to Windows 7 VM. I then wrote some random files to the share. I then shutdown the FN8 VM and connected the vmdk's to the Windows 7 VM. Upon loading, I was able to read and connect the files. Windows detected a native primary NTFS partition. It was only when I loaded those disks into a new FN8 installation and created a new device extent that the partition information was cleared.

I am only pointing out what I saw in my tests and I clearly have some reading to do so I can further understand some of the FN8 technologies.

FWIW, OpenFiler had its own difficulties in recovery scenarios... I still am planning on running FreeNAS on my new configuration.

Thanks!

 

[jeebs01]

Hello,

So, taking a look at a few things, I think we were looking at two different ways to present a file extent. I had created a iSCSI device extent with no other steps other than creating the extent under the iSCSI menu. The zvol steps that were referenced included creating an underlying zfs volume. During my research, I did find that using a zfs file extent would be the proper way to go so I can utilize my 24GB of RAM as cache per my design. (Openfiler used the memory in this manner too)

While testing the configuration, there were several questions that came up, but the most pressing was around how I am allowed to allocate more space within iSCSI -> Extent -> Add than what is actually on the disk (haven't really searched yet). I would imagine that you could select "remaining available" or something similar. Otherwise, there will be space unused on the disk. I will most likely end up posting a few other types of questions under the n00b section tomorrow.

Thanks for the ZFS Ninja videos link, survive. I will have to take a watch tomorrow.

Thanks!