Rvby1
Cadet
- Joined
- Jan 17, 2024
- Messages
- 8
This issue started in an Ubuntu Server VM, but I'm hoping (praying) that I can fix it using some tools in TrueNAS.
I was running a Mirror pool on an Ubuntu Server VM, which itself was running inside of Proxmox. The two 2TB drives were passed through to the VM, and it had them set up in a singular Mirror pool with one dataset inside of it. The drives were both Seagate Barracuda Compute drives.
In the last month or so, I started getting close to filling the pool, so I decided to invest in creating a separate NAS. Got that configured with 2x18TB drives in a Mirror pool, all configured and running through TrueNAS Scale. These were Seagate EXOS drives.
After some configuration and testing, I felt that I was ready to move my data from the Ubuntu VM ZFS Pool over to the TrueNAS ZFS Pool. I had the TrueNAS ZFS pool mounted using cifs on my Ubuntu VM, and I was able to write and read from it without any issues.
I decided to use rsync, as I thought it would copy over the hardlinks my Ubuntu ZFS pool was using. The command was something like... rsync -avH --update --progress. This command worked fine in my testing, and it copied over data without any issues.
When I eventually tried to run it on my Ubuntu's media-vault to copy it over to my TrueNAS vault, everything was working fine for a few thousand files. But then I noticed that my SSH terminal connection had hung, and when I went to check on the VM in Proxmox, it had an IO error status (little yellow triangle). I didn't understand why until digging deeper.
Through some stupidity, I misconfigured the LVM-thin and ended up over-allocating. Something during the rsync process must have written out data to the local machine's LV, which then caused it to fill up the entire LVM-thin. This then starved out all of the VMs on my machine and caused them to start failing.
I was able to resolve the starvation issues by backing up one of my VMs over to my NAS, then deleting it. This let me reboot into my Ubuntu VM, but I immediately noticed that there was some corruption. Some of my folder permissions were changed, and almost all of the data on my Ubuntu's ZFS mirror pool was gone, save for a few random files.
I went from almost 2TB, down to a couple of gigs.
I disconnected the ZFS drives without doing any sort of export and tried to connect them to my TrueNAS system in the hopes that, maybe, the data was there, but just blocked by corruption. After connecting the drives to my mother board's SATA connectors and booting, TrueNAS spit out the following error to the notifications section:
And when I tried to just import it, this is the error that I got: https://pastebin.com/CuxyfgXX
Now, some of the data is on my NAS, but the majority of it is not. I'd like to recover this data if I can, as reobtaining a lot of it would be very time consuming.
I'm currently running a scan with UFSExplorer in the hopes it can find something/fix whatever corruption happened, if that is indeed the case.
Anyone know if there is a way to get this pool to import correctly into TrueNAS Scale in the hopes that the data is there, and the Ubuntu VM is just messed up?
Anyone have any other advice? Programs I should try? Steps I should take? Anything would be appreciated!
Thanks!
I was running a Mirror pool on an Ubuntu Server VM, which itself was running inside of Proxmox. The two 2TB drives were passed through to the VM, and it had them set up in a singular Mirror pool with one dataset inside of it. The drives were both Seagate Barracuda Compute drives.
In the last month or so, I started getting close to filling the pool, so I decided to invest in creating a separate NAS. Got that configured with 2x18TB drives in a Mirror pool, all configured and running through TrueNAS Scale. These were Seagate EXOS drives.
After some configuration and testing, I felt that I was ready to move my data from the Ubuntu VM ZFS Pool over to the TrueNAS ZFS Pool. I had the TrueNAS ZFS pool mounted using cifs on my Ubuntu VM, and I was able to write and read from it without any issues.
I decided to use rsync, as I thought it would copy over the hardlinks my Ubuntu ZFS pool was using. The command was something like... rsync -avH --update --progress. This command worked fine in my testing, and it copied over data without any issues.
When I eventually tried to run it on my Ubuntu's media-vault to copy it over to my TrueNAS vault, everything was working fine for a few thousand files. But then I noticed that my SSH terminal connection had hung, and when I went to check on the VM in Proxmox, it had an IO error status (little yellow triangle). I didn't understand why until digging deeper.
Through some stupidity, I misconfigured the LVM-thin and ended up over-allocating. Something during the rsync process must have written out data to the local machine's LV, which then caused it to fill up the entire LVM-thin. This then starved out all of the VMs on my machine and caused them to start failing.
I was able to resolve the starvation issues by backing up one of my VMs over to my NAS, then deleting it. This let me reboot into my Ubuntu VM, but I immediately noticed that there was some corruption. Some of my folder permissions were changed, and almost all of the data on my Ubuntu's ZFS mirror pool was gone, save for a few random files.
I went from almost 2TB, down to a couple of gigs.
I disconnected the ZFS drives without doing any sort of export and tried to connect them to my TrueNAS system in the hopes that, maybe, the data was there, but just blocked by corruption. After connecting the drives to my mother board's SATA connectors and booting, TrueNAS spit out the following error to the notifications section:
Code:
Device: /dev/sde [SAT], 5 Offline uncorrectable sectors. 2024-01-17 01:02:57 (America/Los_Angeles)
And when I tried to just import it, this is the error that I got: https://pastebin.com/CuxyfgXX
Now, some of the data is on my NAS, but the majority of it is not. I'd like to recover this data if I can, as reobtaining a lot of it would be very time consuming.
I'm currently running a scan with UFSExplorer in the hopes it can find something/fix whatever corruption happened, if that is indeed the case.
Anyone know if there is a way to get this pool to import correctly into TrueNAS Scale in the hopes that the data is there, and the Ubuntu VM is just messed up?
Anyone have any other advice? Programs I should try? Steps I should take? Anything would be appreciated!
Thanks!