Loss of data

[carleycr]

About a week ago I noticed my Plex movie count was cut in half. I'm not sure why, I'm not sure where they went and I have no idea how to even troubleshoot. I restarted the server, the plugin, and the jail. I've had SMART read errors for a while now, but from I've been able to conclude is that it's my motherboard SATA connector that just can't utilized SMART testing. I've tried updating my motherboard firmware with no luck. So i just ignore the errors FreeNAS show me.

My questions. How can I determine what happened to my movies? Where do I even begin? Any help appreciated.

 

[Chris Moore]

You can't ignore errors in FreeNAS. If a drive gets too many errors, it will be marked as defective and the File system will stop using it.
You tell what drives you have but not how is your storage pool defined.
From the limited information you have provided, nobody can help you.

 

[Chris Moore]

I've had SMART read errors for a while now, but from I've been able to conclude is that it's my motherboard SATA connector that just can't utilized SMART testing.

It should be able to and if it did work, then stopped working, it indicates that a component has failed. This gives the impression of the system board having failed. You might want to contact the manufacturer and see if they will replace it. If you are going to have a server, you really must pay more attention to it.

 

[carleycr]

Sorry for not providing enough info. I tinker with my server as a hobby and my life has been a bit busy this year. I've posted the errors I've received on this forum in the past and there have been known issues with the 2 Marvell ports on board. What do you need to know about my data structure? I know it's a ZFS configuration.

The board is out of warranty, so not possible to replace through manufacturer.

Previous thread: https://forums.freenas.org/index.php?threads/volume-degraded.56382/#post-402852

 

[Chris Moore]

Putting together some of the information in the screenshots you posted in the linked thread with the screenshot you posted this time, I would say that you have three failed drives in a pool that can only sustain two drive failures. That would take the storage pool offline and the data would be lost.
Do you have a backup?

 

[carleycr]

From the previous thread link, I ran short and long tests but they came back with no issues. How are you concluding I have failed drives? Can I run short or long tests to prove this?

Unfortunately, I do not have a backup.

 

[garm]

Well post the consol print of

Code:

 zpool list -v

 

[danb35]

...as well as the output of zpool status... In code tags, please.

 

[joeschmuck]

Looking at your previous thread you did not perform Long tests on all the drives. My request is to start over and perform a SMART Long test on all hard drives in your system that haven't had one completed in the last 24 hours, post the results of each, and then we start tracking the drives by serial number. I do agree that in the previous thread nothing stood out as a hard drive failure or even a communications failure.

Also, be very specific and please list exactly which hard drive (by serial number, last 4 digits are fine) is connected to which SATA port.

Are you still using FreeNAS 9.10.2U2 ? You might consider using a new boot device and install FreeNAS 11.0-U4 and give this a shot since it will have newer FreeBSD updates in it. I'm not saying it will fix the commu8nications issue if that is what you really have but it's worth a shot. Make sure you backup your configuration file from the current system first so you can import it into 11.0-U4. When you do this upgrade the ALERT will flash telling you that the ZFS pool should be upgraded to new feature flags, DO NOT DO IT or you will likely not be able to roll back to FreeNAS 9.10.2. Just disable the alert and move on.

Your Motherboard, does it have the most current BIOS?

Also, I don't recall if 9.10.2 had UEFI but just in case, are you running UEFI? If yes then try the BIOS/Legacy version. I'm trying to look at everything here, I've only got a few hours this morning befor I'm moving the daughter out of the house down the road 70 miles.

 

[Chris Moore]

How are you concluding I have failed drives?

The drives may not be failed but ZFS has probably offlined the drive for having too many errors even if those errors were generated by a fault in the system board. If the controller is suspect, which is what I got from the links you posted, then you would not want to attempt testing using the suspect controller. Connect the drives to a different controller to test or even test them using a whole different system board. If you can determine with certainty that your drives are not defective, it is possible that the data may still be there and you may even be able to get the pool available again. You must follow up on this as ignoring it is not going to get your data back.

 

[rs225]

If the pool is operating, perhaps there are snapshots that you could look at to see if your Plex data still exists in them?

 

[carleycr]

Well post the consol print of

Code:

 zpool list -v

Code:

[root@freenas] ~# zpool list -v

NAME									 SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT

Vol1									16.2T  9.05T  7.20T		 -	14%	55%  1.00x  ONLINE  /mnt

  raidz2								16.2T  9.05T  7.20T		 -	14%	55%

	gptid/1c24a040-0d3e-11e4-839a-d050991a7a7c	  -	  -	  -		 -	  -	  -

	gptid/1cf27d76-0d3e-11e4-839a-d050991a7a7c	  -	  -	  -		 -	  -	  -

	gptid/1dc3cca8-0d3e-11e4-839a-d050991a7a7c	  -	  -	  -		 -	  -	  -

	gptid/1e99ac80-0d3e-11e4-839a-d050991a7a7c	  -	  -	  -		 -	  -	  -

	gptid/1f6e3181-0d3e-11e4-839a-d050991a7a7c	  -	  -	  -		 -	  -	  -

	gptid/203f162f-0d3e-11e4-839a-d050991a7a7c	  -	  -	  -		 -	  -	  -

freenas-boot							14.9G  13.2G  1.70G		 -	  -	88%  1.00x  ONLINE  -

  gptid/c61e73ec-ae1f-11e4-9b5e-d050991a7a7c  14.9G  13.2G  1.70G		 -	  -	88%

...as well as the output of zpool status... In code tags, please.

Code:

[root@freenas] ~# zpool status

  pool: Vol1

 state: ONLINE

  scan: scrub repaired 0 in 19h28m with 0 errors on Fri Dec  1 20:29:00 2017

config:


NAME											STATE	 READ WRITE CKSUM

Vol1											ONLINE	   0	 0	 0

  raidz2-0									  ONLINE	   0	 0	 0

	gptid/1c24a040-0d3e-11e4-839a-d050991a7a7c  ONLINE	   0	 0	 0

	gptid/1cf27d76-0d3e-11e4-839a-d050991a7a7c  ONLINE	   0	 0	 0

	gptid/1dc3cca8-0d3e-11e4-839a-d050991a7a7c  ONLINE	   0	 0	 0

	gptid/1e99ac80-0d3e-11e4-839a-d050991a7a7c  ONLINE	   0	 0	 0

	gptid/1f6e3181-0d3e-11e4-839a-d050991a7a7c  ONLINE	   0	 0	 0

	gptid/203f162f-0d3e-11e4-839a-d050991a7a7c  ONLINE	   0	 0	 0


errors: No known data errors


  pool: freenas-boot

 state: ONLINE

  scan: scrub repaired 0 in 0h47m with 0 errors on Tue Nov 14 04:32:12 2017

config:


NAME										  STATE	 READ WRITE CKSUM

freenas-boot								  ONLINE	   0	 0	 0

  gptid/c61e73ec-ae1f-11e4-9b5e-d050991a7a7c  ONLINE	   0	 0	 0


errors: No known data errors

 

[carleycr]

Looking at your previous thread you did not perform Long tests on all the drives. My request is to start over and perform a SMART Long test on all hard drives in your system that haven't had one completed in the last 24 hours, post the results of each, and then we start tracking the drives by serial number. I do agree that in the previous thread nothing stood out as a hard drive failure or even a communications failure.

Also, be very specific and please list exactly which hard drive (by serial number, last 4 digits are fine) is connected to which SATA port.

Are you still using FreeNAS 9.10.2U2 ? You might consider using a new boot device and install FreeNAS 11.0-U4 and give this a shot since it will have newer FreeBSD updates in it. I'm not saying it will fix the commu8nications issue if that is what you really have but it's worth a shot. Make sure you backup your configuration file from the current system first so you can import it into 11.0-U4. When you do this upgrade the ALERT will flash telling you that the ZFS pool should be upgraded to new feature flags, DO NOT DO IT or you will likely not be able to roll back to FreeNAS 9.10.2. Just disable the alert and move on.

Your Motherboard, does it have the most current BIOS?

Also, I don't recall if 9.10.2 had UEFI but just in case, are you running UEFI? If yes then try the BIOS/Legacy version. I'm trying to look at everything here, I've only got a few hours this morning befor I'm moving the daughter out of the house down the road 70 miles.

Is it recommended to not use plugins such as Plex while running long tests? Just need to know if I should turn these off during testing today. How and where do I access the test results? I had short tests run on all the drives overnight last night, and I'd be interested to see those.

I updated my sig with the version of FreeNAS I'm currently using. My motherboard does have the most up to date BIOS. I know I need to update the Marvell firmware, but for the life of me, I can't get the board to boot to a DOS usb drive to update the firmware. If I switch the drives from the marvell controllers to another, slower, intel controller on the board, will that cause issues?

I'm not sure what i'm running regarding UEFI. I know my BIOS is running UEFI. Is that the same thing you're referencing?

Thanks for all the help everyone. Very much appreciated!

 

[joeschmuck]

First of all the good news, it looks like your pool is running. This is a good thing, no errors.

Is it recommended to not use plugins such as Plex while running long tests?

You can run Plex and even use your system. The Long test may take longer to complete but that is fine, it wouldn't be a lot of extra time.

I'm not sure what i'm running regarding UEFI. I know my BIOS is running UEFI. Is that the same thing you're referencing?

When you installed FreeNAS, was it the UEFI version or Legacy version?

I updated my sig with the version of FreeNAS I'm currently using. My motherboard does have the most up to date BIOS. I know I need to update the Marvell firmware, but for the life of me, I can't get the board to boot to a DOS usb drive to update the firmware.

In your BIOS you should have the ability to choose UEFI or Legacy BIOS or DUAL/BOTH. My advice is changing, I don't think you need to run another SMART Long test at this moment, I think you need to focus on two things... Upgrade your firmware and change from UEFI to BIOS booting. Read your user manual that came with your motherboard, it will be in the back of the manual under the BIOS setup, once you figure this out then you will be able to boot to DOS. Also, ensure you have a backup of your configuration file before you do anything at all. Play it safe is all I say.

God Speed.

 

[carleycr]

Thanks for the help @joeschmuck. I spent today replacing my 16gb usb drive with 9.10.2 with a larger 32gb and FreenNAS 11.0-U4. I did backup my config before and still have the old flash drive if needed. I've successfully installed 11.0-U4 and uploaded my config with no issues. When I installed FreeNAS this time, I chose legacy. I don't think this matters, but I was also able to update by BMC version in the IPMI web GUI.

As for my BIOS, I changed the boot to legacy, but didn't really notice any difference booting. I read my manual and it doesn't give any information about booting into a DOS environment. The only references it has is how you UEFI is easier and if you want to update BIOS or Firmware using DOS, to make a bootable drive. I don't have a windows PC in my household so I'm going to install Ubuntu on an old desktop I have that I don't use and try that.

 

[carleycr]

I have updated the marvell firmware. Worthy to note, that I did get the warning recommending I upgrade my Zpool. I did not do this per the advice from joeschmuck.

Anything other recommendations I can do to try and get my data back?

 

[joeschmuck]

Hum, your data should be there unless you deleted it, I see nothing wrong with your pool, it was all online and you have quite a bit of storage used up.

Lets see what you do have on your system, please post the output of df and lets see what data is there. Hopefully you are just not looking in the right area.

 

[carleycr]

Hum, your data should be there unless you deleted it, I see nothing wrong with your pool, it was all online and you have quite a bit of storage used up.

Lets see what you do have on your system, please post the output of df and lets see what data is there. Hopefully you are just not looking in the right area.

Code:

root@freenas:~ # df

Filesystem											 1K-blocks	   Used	  Avail Capacity  Mounted on

freenas-boot/ROOT/default							   29189723	 747901   28441822	 3%	/

devfs														  1		  1		  0   100%	/dev

tmpfs													  32768	   9668	  23100	30%	/etc

tmpfs													   4096		  8	   4088	 0%	/mnt

tmpfs												   11162820	 120064   11042756	 1%	/var

freenas-boot/grub									   28448255	   6433   28441822	 0%	/boot/grub

fdescfs														1		  1		  0   100%	/dev/fd

Vol1												  4777287242		527 4777286715	 0%	/mnt/Vol1

Vol1/Backups										  4777287034		319 4777286715	 0%	/mnt/Vol1/Backups

Vol1/Backups/time-machine							 5058498393  281211678 4777286715	 6%	/mnt/Vol1/Backups/time-machine

Vol1/DS1											  6689659865 1912373150 4777286715	29%	/mnt/Vol1/DS1

Vol1/jails											4777286954		239 4777286715	 0%	/mnt/Vol1/jails

Vol1/jails/.warden-template-pluginjail-9.3-x64		4778188201	 901486 4777286715	 0%	/mnt/Vol1/jails/.warden-template-pluginjail-9.3-x64

Vol1/jails/customplugin_1							 4806103825   28817110 4777286715	 1%	/mnt/Vol1/jails/customplugin_1

Vol1/media											9033762960 4256476245 4777286715	47%	/mnt/Vol1/media

Vol1/.system										  4777698672	 411957 4777286715	 0%	/var/db/system

Vol1/.system/cores									4777287698		983 4777286715	 0%	/var/db/system/cores

Vol1/.system/samba4								   4777287770	   1055 4777286715	 0%	/var/db/system/samba4

Vol1/.system/syslog-ffb84ccc300c4843b1352f93d2beb43e  4777288921	   2206 4777286715	 0%	/var/db/system/syslog-ffb84ccc300c4843b1352f93d2beb43e

Vol1/.system/rrd-ffb84ccc300c4843b1352f93d2beb43e	 4777286906		191 4777286715	 0%	/var/db/system/rrd-ffb84ccc300c4843b1352f93d2beb43e

Vol1/.system/configs-ffb84ccc300c4843b1352f93d2beb43e 4777372767	  86052 4777286715	 0%	/var/db/system/configs-ffb84ccc300c4843b1352f93d2beb43e

devfs														  1		  1		  0   100%	/mnt/Vol1/jails/customplugin_1/dev

procfs														 4		  4		  0   100%	/mnt/Vol1/jails/customplugin_1/proc

/mnt/Vol1/DS1										 6689659865 1912373150 4777286715	29%	/mnt/Vol1/jails/customplugin_1/mnt/media

/mnt/Vol1/media									   9033762960 4256476245 4777286715	47%	/mnt/Vol1/jails/customplugin_1/mnt/media

 

[joeschmuck]

So you have 4.7TB of data in /mnt/Vol1/media. Have you looked in there to ensure you have all the media you expect to have or do you feel some of it's missing?

 

[carleycr]

I did check that file location. I noticed some of it missing about a week or two ago which sparked this thread. I used to have approximately 1300 movies. Now I have just over 500. I noticed when I went to go look for a movie I know I had, but no longer showed up in my plex. I have tvshows in that same location, but I haven't lost any of those.

To confirm I'm not going crazy, I have an external hard drive with just over 1000 movies from before I built my server in 2014 and a lot of movies on there are not on my server. I checked where my plex is pointing for movies and it's correct. I added another location and re-scaned in plex, but no luck. The default location it's trying to pull from is the /var folder within the jail. I'm wondering if the Plex folder locations is causing this?

P.S. When I try to move a movie from the external hard drive onto the server, it says the file already exists, then throws an error saying I don't have access to the movie. How is the file there, but I don't see it? Is there a command to list all the files in my media folder via terminal? Then I suppose I would need to pull the permissions for some of these to see if that aligns.