ZPOOL corrupt

[Alexandre Trevizoli]

I know that it's a topic already very discuted here, but after read and read and read a lot of threads and zfs documents my brain can't resolve what I have to do....
I have two server with same errors, memory are ECC, so I believe that is not a memory problem, but this errors appears after energy breaks. (they have UPS but this not helped after a lighting event).

So I know that here in this community have very expert on Freenas and ZFS so I would like to do a path to follow to try resolve it.
This is a Storage used on a Surveillance Central of a City (24x7), so I can't have way to stop it, to make backup, destroy pool, create pool, recover backup. :(
I'd like to sorry for my poor english, I'm brazilian.
I hope someone can help me, so thanks in advance!

[root@maxstorage] /# zpool status -xv
pool: RECORDING
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 47h19m with 2 errors on Mon Dec 7 23:19:58 2015
config:

NAME STATE READ WRITE CKSUM
RECORDING ONLINE 0 0 2
raidz1-0 ONLINE 0 0 4
gptid/d98eb08c-0bbe-11e5-b431-14cc20034926 ONLINE 0 0 0
gptid/d9e72a63-0bbe-11e5-b431-14cc20034926 ONLINE 0 0 0
gptid/da434a18-0bbe-11e5-b431-14cc20034926 ONLINE 0 0 0
gptid/da974f3d-0bbe-11e5-b431-14cc20034926 ONLINE 0 0 0
gptid/daed002a-0bbe-11e5-b431-14cc20034926 ONLINE 0 0 0
gptid/db42adf6-0bbe-11e5-b431-14cc20034926 ONLINE 0 0 0
gptid/db9bd18f-0bbe-11e5-b431-14cc20034926 ONLINE 0 0 0
gptid/dbfa17f1-0bbe-11e5-b431-14cc20034926 ONLINE 0 0 0
logs
gptid/594e3023-0bc1-11e5-b431-14cc20034926 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

RECORDING/ZVOL_RECORDING:<0x1>

 

[SweetAndLow]

hardware spec and describe how you are using the system? Do you have zvol configured and is that how you are writing your data to freenas?

 

[Alexandre Trevizoli]

Sorry.
Hardware is AMD FX8350, 16GB DDR3 ECC, 8 x HD Seagate SV35 2TB, 1 x SSD 60GB ZIL.
Boot is Mirror of 2 USB Kingston 8GB. We are using 6 SATA3 on motherboard and 2 SATA3 on Extra Card (I can't remember now the chipset model).
We have a Server using Surveillance Software (DIGIFORT) writting around 20MB/s (24x7), we are using RAIDZ1(Pool with 8 Disks+ZIL)
I checked SMART of disks, and found no problem.
If need more information please ask me...

 

[joeschmuck]

Run a scrub on your FreeNAS systems again, ensure everything was fixed. As you indicated, power outages will definitely cause data corruption if you are writing at the time of the power outage. If the error occurs again you may need to just delete the file in question, then run your scrub again and all should be fine.

 

[Alexandre Trevizoli]

Joe I will did it again(on this server), but in another server that it occurs, I tried SCRUB 3 times, without success....and how I find which file is damaged, because on Status appears only.

errors: Permanent errors have been detected in the following files:

RECORDING/ZVOL_RECORDING:<0x1>

I thought ZFS is very safe in case of crash because of COW feature.... :(

 

[joeschmuck]

ZFS is not a perfect file system so sometimes file corruption can occur.

The file is identified as "RECORDING/ZVOL_RECORDING:<0x1>" which looks a bit odd to me but it sounds like you are recording constantly streaming video so I'm not sure what the file format is there. This is something you will need to figure out. If you need to save off any video, you should do so before proceeding with anything you do with respect to deleting the data.

Again, just because I'm not sure how you are recording the video (getting the stream into the FreeNAS system), so as previously asked, could you describe how you use your system, how the FreeNAS system shares are being shared (CIFS, NFS, etc...) and maybe posting the output of "zfs list" would help.

 

[Alexandre Trevizoli]

Hey Joe...sorry for my missing information...
I'm using ISCSI on Windows 2012 connected to a ZVOL 10TB.....formatted with NTFS.....files writting are normal AVI Files and Index database....but I think because NTFS under ZFS....ZFS can't see what file is corrupt, correct? (I thought of it now)
In case of Surveillance software you think CIFS or NFS is better than ISCSI? I never used CIFS or NFS because on ISCSI I use MPIO and use 2 gigabit ethernet, but I can change my idea if other way is better.
On Windows 2012 SMB 3.0 can use two or more cards, but I don't know if CIFS is compatible....
How this files are automatically recycled each 20 days....maybe the corrupted file will be deleted automatically too....is it a possibility?
Sorry again for my poor english...if you can't understand something ask me about, that I will try to explain better.

 

[danb35]

The file is identified as "RECORDING/ZVOL_RECORDING:<0x1>" which looks a bit odd to me but it sounds like you are recording constantly streaming video so I'm not sure what the file format is there.

It appears to be a zvol, and the problem is with the metadata. ZFS can't see past that, as the filesystem created on that zvol is NTFS.

@Alexandre Trevizoli, ZFS is very safe, but it isn't perfect. It gets less perfect when you do risky things like build an 8 x 2 TB RAIDZ1 pool.

 

[Alexandre Trevizoli]

How to correct this metadata problem, without loose entire zvol?

 

[joeschmuck]

I'm way out of my comfort zone when it comes to iSCSI, I've never used it nor do I ever expect to use it. If this were me, and I'm only speaking out of ignorance, If there is a way to backup your data to another drive, I'd do that and then destroy and rebuild your FreeNAS pool, but again, I'm very ignorant to this area so hopefully someone else will chime in and give you some sound advice. Also, if you can use the FreeNAS without using NTFS, that might improve recovery from this type of failure in the future.

Good Luck.

 

[SweetAndLow]

What is this extra card your drivers are attached to?

The reason it doesn't know what file is corrupt is because a zvol can be considered one big file. That is how iscsi works, it creates a single device on the file system that you connect to and your client thinks it's a disk it can format.

 

[joeschmuck]

it creates a single device on the file system that you connect to and your client thinks it's a disk it can format.

Wow, that is eye opening to me. I didn't realize that.