-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum has become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
Storage showing Degraded
- Thread starter FilmBuff5
- Start date
- Status
Stux
MVP
- Joined
- Jun 2, 2016
- Messages
- 4,419
try running
(or alternatively, click on the top NAS2 in that window, then click on status at the bottom (a button appears). It should show you the components of the pool, and their statuses
You need to determine which disk failed, and why, and what to do about it. Then remedy it.
For example, maybe the power supply to the disk was unplugged... plug it back in... or maybe the sata cable failed... replace it... or maybe the disk just died... replace it... etc.
zpool status NAS2 in the terminal window(or alternatively, click on the top NAS2 in that window, then click on status at the bottom (a button appears). It should show you the components of the pool, and their statuses
You need to determine which disk failed, and why, and what to do about it. Then remedy it.
For example, maybe the power supply to the disk was unplugged... plug it back in... or maybe the sata cable failed... replace it... or maybe the disk just died... replace it... etc.
Good morning, Stux. Thanks for the reply.
Running the command with the verbose option shows that a file has been detected with a pemanent error.
Right now, I'm coping all the files from my old NAS to the new NAS2. At the moment, the process has this file in use. When the copy is done, I will delete this file and try again.
Assuming the operation is successful, would the staus update itself to Normal or would I have to run a command to clear this error to restore normal operation to my NAS.
Running the command with the verbose option shows that a file has been detected with a pemanent error.
Right now, I'm coping all the files from my old NAS to the new NAS2. At the moment, the process has this file in use. When the copy is done, I will delete this file and try again.
Assuming the operation is successful, would the staus update itself to Normal or would I have to run a command to clear this error to restore normal operation to my NAS.
If you have a file with a permanent error in a mirror that's not a good sign.
Even if you have one failed disk, this should not result in an permanent error imediately.
You should check what disk is failing and follow up on what stux already said.
if you fixed the problem with the hardware, you should run a scrub on the pool to see if everything is hunkidory or if more of your files are corrupted (don't do the scrub while resilvering if you had to replace the drive as opose to switching out a cable or similar).
If the scrub comes back clean the error should disapear
Even if you have one failed disk, this should not result in an permanent error imediately.
You should check what disk is failing and follow up on what stux already said.
if you fixed the problem with the hardware, you should run a scrub on the pool to see if everything is hunkidory or if more of your files are corrupted (don't do the scrub while resilvering if you had to replace the drive as opose to switching out a cable or similar).
If the scrub comes back clean the error should disapear
Hi, Huib.
The utility I'm using to copy the files is Teracopy. This set of data I'm copying is about 4tb and is about 75% completed.
There is a verify/compare operation which follows and afterwards, I should get a summary of what files didn't successfully copy independent of what FreeNAS reports. So it will probably take another 24-36 hrs till completion.
I can compare the two listings, then dlete and copy again anything which was flagged.
The utility I'm using to copy the files is Teracopy. This set of data I'm copying is about 4tb and is about 75% completed.
There is a verify/compare operation which follows and afterwards, I should get a summary of what files didn't successfully copy independent of what FreeNAS reports. So it will probably take another 24-36 hrs till completion.
I can compare the two listings, then dlete and copy again anything which was flagged.
Hi FilmBuff
That will only make sure that you copied over the files. But that will not tell you if those files are corrupted or not.
You should do it anyway especially since you are having trouble with the data in the pool so backup is step 1.
I still reccomend you to do a scrub after that to see if more files are corrupted.
However now I'm thinking about it, it could be that freenas will report more corrupted files during the copy since it is reading all the files, so it is checking the checksums of all files also so it might not be needed to do a scrub.
I would still do it though. Better safe than sorry but I guess that's just my nature.
On a side note:
I understand you don't have a second freenas system? If you would have that you could use replication instead of teracopy and that should be much faster. the remaining 1 TB should be done in about 3 to 4 hours for a slow pool (over gigabit ethernet).... the full 4 TB should take 11 to 16 hours.
If you can find a way to get a second system (for the future) I would recomend it. I have two systems (see my sig) and my work system keeps up to date with my home server over a crapy internet line every hour. It uploads the changes in a mater of minutes since it only sends the changed blocks (not whole files). and the scanning does not have to take place like with terracopy.
I tried terracopy before and it was useless for me due to the scanning and changes on the filesystem during scanning. It is actually the reason why i changed to a ZFS file server.
That will only make sure that you copied over the files. But that will not tell you if those files are corrupted or not.
You should do it anyway especially since you are having trouble with the data in the pool so backup is step 1.
I still reccomend you to do a scrub after that to see if more files are corrupted.
However now I'm thinking about it, it could be that freenas will report more corrupted files during the copy since it is reading all the files, so it is checking the checksums of all files also so it might not be needed to do a scrub.
I would still do it though. Better safe than sorry but I guess that's just my nature.
On a side note:
I understand you don't have a second freenas system? If you would have that you could use replication instead of teracopy and that should be much faster. the remaining 1 TB should be done in about 3 to 4 hours for a slow pool (over gigabit ethernet).... the full 4 TB should take 11 to 16 hours.
If you can find a way to get a second system (for the future) I would recomend it. I have two systems (see my sig) and my work system keeps up to date with my home server over a crapy internet line every hour. It uploads the changes in a mater of minutes since it only sends the changed blocks (not whole files). and the scanning does not have to take place like with terracopy.
I tried terracopy before and it was useless for me due to the scanning and changes on the filesystem during scanning. It is actually the reason why i changed to a ZFS file server.
Oh for sure...I plan to do a scrub after the teracopy to see if it passes with flying colors.
Getting back to FreeNAS and the error though...is it safe to say though that performance really isn't degraded though?
Yes, FreeNAS is throwing up the alert about that one file, but the overall health of that mirror isn't in jeopardy, is it?
Getting back to FreeNAS and the error though...is it safe to say though that performance really isn't degraded though?
Yes, FreeNAS is throwing up the alert about that one file, but the overall health of that mirror isn't in jeopardy, is it?
Johnnie Black
Guru
- Joined
- May 10, 2017
- Messages
- 838
ZFS won't let you copy a corrupt file, you'll get an I/O error.
So the teracopy job froze at the 98% mark...just stopped on a particular directory (which I've now moved to the side). It's doing a file compare...will take awhile to complete (about 36 hrs).
I definitely have some corrupted files on my old NAS...just a question of how much. It's a ReadyNAS Pro from around 2011 or so.
I definitely have some corrupted files on my old NAS...just a question of how much. It's a ReadyNAS Pro from around 2011 or so.
36 hours for a compare....
And that's why I started using zfs. (virtually) zero second "compare" and a few minute sync for 1.5 TB of verry small files.
Plus the instant recovery of a old state of all files (or part of them) without losing the current state with clone snapshot...
And that's why I started using zfs. (virtually) zero second "compare" and a few minute sync for 1.5 TB of verry small files.
Plus the instant recovery of a old state of all files (or part of them) without losing the current state with clone snapshot...
update: lost connectivity to the ReadyNAS as the verify was running. I haven't yet looked at why or thestate of the ReadyNAS.
But what is strange now is that the freenas is showing as healthy, while the Critcal is still flashing red.
So who do I believe?
But what is strange now is that the freenas is showing as healthy, while the Critcal is still flashing red.
So who do I believe?
Jailer
Not strong, but bad
- Joined
- Sep 12, 2014
- Messages
- 4,977
No inconsistencies with FreeNAS at all, its working exactly as intended. You had an error at some point and you need to find out where. The status indicator wont go green until you run
Please post the output from
zpool clear to clear the error.Please post the output from
zpool status in code tags.Ahhh, that.s what I needed to know...thanks.
There is an 'inconsistency' of a sort I'm noticing. Or maybe inconsistency isn't the right word...
About 2 months ago, I built a Plex server and I'm engaged right now in transferring the Movies to the FreeNAS. So these files are <2 months old...yet I'm seeing many file verify errors during the verify of the teracopy job. Far as I can see, no error is being reported by FreeNAS.
The original files aren't corrupt in any way (bit-rot or otherwise) but somehow, during to copy to the FreeNAS, somwthing doesn't quite copy correctly.
My FreeNAS consists of 3 mirrored vdevs - is there something I could look at that might explain or point to something misbehaving or not optimized in some way?
My understanding is a file is spread across the 3 vdevs...so is the 'syncronization' of the vdevs off in some way? Where 1 part (of the 3 parts) is throwing something off and leading to the verify errors?
I really appreciate all the help so far.
Last edited:
Jailer
Not strong, but bad
- Joined
- Sep 12, 2014
- Messages
- 4,977
Also a complete list of hardware for your new FreeNAS machine.
Code:
[root@freenas ~]# zpool status pool: NAS2 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 0 in 3h56m with 1 errors on Wed Oct 25 17:46:20 2017 config: NAME STATE READ WRITE CKSUM NAS2 DEGRADED 0 0 8.90M mirror-0 ONLINE 0 0 0 gptid/3a0d750a-9745-11e7-8480-2c4d54d2f3bf.eli ONLINE 0 0 0 gptid/3a85a3b4-9745-11e7-8480-2c4d54d2f3bf.eli ONLINE 0 0 0 mirror-1 DEGRADED 0 0 17.8M gptid/3b07f5f1-9745-11e7-8480-2c4d54d2f3bf.eli DEGRADED 0 0 17.8M too many errors gptid/3b7a03e1-9745-11e7-8480-2c4d54d2f3bf.eli DEGRADED 0 0 17.8M too many errors mirror-2 ONLINE 0 0 0 gptid/3c0fd28a-9745-11e7-8480-2c4d54d2f3bf.eli ONLINE 0 0 0 gptid/3c882955-9745-11e7-8480-2c4d54d2f3bf.eli ONLINE 0 0 0 errors: 2 data errors, use '-v' for a list pool: freenas-boot state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Tue Oct 17 03:45:52 2017 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da0p2 ONLINE 0 0 0 da1p2 ONLINE 0 0 0 errors: No known data errors
Hardware:
Intel i3 4160
Asus Motherboard
32 gb non-ECC RAM
500W power supply
6 WD Red Pro 6 tb drives
2 SanDisk 16gb USB drives
Jailer
Not strong, but bad
- Joined
- Sep 12, 2014
- Messages
- 4,977
What ASUS motherboard? The fact that you don't have ECC memory isn't helpingyou much either.
mirror-1 is degraded due to checksum errors. Where are those 2 drives connected? Check your cabling for those 2 drives. Also what power supply are you using?
mirror-1 is degraded due to checksum errors. Where are those 2 drives connected? Check your cabling for those 2 drives. Also what power supply are you using?
Another update:
After some interminable pauses where the teracopy job just paused until I skipped the file it was trying to process. The job finished w/errors of course after 2+ days.
What I like about teracopy is the ability to just do another pass on the files that threw up an error and over the course of the last couple of days, that's what I did...whittling down the unsuccessful files until they all passed muster on the FreeNAS.
I'm doing one last run from the start before I reappropriate the ReadyNAS as a secondary repository copy of the most important files on the FreeNAS going forward.
I ordered some spare SATA cables and will replace both cables on mirror1 once the final teracopy is over.
I really don't have the same level of confidence in this FreeNAS as I did with the ReadyNAS Pro. The latter has been with me for 6 years (and counting) now and not a single hiccup.
I originally had started with three 1 TB drives when I purchased the ReadyNAS and then decided to go with four 2 TB drives instead of adding a fourth 1 TB.
I'm at a loss to understand why the FreeNAS is so finicky...
After some interminable pauses where the teracopy job just paused until I skipped the file it was trying to process. The job finished w/errors of course after 2+ days.
What I like about teracopy is the ability to just do another pass on the files that threw up an error and over the course of the last couple of days, that's what I did...whittling down the unsuccessful files until they all passed muster on the FreeNAS.
I'm doing one last run from the start before I reappropriate the ReadyNAS as a secondary repository copy of the most important files on the FreeNAS going forward.
I ordered some spare SATA cables and will replace both cables on mirror1 once the final teracopy is over.
I really don't have the same level of confidence in this FreeNAS as I did with the ReadyNAS Pro. The latter has been with me for 6 years (and counting) now and not a single hiccup.
I originally had started with three 1 TB drives when I purchased the ReadyNAS and then decided to go with four 2 TB drives instead of adding a fourth 1 TB.
I'm at a loss to understand why the FreeNAS is so finicky...
- Status
