SOLVED Corrupted SSD Mirrored boot drive

Status
Not open for further replies.
MauricioU

MauricioU

Explorer
Joined
May 22, 2017
Messages
64
A few days ago I bought two sandisk 120 gb ssds as boot drives. I have a very similar system to this:

http://jro.io/nas/#build

Only difference is I use an intel s2600cp2j motherboard.

I install the sandisk ssds in, I reinstall freenas on them, everything goes smoothly, but upon loading up the webgui I get the critical error that the boot drive is degraded. So I think okay, one or both of the drives are corrupted or faulty. I replace both drives with brand new sandisk ssds. same thing. I reinstall freenas, still corrupted. I don't understand what to do now.

This is the link I get referenced to:

http://illumos.org/msg/ZFS-8000-8A

then I scrubed the boot drives and I got referred to this:

http://illumos.org/msg/ZFS-8000-9P


If anyone can help, is it something with sandisk SSDs? Is there something I'm doing wrong? I am a pretty savvy user and had no issues with booting from dual usb sticks.
 
joeschmuck

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,994
First of all you really need to follow the forum rules please. What version of FreeNAs are you running? If you are using a Beta or RC then submit a bug report.

Second, ths is just my advice but I'd only run a single SSD as a boot device. These are sognificantly more robust that USB Flash Drives. Of course the fact that it doesn't work is still a problem. you may try to use a single SSD just to see if the problem occurs there too.

Third, oh crap I don't have a third :)

Well provide as much detail as you can and any screen captures. You may need to post the output of dmesg and zpool status in code brackets please.
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
Also look out for the thread(s) and redmine bug report mentioning 3D NAND and boot drives corrupting.
 
MauricioU

MauricioU

Explorer
Joined
May 22, 2017
Messages
64
joeschmuck said:
First of all you really need to follow the forum rules please. What version of FreeNAs are you running? If you are using a Beta or RC then submit a bug report.

Second, ths is just my advice but I'd only run a single SSD as a boot device. These are sognificantly more robust that USB Flash Drives. Of course the fact that it doesn't work is still a problem. you may try to use a single SSD just to see if the problem occurs there too.

Third, oh crap I don't have a third :)

Well provide as much detail as you can and any screen captures. You may need to post the output of dmesg and zpool status in code brackets please.
Click to expand...


I didn't mean to not follow the rules. My apologies. I am running freenas11.1-U6 Stable, no beta or RC. This is the error I'm getting:

upload_2018-10-20_12-58-3.png

upload_2018-10-20_12-57-34.png

upload_2018-10-20_12-59-16.png


When I run zpool status:

Code:
  pool: freenas-boot
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
		corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
		entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 11K in 0 days 00:00:03 with 31 errors on Fri Oct 19 16:30:45 2018
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  DEGRADED	 0	 0	 6
		  mirror-0  DEGRADED	 0	 0	24
			ada0p2  DEGRADED	 0	 0	24  too many errors
			ada1p2  DEGRADED	 0	 0	24  too many errors

errors: 30 data errors, use '-v' for a list

  pool: tank
 state: ONLINE
  scan: none requested
config:

		NAME											STATE	 READ WRITE CKSUM
		tank											ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/af82b9b0-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b01bedb5-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b0b46ee8-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b14bfec8-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b1f8d0bf-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b28bfc07-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b3239d94-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b3bd257e-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
		  raidz2-1									  ONLINE	   0	 0	 0
			gptid/b4678646-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b505d34a-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b5ade27d-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b6623934-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b71623e8-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b7c8a2a9-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b87f25b7-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b931e420-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
		spares
		  gptid/ba549b6b-d1d9-11e8-b1d7-a0369f5094b8	AVAIL

errors: No known data errors


When I run zpool status -v:

Code:

  pool: freenas-boot
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
		corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
		entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 11K in 0 days 00:00:03 with 31 errors on Fri Oct 19 16:30:45 2018
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  DEGRADED	 0	 0	 6
		  mirror-0  DEGRADED	 0	 0	24
			ada0p2  DEGRADED	 0	 0	24  too many errors
			ada1p2  DEGRADED	 0	 0	24  too many errors

errors: Permanent errors have been detected in the following files:

		<metadata>:<0x2a>
		<metadata>:<0x3c>
		<metadata>:<0x3e>
		<metadata>:<0x48>
		<metadata>:<0x4c>
		freenas-boot/ROOT/default:<0x0>
		//usr/local/lib/migrate93/django/db/__pycache__/transaction.cpython-36.pyc
		freenas-boot/ROOT/default:<0x282e7>
		freenas-boot/grub:<0x0>
		freenas-boot/ROOT/default@2018-10-19-23:19:53:<0x0>
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/perl5/site_perl/man
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/perl5/site_perl/man/man1
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/encodings/__pycache__/johab.cpython-36.opt-1.pyc
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/encodings/__pycache__/johab.cpython-36.opt-2.pyc
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/site-packages/babel/global.dat
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/perl5/site_perl/man/man3
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/site-packages/babel/locale-data/af.dat
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/site-packages/babel/locale-data/am.dat
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/site-packages/google/protobuf/__pycache__/duration_pb2.cpython-36.opt-1.pyc
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/site-packages/urllib3/util/__pycache__/wait.cpython-36.opt-1.pyc
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/site-packages/chardet/__pycache__/utf8prober.cpython-36.opt-1.pyc
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/www/data/docs/_static/plus.png
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/site-packages/dojango/util/__pycache__/config.cpython-36.opt-1.pyc
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/site-packages/chardet/cli
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/site-packages/gevent-1.2.2-py3.6.egg-info
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/share/locale/sl
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/share/locale/sl/LC_MESSAGES
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/lib/python3.6/site-packages/gevent
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/share/locale/sq
		freenas-boot/ROOT/default@2018-10-19-23:19:53:/usr/local/share/locale/sq/LC_MESSAGES

  pool: tank
 state: ONLINE
  scan: none requested
config:

		NAME											STATE	 READ WRITE CKSUM
		tank											ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/af82b9b0-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b01bedb5-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b0b46ee8-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b14bfec8-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b1f8d0bf-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b28bfc07-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b3239d94-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b3bd257e-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
		  raidz2-1									  ONLINE	   0	 0	 0
			gptid/b4678646-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b505d34a-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b5ade27d-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b6623934-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b71623e8-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b7c8a2a9-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b87f25b7-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
			gptid/b931e420-d1d9-11e8-b1d7-a0369f5094b8  ONLINE	   0	 0	 0
		spares
		  gptid/ba549b6b-d1d9-11e8-b1d7-a0369f5094b8	AVAIL

errors: No known data errors


I ran the dmesg cmd and it gave me so much information I didn't think it would be relevant, but if you'd like that I would be willing to copy and paste.

I hope this helps.
 
MauricioU

MauricioU

Explorer
Joined
May 22, 2017
Messages
64
sretalla said:
Also look out for the thread(s) and redmine bug report mentioning 3D NAND and boot drives corrupting.
Click to expand...


I did not think to look this up as a 3D NAND issue. Will do that right now, as the sandisk ssds are 3D NAND. Care to point me in any one direction?

Appreciate the help guys! Sorry if my initial post was a bit off putting or in the least lacking of appropriate information. I've been researching this for some days, normally I post here as a last resort, mostly because I hate having to wait for an answer lollll.
 
anmnz

anmnz

Patron
Joined
Feb 17, 2018
Messages
286
MauricioU

MauricioU

Explorer
Joined
May 22, 2017
Messages
64
anmnz said:
Seems to be, yes.

Lots of detail in this bug report:
https://redmine.ixsystems.com/issues/35065

and the threads it references:
https://forums.freenas.org/index.php?threads/boot-disk.63829/
https://forums.freenas.org/index.php?threads/freenas-installer-sandisk-ssd-checksum-errors.64049/
https://forums.freenas.org/index.php?threads/transcend-ssd-boot-disk-zfs-checksum-errors.64321/
Click to expand...


Been reading these since I responded, this is definitely what is happening to me. Thanks for this! I just wasn't searching the right keywords.
 
MauricioU

MauricioU

Explorer
Joined
May 22, 2017
Messages
64
joeschmuck

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,994
MauricioU said:
I didn't mean to not follow the rules. My apologies.
Click to expand...
Not a problem, I like to remind people that we do have them. I'm not upset or irritated, it happens. Had I been upset you would know it, I'm not shy, but I would also hope my fellow moderators would have my back and tell me to go eat a Snickers Bar :)

MauricioU said:
the solution is either turn off trim on the system which is not recommended as we don't know what kind of consequences that could have
Click to expand...
Turning off TRIM would only lead to slower writing a block of data should the block need to be erased first. As a boot drive this will not be an issue.

Isn't it crazy that a SSD TRIM routine can cause these errors. I take it that you are going to buy a pair of drives again? And that isn't a bad price, it was much worse several years ago.

Thank @anmnz for the help!
 
A

andrema2

Explorer
Joined
Aug 3, 2011
Messages
83
MauricioU said:
In case anybody stumbles upon this in the future and has the same issue I'm having, I figured it out, the solution is either turn off trim on the system which is not recommended as we don't know what kind of consequences that could have OR use different SSDs that are not based on 3D NAND/silicon motion controller.

SO I'm buying these:

https://www.amazon.com/gp/product/B01N6JQS8C/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1
Click to expand...

Hi Mauricio

After you got this drive did the problem stop ?

Thanks


Sent from my iPhone using Tapatalk
 
Status
Not open for further replies.

Similar threads

A
Degraded Pools But No Corrupted?
Replies
5
Views
1K
garm
garm
B
  • Locked
Cannot import pool - Corrupted metadata, I/O error
Replies
10
Views
3K
brendanshoe
B
jimwise00
Corrupt file on brand new install?
Replies
15
Views
4K
anmnz
anmnz
J
freenas-boot is degraded
2
Replies
23
Views
6K
Balu5
Balu5
J
  • Locked
Can't Access Pool?
2
Replies
27
Views
4K
jgreco
jgreco
Top