Boot Volume Condition: DEGRADED on several new disks.

[Spog]

Thank you for reading this.I know this has been covered before but not on new disks as I can find.

I am running a Dell FS12-NV7 16GB ECC RAM 12x2TB drives replaced controller cards running RAIDZ2.

Until recently I have been running off USB drives but these seem to be lasting about 3 months so I decided to upgrade to SSD with SDSSDA-12G by SanDisk as I have several of these servers I bought several disks. On installing the first disk I immediately got Boot Volume Condition: DEGRADED One or more devices has experienced an error resulting in data corruption. Applications may be affected. So I swapped the drive thinking it may be a bad drive, also swapping the cable and port which the drive is connected to. Even after the change I got the same error. This is fresh install using 11.1-U4 downloaded several times from the website and burnt using RUFUS to USB stick.

Output from zpool status -v freenas-boot

pool: freenas-boot
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 9.50K in 0 days 00:00:06 with 42 errors on Sat Apr 7 15:30:37 2018
config:

NAME STATE READ WRITE CKSUM
freenas-boot DEGRADED 0 0 44
ada0p2 DEGRADED 0 0 95 too many errors

errors: Permanent errors have been detected in the following files:

<metadata>:<0x22>
freenas-boot/ROOT/default:<0x0>
//usr/local/lib/migrate93/freenasUI/sharing/__pycache__
//usr/local/lib/migrate93/freenasUI/storage/__pycache__
freenas-boot/ROOT/default@2018-04-07-14:10:28:<0x0>
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/share/netdata/web/old/datasource.html
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/python3.6/__pycache__/rlcompleter.cpython-36.opt-2.pyc
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/share/netdata/web/version.txt
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/www/dojo/dojox/dgauges/components
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/www/dojo/dojo/nls/mk
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/www/dojo/dojox/dgauges/components/black
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/migrate93/freenasUI/storage/migrations/__pycache__
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/www/dojo/dojo/nls/nb
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/share/locale/pl_PL.UTF-8
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/www/dojo/dojo/nls/pl
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/bin/slsh
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/www/dojo/dojox/jq
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/migrate93/django/core/management
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/www/dojo/dojox/json
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/migrate93/django/core/management/__pycache__
freenas-boot/ROOT/default@2018-04-07-14:10:28:/conf/base/etc/devd/asus.conf
freenas-boot/ROOT/default@2018-04-07-14:10:28:/conf/base/etc/devd/ulpt.conf
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/migrate93/django/core/management/commands
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/migrate93/django/core/management/commands/__pycache__
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/share/locale/ko_KR.UTF-8/LC_COLLATE
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/migrate93/django/contrib/gis/gdal/prototypes
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/migrate93/django/contrib/gis/gdal/prototypes/__pycache__
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/migrate93/django/core
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/migrate93/django/core/__pycache__
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/www/dojo/dojox/widget/nls/zh-tw
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/bin/libvncserver-config
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/python3.6/site-packages/pysnmp/carrier/asyncio
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/migrate93/freenasUI/system/migrations
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/python3.6/site-packages/pysnmp/carrier/asyncio/__pycache__
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/python3.6/site-packages/flask/__pycache__/exthook.cpython-36.pyc
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/www/dojo/dojox/widget/nls/zh
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/python3.6/site-packages/markdown/__pycache__/postprocessors.cpython-36.opt-1.pyc
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/lib/python3.6/multiprocessing
freenas-boot/ROOT/default@2018-04-07-14:10:28:/usr/local/www/dojo/dojox/widget/rotator

I am wondering if I genuinely have several bad SSDs or am I doing something wrong.
I tried to do a Verify Install and I get

Request Method: POST
Request URL: http://192.168.1.23/system/update/verify/
Software Version: FreeNAS-11.1-U4 (89e3d93bc)
Exception Type: MiddlewareError
Exception Value:
[MiddlewareError: [Errno 5] Input/output error]
Exception Location: ./freenasUI/system/views.py in update_verify, line 1783
Server time: Sat, 7 Apr 2018 16:04:53 +0100
Traceback



Request information
GET
No GET data

POST
Variable Value
__form_id 'form_str'
FILES
No FILES data

COOKIES
Variable Value
csrftoken '********'
sessionid 'z3tin1fugintxozbajv9hu45ev6w2s8w'
fntreeSaveStateCookie 'root%2Croot%2F10%2Croot%2F66%2Croot%2F66%2F67'
META
Variable Value

Thanks very much for reading this and looking forwards to some suggestions

 

[Spog]

Just checked SMART to see if that could shed any light on the situation

smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: SanDisk SSD PLUS 120 GB
Serial Number: 180485802123
LU WWN Device Id: 5 001b44 8b65a2d4b
Firmware Version: UE3600RL
User Capacity: 120,040,980,480 bytes [120 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 T13/2015-D revision 3
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Apr 7 16:25:04 2018 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 32) The self-test routine was interrupted
by the host with a hard or soft reset.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x15) SMART execute Offline immediate.
No Auto Offline data collection support.
Abort Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 21) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 1
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 2
165 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
166 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 0
167 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 0
168 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 0
169 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 95
170 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 0
171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
173 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 --- Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 --- Old_age Always - 0
194 Temperature_Celsius 0x0022 079 050 000 Old_age Always - 21 (Min/Max 11/50)
199 UDMA_CRC_Error_Count 0x0032 100 100 --- Old_age Always - 0
230 Unknown_SSD_Attribute 0x0032 100 100 000 Old_age Always - 0
232 Available_Reservd_Space 0x0033 100 100 005 Pre-fail Always - 100
233 Media_Wearout_Indicator 0x0032 100 100 --- Old_age Always - 0
234 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 3
241 Total_LBAs_Written 0x0030 100 100 000 Old_age Offline - 1
242 Total_LBAs_Read 0x0030 100 100 000 Old_age Offline - 3
244 Unknown_Attribute 0x0032 000 100 --- Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

Selective Self-tests/Logging not supported

Not sure if that helps but I can not see any problems.

Thanks again.

 

[joeschmuck]

So lets do this the easy way and I'll ask you a few questions and hopefully you will be able to answer them. Also, lets not make any assumptions as these can lead everyone down the wrong path. I will assume you know nothing and you will assume I have no clue what you actually did. It sounds harsh but it levels the playing field and gets the job done faster. Expalin what you did like I'm an idiot and all will be good.

1) Do you have a backup of your configuration file before you made any hardware changes? And what version of FreeNAS is it from?
You will use this if you need to restore your configuration. Hopefully it is FreeNAS 11.0 or later.

2) Do you have only a single SSD installed as your boot device and no other boot devices install, meaning USB too?
In my opinion you only need a single boot device, and if going SSD then leave it as a single device.

3) If you have a config backup and only a single boot device then perform the following steps otherwise explain more about your setup and if you can roll back to your earlier version. Also if your configuration backup is older than FreeNAS 11.0 then I'd restore the version of FreeNAS it was made with as there is a possibility the data is not going to work with the current version of FreeNAS and without knowing what you have then I can only speculate and that is just not good to troubleshoot your problem.

4) So you have a backup of your configuration and a single SSD, lets reinstall
5) Boot up the FreeNAS installer and reinstall FreeNAS to your SSD, do not use UEFI as this could give you problems depending on your system, if that fails then try UEFI as a last resort.
6) When FreeNAS reboots it should be up and running just fine and want you to configure your system, exit out of that. If FreeNAS fails to boot up to this configuration screen then something is wrong.
7) Restore your configuration backup, it should work.
8) Report your results in painful detail. I'm hoping it all just works.

 

[Spog]

Thank you so much for your reply, I work in I.T. my self so idiot walk through is good some times

1) I have a backup from when I used the USB I even still have the USB stick en though it has faults its self. The old version was 11.1-U2 but even clean install of 11.1-U4 gives error.

2) it is now a single SSD boot device I am not worried about duplexing the boot device as I have regular backups of the config.

3) it is a fresh install of 11.1-U4 even before I install the config I get degraded boot device. The original version was 11.1-U2 but even before I install the old config the error exists.

4) I have reinstalled on 3 different SSDs all give exactly the same problem with different cables and different ports on the SATA interface.

5) UEFI unfortunately does not work on my hardware as it is too old.

6) as far as I can see no errors on boot.

7) I have restored backup still same errors.

8) I am in pain not sure how much more information I can give but I will explain:-

the system is:-
FreeNAS-11.1-U4
Platform Quad-Core AMD Opteron(tm) Processor 2373 EE
Memory 16344MB it is ECC RAM

How I installed:-

Downloaded from freenas website and burnt using RUFUS to known working USB for install( done 3 times)
Booted from USB Stick to install
Followed instructions to install to SSD no special configuration. Have done this many times to USB with no issues on this hardware.
Logged on to webconfig page and immediately got the error of degraded boot device.
Installed old configuration installed fine no errors.
rebooted
Still same error but now fully working config as before.

Thanks for taking the time to help I hope I have answered your questions. I am really not sure where to go next my system seems fine not sure if it is a bad batch of SSD's but SMART seems ok so no idea where to go next I have 2 more to upgrade but think I might keep them on USB and just keep buying them.

 

[joeschmuck]

First of all I do not suspect your SSD's as the failure.

Second it sounds like you are installing it correctly, it's pretty difficult to screw up, it either works or it doesn't boot at all.

Third, do I understand you correctly in that this version of FreeNAS does work when installed on a USB Flash Drive? If yes then I have seen this problem before and the only options are to either use the USB or possibly try to toss in an add-on SATA card and try to boot off of that. I can't tell you the add-on card will work, it's a risk to be tested if you desire to use a SSD.

Lets talk USB Flash drive, if you go this route then buy a USB 2.0 drive, not a USB 3.0 drive. FreeNAS has been redesigned to stop eating up Flash Drives but USB 3.0 drives, especially small ones can generate too much heat and wear out fast. USB 2.0 drives don't typically have that problem, and you don't need a fast USB flash drive.

 

[Spog]

Thank you so much for the time you have given. I hope one day i will be able to help some one as you have helped me.

Unfortunetly a sata card is not an option as there are no free slots in my server. Looks like I will have to go back to USB sticks, ah well worth a try now I have some free SSDs.

 

[joeschmuck]

Unfortunetly a sata card is not an option as there are no free slots in my server. Looks like I will have to go back to USB sticks, ah well worth a try now I have some free SSDs.

Sometimes that happens and it's unfortunate. But USB Flash drives are not the end of the world, just stick to USB 2.0 and it should be fine. If you really wanted USB 3.0 then you should find one that has a metal shell and is at least 1" long so it can dissapate heat. I have one I purchased for this project but I ended up going SSD pretty quickly, now I use that USB 3.0 drive to boot Ubuntu to my laptop when I want something other than Windoze 10.

 

[anmnz]

Hello, @Spog. Looks like you may have bad SSDs in fact -- I saw the same errors when trying to install FreeNAS to the exact same SSD model (SanDisk SSD PLUS 120 GB) and firmware version (UE3600RL). @capa posted a bug report for this problem, reported by several of us: https://redmine.ixsystems.com/issues/35065

 

[moelassus]

I still have two of the non-working SanDisk SSDs (though I need to return them soon). Is there anything I can do to help with this issue? I have a spare system that I'm happy to install them in if there is some data that can be collected.

 

[anmnz]

Well the iXsystems engineer on the bug report knows about a billion times as much as I do, and says he has the same SSD with the same firmware to hand, so I imagine it's all covered at that end.

Purely selfishly, I'd love to see another independent confirmation that the problem goes away with TRIM disabled, just to put to rest any nagging worry I might have that I had messed up my testing... :)

 

[joeschmuck]

Well the iXsystems engineer on the bug report knows about a billion times as much as I do, and says he has the same SSD with the same firmware to hand, so I imagine it's all covered at that end.

Well, as reliable as the iXsystems team are, you never know if the problem was replicated exacty the same. It'a always best to verify it if possible, and it gives them a good feeling as well if the results are the same.

Purely selfishly, I'd love to see another independent confirmation that the problem goes away with TRIM disabled, just to put to rest any nagging worry I might have that I had messed up my testing... :)

So I agree with your statement here but it's not being selfish.