So i got a problem and i'm not sure how it happened. In the past i've had issues like this aswell but then i saw things that were wrong in the shell. This time when checking i never saw anything wrong. However today my server needed a shutdown because i wanted to implement a UPS and needed 3 reboots before turning on. Naturally i went to go check why i needed that.
First things first, i've been getting this error either for ada1 or ada0 they seem to alternate and never did give me an error when running smartctl -a /dev/ada1 (or ada0 if the situation required it)
Notification when logged into the shell.
Device: /dev/ada1, not capable of SMART self-check
This is the smart ctl response.
Seems ok to me?
But since it needed 3 reboots i decided to check the status
Now why it says REMOVED or replacing in there? One thing that i thought of myself is that it could be because of power shortages, we had 3 outages in a short span of time. Is it from that? I have no idea but it's a possibility, nothing got changed, the case that holds it all didn't move to my knowledge. The last time i did zpool status command, it showed both drives online and everything was working fine. But those outages are why i bought the UPS and it's plugged into that at the moment.
If anyone has any idea how this happened i'm open to ideas. The freenas box was updated to FreeNAS-11.3-U4.1 from 11.3-U3.2 in september, i still have the DB file and a compressed tar from then as it was recommended to backup those. At the time i checked after the update and all commands came back positive, showing everything online, so i don't think the update is the problem.
If more information is needed, just ask i'll do my best. Still new at all this and have not encountered this before.
First things first, i've been getting this error either for ada1 or ada0 they seem to alternate and never did give me an error when running smartctl -a /dev/ada1 (or ada0 if the situation required it)
Notification when logged into the shell.
Device: /dev/ada1, not capable of SMART self-check
This is the smart ctl response.
Code:
root@Brisingr[~]# smartctl -i /dev/ada1 smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p11 amd64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: WD Blue and Green SSDs Device Model: WDC WDS120G2G0A-00JH30 Serial Number: 2003BQ461911 LU WWN Device Id: 5 001b44 4a830af06 Firmware Version: UE510000 User Capacity: 120,040,980,480 bytes [120 GB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 T13/2015-D revision 3 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Dec 4 15:04:34 2020 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled
Seems ok to me?
But since it needed 3 reboots i decided to check the status
Code:
root@Brisingr[~]# zpool status -v freenas-boot pool: freenas-boot state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 0 in 0 days 00:15:56 with 139 errors on Mon Nov 30 04:00:56 2020 config: NAME STATE READ WRITE CKSUM freenas-boot DEGRADED 0 0 103 mirror-0 DEGRADED 0 0 413 replacing-0 UNAVAIL 0 0 0 15747358114187901216 REMOVED 0 0 0 was /dev/ada0p2/old 25115368915956188 FAULTED 0 0 0 was /dev/ada0p2 ada1p2 DEGRADED 0 0 413 too many errors errors: Permanent errors have been detected in the following files: <metadata>:<0x1d> <metadata>:<0x29> <metadata>:<0x2a> <metadata>:<0x2b> <metadata>:<0x38> <metadata>:<0x42> <metadata>:<0x47> <metadata>:<0x54> <metadata>:<0x6f> <metadata>:<0x83> freenas-boot/ROOT:<0x0> freenas-boot/ROOT/11.3-U4.1@2020-05-16-00:55:51:<0x0> freenas-boot/ROOT/11.3-U4.1@2020-05-16-00:55:51:/usr/local/lib/python3.7site-packages/docutils/writers freenas-boot/ROOT/11.3-U4.1@2020-05-16-00:55:51:/usr/local/lib/python3.7/site-packages/docutils/writers/__pycache__ freenas-boot/ROOT/11.3-U4.1@2020-05-16-00:55:51:/usr/local/lib/python3.7/site-packages/dojango/contrib/auth/__pycache__ freenas-boot/ROOT/11.3-U4.1@2020-05-16-00:55:51:/usr/local/lib/python3.7/site-packages/dns_lexicon-3.3.3-py3.7.egg-info freenas-boot/ROOT/11.3-U4.1@2020-05-16-00:55:51:/usr/local/lib/python3.7/site-packages/dojango/data/modelstore freenas-boot/ROOT/11.3-U4.1@2020-05-16-00:55:51:/usr/local/lib/python3.7/site-packages/docutils/writers/pep_html freenas-boot/ROOT/11.3-U4.1@2020-05-16-00:55:51:/usr/local/lib/python3.7/site-packages/dns/rdtypes/CH/__pycache__ freenas-boot/ROOT/11.3-U4.1@2020-05-16-00:55:51:/usr/local/lib/python3.7/site-packages/dns/rdtypes freenas-boot/ROOT/11.3-U4.1@2020-05-16-00:55:51:/usr/local/share/locale/uk/LC_MESSAGES/gas.mo freenas-boot/ROOT/11.3-U4.1@2020-06-16-01:02:20:<0x0> freenas-boot/ROOT/11.3-U4.1@2020-06-16-01:02:20:/usr/local/lib/python3.7/site-packages/django/views/decorators/__pycache__/debug.cpython-37.pyc freenas-boot/ROOT/11.3-U3.2:/data freenas-boot/ROOT/11.3-U3.2:<0xffffffffffffffff> freenas-boot/ROOT/11.3-U4.1@2020-09-18-18:07:25:<0x0> freenas-boot/ROOT/11.3-U4.1@2020-09-18-18:07:25:/usr/local/lib/python3.7/site-packages/django/core/__pycache__ freenas-boot/ROOT/11.3-U4.1@2020-09-18-18:07:25:/ freenas-boot/ROOT/11.3-U4.1@2020-09-18-18:07:25:/usr/local/x86_64-portbl-freebsd11.0/bin/ld.gold freenas-boot/ROOT/11.3-U4.1@2020-09-18-18:07:25:/root freenas-boot/ROOT/11.3-U4.1@2020-09-18-18:07:25:/usr/local/lib/python3.7/site-packages/django/contrib/auth/__pycache__ freenas-boot/ROOT/11.3-U4.1@2020-09-18-18:07:25:/usr/local/lib/python3.7/site-packages/mako/__pycache__/cmd.cpython-37.pyc freenas-boot/ROOT/11.3-U4.1@2020-09-18-18:07:25:/usr/local/lib/python3.7/site-packages/onedrivesdk/model/__pycache__ freenas-boot/ROOT/11.3-U4.1@2020-09-18-18:07:25:/usr/local/lib/python3.7/site-packages/middlewared/plugins/__pycache__ freenas-boot/ROOT/11.3-U4.1@2020-09-18-18:07:25:/usr/local/lib/python3.7/site-packages/nacl/__pycache__ freenas-boot/ROOT/11.3-U4.1@2020-09-18-18:07:25:/usr/local/lib/python3.7/site-packages/django/contrib/sessions/backends/__pycache__ freenas-boot/ROOT/11.3-U4.1:<0x0> //usr/local/share/smartmontools //usr/local/lib/python3.7/site-packages/django/contrib/auth/migrations //usr/local/lib/python3.7/site-packages/cryptography/hazmat/backends //usr/local/lib/python3.7/xml //usr/local/lib/python3.7/site-packages/django/conf/locale/sv //usr/local/lib/python3.7/site-packages/iocage_lib/__pycache__/ioc_clean.cpython-37.opt-1.pyc //usr/local/lib/python3.7/site-packages/django/conf/locale/ta //usr/local/lib/python3.7/site-packages/gitdb/utils //usr/local/lib/python3.7/site-packages/asyncssh/__pycache__/sftp.cpython-37.pyc //usr/local/www/dojo/dojox/xml //usr/share/misc/magic //usr/local/lib/python3.7/site-packages/paramiko/__pycache__/sftp.cpython-37.opt-1.pyc //usr/local/www/freenasUI/tasks //usr/local/www/freenasUI/system //usr/local/www/dojo/dojox/widget/nls/ro //usr/local/lib/python3.7/site-packages/django/contrib/auth/tests/__pycache__ //usr/share/misc/termcap //usr/local/share/git-core/templates/hooks/fsmonitor-watchman.sample //usr/local/lib/python3.7/site-packages/django/contrib/sitemaps/__pycache__ //usr/local/lib/python3.7/site-packages/git/test/performance //usr/local/lib/python3.7/site-packages/django/conf/locale/is/__pycache__ //usr/local/lib/python3.7/site-packages/middlewared/etc_files/__pycache__ //usr/local/lib/python3.7/site-packages/botocore/data/neptune/2014-10-31/service-2.json //usr/local/lib/python3.7/site-packages/git/test/performance/__pycache__ freenas-boot/ROOT/11.3-U4.1:<0x30556> freenas-boot/ROOT/11.3-U4.1:<0x30557> //usr/local/pydevd_attach_to_process/winappdbg/plugins //usr/local/lib/python3.7/site-packages/django/utils/__pycache__/version.cpython-37.opt-1.pyc //usr/local/lib/python3.7/site-packages/django/conf/locale/tr/__pycache__ //usr/local/lib/python3.7/site-packages/django/contrib/staticfiles/management/commands //usr/local/lib/python3.7/site-packages/django/contrib/staticfiles/management/commands/__pycache__ //usr/local/www/dojo/dojox/widget/gauge //usr/local/lib/python3.7/site-packages/botocore/data/organizations/2016-11-28/service-2.json //usr/local/lib/python3.7/site-packages/aiohttp/__pycache__/http_exceptions.cpython-37.opt-1.pyc //usr/local/lib/migrate93/django/test //usr/local/lib/migrate93/django/test/__pycache__ //usr/local/www/dojo/dojox/widget/nls/sk //usr/local/www/freenasUI/tools/__pycache__ //usr/local/lib/python3.7/site-packages/django/contrib/staticfiles/templatetags/__pycache__ //usr/local/share/glib-2.0/codegen/__pycache__/utils.cpython-37.opt-1.pyc //usr/local/lib/python3.7/site-packages/aiohttp/__pycache__/log.cpython-37.opt-1.pyc //usr/local/lib/python3.7/site-packages/ws4py/server //usr/share/locale/is_IS.ISO8859-1 //usr/local/lib/python3.7/site-packages/django/conf/locale/km //usr/local/lib/python3.7/site-packages/aiohttp/__pycache__/payload_streamer.cpython-37.opt-1.pyc //usr/local/lib/python3.7/site-packages/requests_toolbelt/utils //usr/local/lib/python3.7/site-packages/botocore/data/pinpoint/2016-12-01/service-2.json //usr/local/lib/perl5/5.30/Tie //usr/local/lib/python3.7/site-packages/zettarepl/snapshot/__pycache__ //usr/local/lib/python3.7/site-packages/xattr-0.9.6-py3.7.egg-info //usr/local/lib/python3.7/site-packages/django/conf/locale/kn/__pycache__ //usr/local/lib/python3.7/site-packages/django/db/models/sql //usr/local/lib/python3.7/site-packages/git/test/fixtures //usr/local/lib/python3.7/site-packages/middlewared/etc_files/local //usr/local/lib/python3.7/site-packages/jsonschema/__pycache__ //usr/local/lib/python3.7/site-packages/botocore/data/lex-models/2017-04-19/service-2.json //usr/local/lib/python3.7/site-packages/nacl/bindings/__pycache__ //usr/local/lib/perl5/5.30/Unicode/Collate/CJK //usr/local/lib/python3.7/site-packages/samba/dcerpc/drsuapi.so //usr/local/lib/python3.7/site-packages/middlewared/etc_files/local/nut //usr/local/libexec/freenas-debug //usr/local/www/dojo/dojox/widget/nls/th //usr/local/lib/python3.7/site-packages/south/tests/brokenapp/migrations/__pycache__ //usr/local/www/freenasUI/common/__pycache__ //usr/local/lib/python3.7/site-packages/django/forms //usr/local/lib/python3.7/site-packages/zettarepl/transport/__pycache__ //usr/share/locale/ja_JP.UTF-8 //usr/local/lib/migrate93/freenasUI/account root@Brisingr[~]#
Now why it says REMOVED or replacing in there? One thing that i thought of myself is that it could be because of power shortages, we had 3 outages in a short span of time. Is it from that? I have no idea but it's a possibility, nothing got changed, the case that holds it all didn't move to my knowledge. The last time i did zpool status command, it showed both drives online and everything was working fine. But those outages are why i bought the UPS and it's plugged into that at the moment.
If anyone has any idea how this happened i'm open to ideas. The freenas box was updated to FreeNAS-11.3-U4.1 from 11.3-U3.2 in september, i still have the DB file and a compressed tar from then as it was recommended to backup those. At the time i checked after the update and all commands came back positive, showing everything online, so i don't think the update is the problem.
If more information is needed, just ask i'll do my best. Still new at all this and have not encountered this before.