Volume 1 state is Degraded. One or more devices has been removed

[bernd67]

Hello Forum,

Freenas Critical State, degraded..Volume 1 state is Degraded. One or more devices has been removed by the Administrator.
Sufficient replicas exists for the pool to continue functioning in a degraded state.
i do not know hard disk manufacturer as i am remote, 2 TB. Serial Nr beginning with S2H7J....
2 Hard disks in NAS.
I did not really have removed one so far...:>))
Freenas Version 11.1-U7

I think the 2 hard disks have been mirrored...

Is one hard disk defect and needs to be replaced?

But ... under Section Hard Disk i can see:
ada 0
ada 1
?

How to perform Change exactly?

What is the difference between Freenas and Truenas, is there a possibility to upgrade directly without
new setup on NAS?
IS Freenas 11.1-U the last Version of Freenas?

If i use "action: Online the device using 'zpool online'" in terminal with zpool online Volume1......i am told to use zpool - e ...?


Log:
Checking status of zfs pools:
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
Volume1 1.82T 779G 1.06T - 19% 41% 1.00x DEGRADED /mnt
freenas-boot 14.2G 9.83G 4.42G - - 69% 1.00x ONLINE -

pool: Volume1
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0 in 0 days 02:37:23 with 0 errors on Sun Feb 27 02:37:24 2022
config:

freenas.local kernel log messages:
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 426266, size: 4096
> ahcich0: Timeout on slot 27 port 0
> ahcich0: is 00000000 cs e000007f ss f800007f rs f800007f tfd 40 serr 00000000 cmd 0004dd17
> (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 c0 e3 20 40 16 00 00 01 00 00
> (ada0:ahcich0:0:0:0): CAM status: Command timeout
> (ada0:ahcich0:0:0:0): Retrying command
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 426266, size: 4096
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 26173, size: 4096
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 8810, size: 8192
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 196574, size: 4096
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 445875, size: 4096
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 426266, size: 4096
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 26173, size: 4096
..
...

Maybe a Boot Problem with USB Stick? Should i not reboot now?
Thx for Information.
bernd

 

[bernd67]

Rebooted, was able to reboot.. same error after Reboot..

which commands should i run in console now to get more informations?

 

[bernd67]

How to find out which Hard disk need to be replaced and how to perform that?
In

8. Storage — FreeNAS®11.1-U7 User Guide Table of Contents

www.ixsystems.com

My Version installed is 11.1 U7
i see:
8.1.10. Replacing a Failed Drive
Before physically removing the failed device, go to Storage ‣ Volumes. Select the volume name. At the bottom of the interface are several icons, one of which is Volume Status. Click the Volume Status icon and locate the failed disk.

.. but i can not find that section on web interface.. : "Storage ‣ Volumes."
i do only find:

Speicher
Datenträger
/mnt/Volume1
/mnt/Volume1/DS1
/mnt/Volume1/jails
Volume Manager
Import Disk
Import Datenträger
Zeige Datenträger
Zeige Festplatten

.. and Volume Manager is not configured as i see...

Checking status of zfs pools:
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
Volume1 1.82T 814G 1.02T - 18% 43% 1.00x DEGRADED /mnt
freenas-boot 14.2G 9.83G 4.42G - - 68% 1.00x ONLINE -

pool: Volume1
state: DEGRADED
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://illumos.org/msg/ZFS-8000-9P
scan: scrub repaired 0 in 0 days 02:37:23 with 0 errors on Sun Feb 27 02:37:24 2022
config:

NAME STATE READ WRITE CKSUM
Volume1 DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
gptid/eb0192da-0536-11e4-bcda-f46d045fb858 ONLINE 0 0 0
gptid/eb986d1e-0536-11e4-bcda-f46d045fb858 DEGRADED 0 0 391 too many errors

errors: No known data errors

 

[bernd67]

I wonder why, in reports / disks / disk busy ada0 and ada1 are the same color
and the same picture all in all, both in action as it seems?
are the disks ok and only the zpool over the disks has a problem?

is ther any possibility to get information about running hours of each disks and health status?

i performed smartctl -t long /dev/ada0 on command line of Terminal yesterday evening
but not sure where to look for the output...

 

[bernd67]

Where can i see the result after this Test?:

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-li
ne mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line
mode" successful.
Testing has begun.
Please wait 345 minutes for test to complete.
Test will complete after Wed Mar 2 23:12:49 2022

 

[NugentS]

If you are wondering why you haven't had a response - try reading up on the forum rules and correcting the obvious issues
As to your last question - google smartctl on freebsd
Lastly - you are on a very old version of FreeNAS

 

[bernd67]

• Motherboard make and model
• CPU make and model
• RAM quantity
• Hard drives, quantity, model numbers, and RAID configuration, including boot drives
• Hard disk controllers
• Network cards

Chembro ES30068 Mini ITX
AT5NM10T-I
Dual Core D525
1,8 GHZ
Hard Disks, 2, Samsung HD204UI
Raid / Mirroring i think
Network Card not known

 

[NugentS]

RAM?
And find out what network card you are using - I certainly can't tell.

Your second disk is failing / has failed
gptid/eb986d1e-0536-11e4-bcda-f46d045fb858 DEGRADED 0 0 391 too many errors

Replace it
Typing glabel status should (it is an old version of freenas, with which I am not familiar) tell you which disk it is

Actually those are chksum errors. Try powering off the NAS, then replace / reseat the cable to the drive. That MIGHT solve some issues

 

[bernd67]

Thx a lot for your help so far!

Glabel / GPTID:

[root@freenas ~]# glabel status
Name Status Components
gptid/ee3bb4c5-9966-11e7-8b3f-f46d045fb858 N/A da0p1
gptid/e3a239df-9309-11e7-b6c3-54a050800643 N/A da1p1
gptid/e3aae27d-9309-11e7-b6c3-54a050800643 N/A da1p2
gptid/eb986d1e-0536-11e4-bcda-f46d045fb858 N/A ada0p2
gptid/eb0192da-0536-11e4-bcda-f46d045fb858 N/A ada1p2
[root@freenas ~]#


pciconf -lv | grep -A1 -B3 network

Network Card?:

re0@pci0:3:0:0: class=0x020000 card=0x84321043 chip=0x816810ec rev=0x06 hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet

Sata Controller:

atapci0@pci0:2:0:0: class=0x010185 card=0x824f1043 chip=0x2362197b rev=0x10 hdr=0x00
vendor = 'JMicron Technology Corp.'
device = 'JMB362 SATA Controller'
class = mass storage
subclass = ATA

Hard Disk Problem:

show disks tells me
ada0 serialnr s2h7j90b83160..
ada1 serialnr s2h7j90b83161...

so how to get the relation betwen disk ada0 /1 and failed gptid/eb986d1e-0536-11e4-bcda-f46d045fb858 disk?
is ada1 serialnr s2h7j90b83161...the one with the problem cause its the second one listed?
and in glabel status told with gptid/eb986d1e-0536-11e4-bcda-f46d045fb858 N/A ada0p2 ?
but what does p2 on the end means?

To replace, do i need the same Hard Disk Model/ Vendor as before or can i use any 5400
Hard Disk with 2 TB (for NAS?)
Or can i use a bigger one too maybe another Model?
So i could use it later on in new NAS..?

Like:
6000GB WD Red Plus WD60EFZX NAS - 3,5" Serial ATA-600 Festplatte
4000GB WD Red Plus WD40EFZX NAS - 3,5" Serial ATA-600 Festplatte
2000GB WD Red Plus WD20EFZX NAS - 3,5" Serial ATA-600 Festplatte

so i would use only 2 TB of the bigger ones..

How to perform the Hardware Change, which Partionlayout before?
How to get on if used Hard disk is used to exchange?


RAM:
how to get more info to ram?
devinfo -rv grep ...?

how to get one page after another in terminal, not found in google..:>))

[root@freenas ~]# sysctl hw|egrep 'hw.(phys|user|real)'
hw.physmem: 4245053440
hw.usermem: 1097564160
hw.realmem: 5368709120
[root@freenas ~]#

i do not find out vendor..:>)) Information.

New Hardware?:

i do know i have to upgrade to truenas most recent version
but therefor i have first to look after better hardware, with more power.
any suggestions for that? not need to many energy, small, ..

 

[NugentS]

1. p1 = partition, just ignore it. Its the /dev/da'n' that counts
2. In theory any HD of the same size or better, but you will only be able to use capacity = the previous disk
3. Memory should be displayed on the dashboard (but I am not familiar with V11)

As for upgrades - highly dependant on usecase, budget (money and power) so no-one here will make a suggestion.
Now if you came up with a list - we might comment. But you have to do the work.

The only things that people on the board might say are:
1. Server Hardware is better than gamer/consumer hardware. Thats not to say it won't run on gamer gear
2. System from IX systems are garuanteed to work
3. For god's sake, upgrade
4. ECC is good (but not 100% nessesary, depending on use case)

 

[bernd67]

Thx!
gptid/eb986d1e-0536-11e4-bcda-f46d045fb858 DEGRADED 0 0 391 too many errors

gptid/eb986d1e-0536-11e4-bcda-f46d045fb858 N/A ada0p2

So. .
ada0 ist the one with errors to be exchanged
....
ada0 serialnr s2h7j90b83160..

Correct?

How to perform exchange then.. .

And whats Differenzen Using new hard disk or a used one... ? In re placement process?

 

[NugentS]

The manual will tell you
The drive may be fine - it could be cabling issues.

 

[bernd67]

Actually those are chksum errors. Try powering off the NAS, then replace / reseat the cable to the drive. That MIGHT solve some issues

--- i die Reboot and re connected cable but same Situation... In a next step in only could exchange the network cable.. .

But in thought checksum errors are disk related.. . ?

Is there any terminal command to bringen Pool again or to try it after for example cable exchange...or should freenas find bis Pool Inself up and running again?

 

[bernd67]

If it would or should bei a network cable Problem.. Why Gui ist then reachable without Problems and still able to write Data on NAS?

 

[bernd67]

As of:

https://illumos.org/msg/ZFS-8000-9P

Find the device with a non-zero error count for READ, WRITE, or CKSUM. This indicates that the device has experienced a read I/O error, write I/O error, or checksum validation error. Because the device is part of a mirror or RAID-Z device, ZFS was able to recover from the error and subsequently repair the damaged data.

If these errors persist over a period of time, ZFS may determine the device is faulty and mark it as such. However, these error counts may or may not indicate that the device is unusable. It depends on how the errors were caused, which the administrator can determine in advance of any ZFS diagnosis. For example, the following cases will all produce errors that do not indicate potential device failure:

A network attached device lost connectivity but has now recovered
A device suffered from a bit flip, an expected event over long periods of time
An administrator accidentally wrote over a portion of the disk using another program



[root@freenas ~]# zpool status -x
pool: Volume1
state: DEGRADED
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://illumos.org/msg/ZFS-8000-9P
scan: scrub repaired 0 in 0 days 02:37:23 with 0 errors on Sun Feb 27 02:37:24 2022
config:

NAME STATE READ WRITE CKSUM
Volume1 DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
gptid/eb0192da-0536-11e4-bcda-f46d045fb858 ONLINE 0 0 0
gptid/eb986d1e-0536-11e4-bcda-f46d045fb858 DEGRADED 0 0 4.51K too many errors

errors: No known data errors




[root@freenas ~]# zpool clear Volume1
[root@freenas ~]# zpool status -x
pool: Volume1
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://illumos.org/msg/ZFS-8000-9P
scan: scrub in progress since Sat Mar 5 09:54:06 2022
155M scanned at 12.9M/s, 0 issued at 0/s, 812G total
168K repaired, 0.00% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
Volume1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/eb0192da-0536-11e4-bcda-f46d045fb858 ONLINE 0 0 0
gptid/eb986d1e-0536-11e4-bcda-f46d045fb858 ONLINE 0 0 37 (repairing)

errors: No known data errors
[root@freenas ~]#



NAME STATE READ WRITE CKSUM
Volume1 DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
gptid/eb0192da-0536-11e4-bcda-f46d045fb858 ONLINE 0 0 0
gptid/eb986d1e-0536-11e4-bcda-f46d045fb858 DEGRADED 0 0 1.85K too many errors (repairing)


Does this mean hard disk or cable error?

 

[NugentS]

could be either, so start be replacing the cable, movingthe cable to a different port

 

[bernd67]

A.) Network Adapter:

i have only one port to connect to... only one ethernet, some more usb..
so i replaced cable, and not restartet the nas, only reconnected cable,
did a ifconfig..
[root@freenas ~]# ifconfig
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
ether f4:6d:04:5f:b8:58
hwaddr f4:6d:04XXXXXX
inet 192.168.2.20 netmask 0xffffff00 broadcast 192.168.2.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

but still red warning symbol in the right on top..
so i assume its the hard disk or?
the one :
ada0 serialnr s2h7j90b83160.

Backup done on external Hard Drive.


B.
FreeNAS-11.1-U7 - On Reboot, Now - Keep No
11.1-U5 Keep Yes
What does Keep mean? is it really starting with 11.1 U7?

In 8. Storage — FreeNAS®11.1-U7 User Guide Table of Contents i read:

Before physically removing the failed device, go to Storage ‣ Volumes. Select the volume name. At the bottom of the interface are several icons, one of which is Volume Status. Click the Volume Status icon and locate the failed disk. Then perform these steps:
...

But on my Side there is only Speicher/Datenträger
Snapshot
Replizierungsaufgaben
Resilver Priority
Scrubs
Snapshots
Vmware Snapshots
Verzeichnisdienst
Freigaben
...

i do not find Volumes!

and in Volume Manager there is none defined!

Was the Volume damaged while degration process?

I assumed to have a Volume defined to mirror ada0 and 01.

How to perform Disk Exchange then?
Can i only take Disk ada0 serialnr s2h7j90b83160 out of nas
and take a new Disk in?
Whats Next?

c.) What is the biggest disk size to be managed by Freenas in TB?

 

[bernd67]

Hello Forum,

- ada0 serialnr s2h7j90b83160.. -->> put out of NAS.
- replaced that slot with a used bought HDD HD204UI 2 TB Samsung Disk.

so ada0 has a new Hardware..

web Access possible..
show disks:
ada0 S2H7JXOD10032...... (used new disk)
ada1 S2H7J90B83161.....
so ada0 has changed ?

show data disk:
Volume1 807 GIB 1 TB
Volume1 807 GIB 44%
DS1 804GIB 44%
jails 2,3 GIB
Testjail 512 MIB

So the replaced Hardware is now known by System.
How to go on now? Error Message still red.. Crititcal Volume "Volume1" "degraded"

I can still access Data via Windows Share..

If i go to "Volume Manager" a Window pops up and ask me to give a Name for it
and under available Hard Disks there is only one 2 TB Hard Disk showed.
.. and Text existing Data will be erased...so i stopped.

Questions:
1.) How to go on to get old Pool back?
2.) I think no Pool is existing to time, correct?
3.) Is Pool and Volume Manager the same? In Volume Manager Pools are showed correct?
4.) There is no Pool to time cause one hard disk failed and therefore Pool failed?
5.) Howto go on ... resilver has run i assume cause hard disk is known, but how to get pool again?

I checked Freenas Guide

FreeNAS® 11.1-U7 User Guide — FreeNAS®11.1-U7 User Guide Table of Contents

www.ixsystems.com

but there is no section called replace Disk after Pool Failure or Restore Pool?

I assume using Volume Manager Volumes are configured ? but what is the difference to "Pool" then?

Data Backup is done..
Bernd

 

[sretalla]

OK, it's time for us to see the output from zpool status -v so we can get an understanding of what's going on and what should come next.

Please post the output in code tags.

 

[NugentS]

BTW - when I said replace the cable I meant the SATA cable. Not the ethernet. And by different port - I meant different SATA port