All disk change to de DEGRADED , all new disk setup

[zhe]

yes you are right

root@truenas[~]# zpool detach HomePool e8382bbd-d369-49d7-b1e9-709fea3e6de6
cannot detach e8382bbd-d369-49d7-b1e9-709fea3e6de6: only applicable to mirror and replacing vdevs
root@truenas[~]# zpool detach HomePool e8382bbd-d369-49d7-b1e9-709fea3e6de6 -f
invalid option 'f'
usage:
detach <pool> <device>
root@truenas[~]#

only use # zpool offline

but i not want to off and out the disk . i am copying out data .....

 

[Patrick M. Hausen]

zpool detach -f <poolname> <device-id>

But please post zpool status HomePool again, first. And enclose it in code tags like so. Thanks.

Code:

this
is
    formatted
    as
code

 

[zhe]

Code:

root@truenas[~]# zpool status HoomPool
cannot open 'HoomPool': no such pool
root@truenas[~]# zpool status
^[[A  pool: HomePool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Dec  1 17:56:15 2022
        1.34T scanned at 175M/s, 669G issued at 85.5M/s, 3.64T total
        29.3G resilvered, 17.94% done, 10:10:57 to go
config:


        NAME                                        STATE     READ WRITE CKSUM
        HomePool                                    DEGRADED     0     0     0
          raidz3-0                                  DEGRADED     0     0     0
            a2a4471f-c7ef-4e31-a39c-71345f922ce8    DEGRADED     0     0   732  too many errors
            e2c94101-3f15-435f-95b4-53f26b1f5e72    DEGRADED     0     0   732  too many errors
            e96c2b49-98cc-4b72-b059-515f268ff346    DEGRADED     0     0   852  too many errors
            e8382bbd-d369-49d7-b1e9-709fea3e6de6    DEGRADED 27.2K     1   858  too many errors  (resilvering)
            e9ddfd9a-997c-4806-859d-868aa8a8c2cb    DEGRADED     0     0   852  too many errors
            1ec87af4-7b0b-4e4a-a461-b48c537f8e44    DEGRADED 1.26K     0   852  too many errors  (resilvering)
            9e0c8a31-01eb-4766-95f3-f26e682380e3    DEGRADED     0     0   852  too many errors
            b8035223-6b9c-4954-801a-07d461ee037b    DEGRADED     0     0   852  too many errors
            dba5b5d2-1b59-44bf-8144-d2b5cf9e4ac3    DEGRADED     0     0   852  too many errors
            53d01a6b-cced-48db-8a1c-918c84652846    DEGRADED 10.0K     0   852  too many errors  (resilvering)
            0e2d892d-bfb7-47a6-8210-47054905a378    DEGRADED 7.68K     2   850  too many errors  (resilvering)
            2d44d0ab-02c5-4c72-8eac-bf607ca7c672    DEGRADED     0     0   852  too many errors
            a2e69f62-a7a0-4a92-8c92-c509fca4b9fc    DEGRADED     0     0   852  too many errors
            615105ba-558f-4828-923d-eb8ba790fb22    DEGRADED     0     0   852  too many errors
            replacing-14                            DEGRADED     0     0   852
              16752858698199900939                  UNAVAIL      0     0     0  was /dev/disk/by-partuuid/a64222d9-5a9f-4e5a-b7a4-3f295a866654
              da148efe-31e2-4e0a-927d-1925378a89ed  DEGRADED     0     0     0  too many errors  (resilvering)
            2d251bba-1faf-43c5-98c9-04c13df6f4d1    DEGRADED     0     0   844  too many errors
            61d6cf52-c171-49ff-ae69-c45bf13c7059    DEGRADED     0     0   744  too many errors
            eee02552-500a-4143-8776-36158ab2385f    DEGRADED     0     0   732  too many errors
            a726ac5f-4c98-4cea-9a4c-625e337d3c08    DEGRADED    88     0   732  too many errors  (resilvering)
            cf9010f9-05a7-4d5f-9491-f1363d22e590    DEGRADED     0     0   732  too many errors
            12172b1d-8078-491f-8377-ca4c057c5811    DEGRADED     0     0   732  too many errors
            ff8f4b49-4e61-4a73-b814-f96e94b925d2    DEGRADED     0     0   732  too many errors
        cache
          b0f87678-043d-4b8d-b33c-3c71c3f58989      ONLINE       0     0     0


errors: 183 data errors, use '-v' for a list


  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:06 with 0 errors on Mon Nov 28 03:45:08 2022
config:


        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sda3      ONLINE       0     0     0


errors: No known data errors

root@truenas[~]# zpool detach -f HomePool e8382bbd-d369-49d7-b1e9-709fea3e6de6
invalid option 'f'
usage:
        detach <pool> <device>
root@truenas[~]#

i want stop the resilvering on 17.94% .....

 

[Patrick M. Hausen]

You cannot detach a disk that is part of a RAIDZ3 vdev. Only mirrors - the error message you got from your first attempt said that quite explicitly.

You can offline the disk:

Code:

zpool offline HomePool e8382bbd-d369-49d7-b1e9-709fea3e6de6

Proceed at your own risk.

 

[zhe]

if i offline the disk , i will be not go on to copying out my data ...

i need copying by all resource such as cpu and memory

 

[Patrick M. Hausen]

You need to offline the disk that it is resilvering to. Possibly I got the ID wrong. The effect will be the same as detaching, IMHO.

 

[jgreco]

A reminder that we're still waiting on details of how the disks are connected. There was a vague reference to a 24 SATA something or other. This could well be the source of the degraded status, and participants are encouraged to help try to identify what it is that is being used here. This has the feeling of a bad SATA port multiplier setup.

 

[zhe]

if i copy out finish , i will can do everthing.

 

[HoneyBadger]

A reminder that we're still waiting on details of how the disks are connected. There was a vague reference to a 24 SATA something or other. This could well be the source of the degraded status, and participants are encouraged to help try to identify what it is that is being used here. This has the feeling of a bad SATA port multiplier setup.

It's still possible that we have an HBA with an expander, especially if this is in something like an SC846 chassis - but I agree we have some manner of storage layer issue, looking at the CKSUM errors and large numbers of READ errors. We could also be looking at an overheating HBA.

@zhe please provide the output of the following two commands inside of separate blocks of [code][/code] tags:

lspci
dmesg

This will assist in identifying how your disks are connected.

 

[zhe]

cmd : lspci

Code:

root@truenas[~]# root@truenas[~]#
root@truenas[~]# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Ho                                                                                                             st Bridge/DRAM Registers (rev 05)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controlle                                                                                                             r (x16) (rev 05)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 630 (rev 04)
00:14.0 USB controller: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0                                                                                                              xHCI Controller
00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1
00:17.0 SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI                                                                                                              mode]
00:1b.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (                                                                                                             rev f0)
00:1b.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #21 (                                                                                                             rev f0)
00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (r                                                                                                             ev f0)
00:1c.7 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #8 (r                                                                                                             ev f0)
00:1d.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (r                                                                                                             ev f0)
00:1f.0 ISA bridge: Intel Corporation 200 Series PCH LPC Controller (Z270)
00:1f.2 Memory controller: Intel Corporation 200 Series/Z370 Chipset Family Powe                                                                                                             r Management Controller
00:1f.3 Audio device: Intel Corporation 200 Series PCH HD Audio
00:1f.4 SMBus: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Netwo                                                                                                             rk Connection (rev 01)
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Cont                                                                                                             roller SM951/PM951 (rev 01)
03:00.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)
04:00.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)
04:02.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)
04:03.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)
04:08.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)
04:0a.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)
04:0b.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)
05:00.0 SATA controller: ASMedia Technology Inc. Device 1064 (rev 02)
06:00.0 SATA controller: ASMedia Technology Inc. Device 1064 (rev 02)
07:00.0 SATA controller: ASMedia Technology Inc. Device 1064 (rev 02)
08:00.0 SATA controller: ASMedia Technology Inc. Device 1064 (rev 02)
09:00.0 SATA controller: ASMedia Technology Inc. Device 1064 (rev 02)
0c:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 P
root@truenas[~]#

cmd : dmesg

Code:

root@truenas[~]# dmesg

[50396.945295] ata32.00: cmd 60/00:30:78:63:bd/08:00:3d:00:00/40 tag 6 ncq dma 1048576 in

                        res 40/00:d8:98:72:2c/00:00:1b:00:00/40 Emask 0x10 (ATA bus error)

[50396.961972] ata32.00: status: { DRDY }

[50396.965908] ata32: hard resetting link

[50397.680020] ata32: SATA link up 6.0 Gbps (SStatus 133 SControl 310)

[50397.745288] ata32.00: configured for UDMA/33

[50397.749711] sd 32:0:0:0: [sdk] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=20s

[50397.759351] sd 32:0:0:0: [sdk] tag#6 Sense Key : Illegal Request [current]

[50397.766522] sd 32:0:0:0: [sdk] tag#6 Add. Sense: Unaligned write command

[50397.773381] sd 32:0:0:0: [sdk] tag#6 CDB: Read(16) 88 00 00 00 00 00 3d bd 63 78 00 00 08 00 00 00

[50397.782611] blk_update_request: I/O error, dev sdk, sector 1035821944 op 0x0:(READ) flags 0x700 phys_seg 154 prio class 0

[50397.793880] zio pool=HomePool vdev=/dev/disk/by-partuuid/e8382bbd-d369-49d7-b1e9-709fea3e6de6 error=5 type=1 offset=528193286144 size=1048576 flags=40080ca8

[50397.808140] ata32: EH complete

[50418.864262] ata53.00: exception Emask 0x10 SAct 0xa0000200 SErr 0x4010000 action 0xe frozen

[50418.872930] ata53.00: irq_stat 0x80400040, connection status changed

[50418.879529] ata53: SError: { PHYRdyChg DevExch }

[50418.884325] ata53.00: failed command: READ FPDMA QUEUED

[50418.889751] ata53.00: cmd 60/f8:48:90:20:be/07:00:3d:00:00/40 tag 9 ncq dma 1044480 in

                        res 40/00:70:78:50:fa/00:00:21:00:00/40 Emask 0x10 (ATA bus error)

[50418.906187] ata53.00: status: { DRDY }

[50418.910056] ata53.00: failed command: READ FPDMA QUEUED

not end.....

no permit to input more 3000 word here, so a save a txt-file in attachment

 

[jgreco]

03:00.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01) 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01) 04:02.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01) 04:03.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01) 04:08.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01) 04:0a.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01) 04:0b.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01) 05:00.0 SATA controller: ASMedia Technology Inc. Device 1064 (rev 02) 06:00.0 SATA controller: ASMedia Technology Inc. Device 1064 (rev 02) 07:00.0 SATA controller: ASMedia Technology Inc. Device 1064 (rev 02) 08:00.0 SATA controller: ASMedia Technology Inc. Device 1064 (rev 02) 09:00.0 SATA controller: ASMedia Technology Inc. Device 1064 (rev 02)

Why do I get the feeling that this is that bridge chip based SATA controller we were discussing in some other thread earlier this week?

 

[Ericloewe]

Yeah, I hate that lspci output, it just made my day three times worse. That looks like an ASMedia ASM1812 PCIe 2.0 switch (a part so crappy it can't decide if it's a 10-lane or 12-lane switch, by simultaneously claiming a total of 12 lanes and up to two upstream lanes and eight downstream lanes) with five ASMedia ASM 1064 PCIe 3.0 AHCI controllers, with four SATA 6Gb/s ports each.
Also, a Realtek 8111 attached to the same host as an Intel X520 is not something you see every day.

 

[zhe]

i have added the memory to 64GB .... It scared me that used 40GB....for the TureNas system....
perhps before litte memory not good working write data to 22disks. ..
i am copying out ...now ..not finished yet
when copy it finish i will reintall turenas scale again in 64GB , yes i want try it

to be continue.......

 

[zhe]

i have cpoyed all datas finish, 10%datas were lose , if i use xfs with raid6 , my data will all lose when all disks on degraded.

i have reinstall trueNas scale

all disks online

on one day

to be continue ..... test it 30 days

one day

 

[zhe]

D+2 day
have 3 disks FAULTED , stop copy in . wait to buy new 3disk replaced.
-------
by the way

Each device can fall into one of the following states:

ONLINE
The device or virtual device is in normal working order. While some transient errors might still occur, the device is otherwise in working order.

DEGRADED
The virtual device has experienced failure but is still able to function. This state is most common when a mirror or RAID-Z device has lost one or more constituent devices. The fault tolerance of the pool might be compromised, as a subsequent fault in another device might be unrecoverable.

FAULTED
The device or virtual device is completely inaccessible. This status typically indicates total failure of the device, such that ZFS is incapable of sending or receiving data from it. If a top-level virtual device is in this state, then the pool is completely inaccessible.

OFFLINE
The device has been explicitly taken offline by the administrator.

UNAVAIL
The device or virtual device cannot be opened. In some cases, pools with UNAVAIL devices appear in DEGRADED mode. If a top-level virtual device is unavailable, then nothing in the pool can be accessed.

REMOVED
The device was physically removed while the system was running. Device removal detection is hardware-dependent and might not be supported on all platforms.

The health of a pool is determined from the health of all its top-level virtual devices. If all virtual devices are ONLINE, then the pool is also ONLINE. If any one of the virtual devices is DEGRADED or UNAVAIL, then the pool is also DEGRADED. If a top-level virtual device is FAULTED or OFFLINE, then the pool is also FAULTED. A pool in the faulted state is completely inaccessible. No data can be recovered until the necessary devices are attached or repaired. A pool in the degraded state continues to run, but you might not achieve the same level of data redundancy or data throughput than if the pool were online.

 

[zhe]

D+10days
all disks are online
the problem is litte 16GB memory is not enough to run heathely ,
yes, add to 64GB memory for sys , so all issue have done

i am happy , and sthanks for your helpping of up
its have solved