How to find offline drive?

[Airgert]

Hello I have this Raid with 72x drives.
And it says one drive is unavailable

Also in the status review.

But all the drives are also in both these lists as ONLINE.
How can I find which drive this is, it's saying is offline?

 

[Ericloewe]

But all the drives are also in both these lists as ONLINE.

I don't follow. Your screenshots show a single drive that's not operating correctly.

How can I find which drive this is, it's saying is offline?

In your situation, it's easiest to start doing something, like a scrub, and look at which disk's LED does not light up.

 

[probain]

I'm really hoping that it's not a single VDEV 72 drives wide

 

[Airgert]

I don't follow. Your screenshots show a single drive that's not operating correctly.

In your situation, it's easiest to start doing something, like a scrub, and look at which disk's LED does not light up.

The screenshot doesn't show all 72 drives. But if you would see it, there is not drive missing in the rest of the list.

 

[Airgert]

I'm really hoping that it's not a single VDEV 72 drives wide

Your hope is missplaced
It is indeed 72 drives linked together with Z3

 

[Airgert]

Something that might be usefull would be something like this:
sesutil locate all online on
Of course this should be a feature in the GUI, strange thing to not have in the GUI. Such a useful command.

 

[Ericloewe]

Your hope is missplaced
It is indeed 72 drives linked together with Z3

A single vdev? That's a recipe for disaster, how is your system even functional?

sesutil locate all online on

Handy little snippet, I'll have to remember that option for when a drive outright dies.

Of course this should be a feature in the GUI, strange thing to not have in the GUI. Such a useful command.

GUI support for chassis operations is limited to iXsystems hardware, due to difficulties in supporting arbitrary systems.

 

[Airgert]

A single vdev? That's a recipe for disaster, how is your system even functional?

Handy little snippet, I'll have to remember that option for when a drive outright dies.

GUI support for chassis operations is limited to iXsystems hardware, due to difficulties in supporting arbitrary systems.

It's performing at 1200MB/s with ease.

One thin with the command line, it does not exist.
This is a "wish" command I think should be created.
sesutil locate all online on

The command that is actually real is this one.
sesutil locate all on


But I have closure now.
I have not found what Dev XX is gone missing.
But I found the location of it in the box.
I did a slightly more aggressive method of getting all drives to blink constantly.
I just pulled out one drive and replaced it, and did a Replace / Resilver on it.
All light started blinking as expected, except one, so now I can pull it and replace.

Thanks for you input.

 

[Ericloewe]

Do us a favor and post the output of zpool status

 

[Airgert]

Sure, no problem.
Have in mind that drive 57 is being resilvered.
Since I pulled it out intentionally to see what drive was not blinking in the rebuild/resilver.

Here goes:

Code:

  pool: RaidZ3_3TB
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Oct 31 02:43:19 2023
    59.5T scanned at 5.18G/s, 36.5T issued at 3.18G/s, 116T total
    519G resilvered, 31.35% done, 07:09:16 to go
config:

    NAME                                              STATE     READ WRITE CKSUM
    RaidZ3_3TB                                        DEGRADED     0     0     0
      raidz3-0                                        DEGRADED     0     0     0
        gptid/2f0d1d03-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/31b8e406-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/34f4604a-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/34ddb08e-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/44318f33-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/4886c84f-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/46213b5e-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/4fc66b8a-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/50adf8a4-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/5097632b-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/5fa0366b-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/5e06e9a2-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/6130b3f7-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/621158b8-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/6b127062-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        14410375281866171935                          UNAVAIL      0     0     0  was /dev/gptid/7199917b-6e5e-11ee-b8f3-002590c32028
        gptid/846aaa90-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/8959f3cb-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/8cf1f08e-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/8efd3ed3-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/93dc95ea-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/95de0c4b-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/999e1f66-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/9c3d9eb5-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/a325ce0b-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/a3f8ab3f-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/a58c1f45-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/a5b66f5a-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/a68b10c6-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/a9119d42-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/aab667e9-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/abece74b-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/aed8a90b-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/aeb8e340-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/b29ac6b3-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/b2a5a82f-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/b81ef33c-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/c1ade0c1-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/bec8db4f-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/bef7c974-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/d7c693d7-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/dbbb11e8-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/e07b1508-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/e4682576-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/e42925ee-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/fa8d8e20-6e5e-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/0574c092-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/009d158b-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/0d3b3d1f-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/0e112856-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/1305d6a8-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/1584efc0-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/1c87ec5c-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/21c8f4ba-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/367bdf13-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/393da872-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/623ace3c-7709-11ee-aaa4-002590c32028    ONLINE       0     0     0
        replacing-57                                  DEGRADED     0     0     0
          gptid/cbedf7e3-71b2-11ee-aaa4-002590c32028  REMOVED      0     0     0
          gptid/eb0bb6e7-77d1-11ee-8e1a-002590c32028  ONLINE       0     0     0  (resilvering)
        gptid/461f14e6-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/4066c285-71b3-11ee-aaa4-002590c32028    ONLINE       0     0     0
        gptid/5f728604-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/611430b1-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/6554511e-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/67477cdc-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/60fdd13a-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/66215176-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/629f7ebf-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/6776a184-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/74f85e0c-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/75dd2169-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/7764043a-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0
        gptid/78992542-6e5f-11ee-b8f3-002590c32028    ONLINE       0     0     0

errors: No known data errors

Mod - Added code tags on your output.

 

[Ericloewe]

It's not a record, but it's certainly the widest vdev I've seen in a pool that's actually up and running. Expect performance to get miserable as the pool fills up... But holy crap, this is somehow sort of chugging along at a decent pace. ZFS does work miracles...

Were the disks free or something? For 200ish TB, you could easily have a much smaller system using 20 TB drives, with 22 TB drives you could even almost cram it all into just 12 disks/2U.

Since I pulled it out intentionally to see what drive was not blinking in the rebuild/resilver.

Next time you can just issue a scrub with zpool scrub poolname, even with RAIDZ3 I wouldn't want to pull out a working disk while the pool is already missing a disk.

 

[Airgert]

I have 2x of these boxes.
One with 72x 3TB mixed drives
and the other
with 72x : 48x4TB and 24x 8TB drives. One pool, 2devs. And it's stable and nice.

The 3TB drives are some 10years old so I have been expecting some of them to fail in the recent future.

But yes, these were "free" drives lying around the office. At some point we might switch them out for 20TB in all of it.
So 144x 20TB drives.

But the read speed internally is good, scanned at 5.18G/s, and you need around 50x 7200rpm drives to get that speed.