Resilver Display has Stopped Automatically Updating?

[eewiz]

Hello,


I replaced a drive and the resilver display (Pool Status) updated continuously with "Completed %" and "Time Remaining" data.
I offlined another drive, shut down, replaced it, powered up and did replace on it.
Now, the resilver display is dead. It no longer updates unless I hit the refresh button and only "% done." No "Time Remaining."
Both drives are identical. They're 8 TB HGST Ultrastar HUH728080ALE601's

Does anyone know how this display is supposed to work or why it has now failed to automatically update?

Thank You

 

[joeschmuck]

Welcome to the forums. This is a good opportunity to remind people to please read and follow the forum rules. It provides some great advice on what to post when asking questions. Why? Because we are not clairvoyant. For example: What TrueNAS software is being run? This is the first most important piece of information we need. If I were to guess, I'd think you are running an early version of Scale but that is a guess and apt to be incorrect. Why should we guess?

Also you should list your hardware per the forum rules, and any other information like, is there drive activity? Have you trye to refresh the GUI?

We will offer assistance but you need to provide some information for us to help you out. We can't just guess.

Let me give you an example: You stated

I replaced a drive and the resilver display (Pool Status) updated continuously with "Completed %" and "Time Remaining" data.

and then stated

I offlined another drive, shut down, replaced it, powered up and did replace on it.

which leads me to believe that you took two drives offline. Did you just kill your pool? I hope not but if you have a RAIDZ1 then I suspect your data is all gone.

When you post the additional information, if you can open a shell window and post the output of zpool status then that shoudl answer that concern.

Good luck.

 

[eewiz]

Thank you for your reply.

Running 12.0_U8.1.
Tyan S5512 Mobo.
Intel Xeon Processor E3-1220 8M Cache 3.1 GHz.
32GB DDR3 ECC.
Boot: 512GB Samsung SS drive.
Pool: 12 WD 4TB drives.

I am replacing all of the 4TB drives with 8TB drives.
When I wrote "updated continuously with ""Completed %" and "Time Remaining" data":
I should have said; the first drive resilvered successfully in about 24 hours and all the while the Pool Status display displayed "Completed %" and "Time Remaining" data on a continuous basis throughout the 24 hours required to complete the resilver.
Let me reiterate; the first resilver was successful and the Drive Status display worked throughout.

Now, I am doing the second of twelve drives.
The resilver is moving along successfully.
BUT, now I must hit the Refresh link each and every time and only then does the "Completed %" display update.
The "Time Remaining" display is blank and remains blank, no matter how many times I hit the Refresh link.

This is not how it worked while resilvering the first drive. (Continuous Automatic Refresh)
To be clear. I have no issue with resilvering drives.
I am pointing out that the Pool Status display has failed.

Thank You

 

[joeschmuck]

In a shell type zpool status and it should provide you the actual status.

I am pointing out that the Pool Status display has failed.

You could file a ticket to report this but since version 12 has been superseded it may not get any traction.

After this drive completes you might wan to power off the system and then power it back on. Give it a fresh start before replacing the next hard drive. Good luck in replacing the rest of those drives.

 

[eewiz]

After this drive completes you might wan to power off the system

Hello

The system is not hot-swap capable.
I have to power down after each resilver is complete in order to offline then replace the next drive.
I've already tried rebooting several times during the last resilver to test that very scenario.
The display still fails.
It is possible that once the GUI has lost connection to the underlying command line software, it is not regained on reboot.
Although, the Pool Status does still respond to the Refresh link, even after a reboot during a resilver.

The 2nd resilver has just finished.
Now, it has gone totally bonkers.
When the resilver of the 2nd drive finished, the Pool Status display took off again with auto-refresh now working.
BUT, I should not have a resilver progressing at this time, it has finished resilvering the last drive.

So I did Zpool Status and here are the results.

root@shed[~]# zpool status
pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:19 with 0 errors on Thu Sep 29 03:45:19 2022
config:

NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
ada0p2 ONLINE 0 0 0

errors: No known data errors


354G scanned at 2.58G/s, 21.1G issued at 157M/s, 41.3T total
0B resilvered, 0.05% done, 3 days 04:28:59 to go
config:

NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/4a48b89b-3f98-11ed-b30c-a0423f2d97be ONLINE 0 0 0
gptid/2b250da9-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/b2a6fe0e-4126-11ed-82ef-a0423f2d97be ONLINE 0 0 0
gptid/2c0deac8-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/2c7da563-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/2d23bea8-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/99513bf4-73a0-11ec-a041-a0423f2d97be ONLINE 0 0 0
gptid/2d161187-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/600a2dce-74d7-11ec-89dc-a0423f2d97be ONLINE 0 0 0
gptid/2e0730ad-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/97300069-81c5-11ec-9c80-a0423f2d97be ONLINE 0 0 0
gptid/2e69d8ad-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0

As can be seen, it says it is resilvering something but identifies no drive as the one being resilvered.
I have run zpool status in the past and it always identifies the specific drive that is being resilvered.

At this point it appears to be seriously broken.
But, I now have no choice but to spend another 24 hours waiting for this "PHANTOM" resilver to progress to completion.

Thank You

 

[joeschmuck]

354G scanned at 2.58G/s, 21.1G issued at 157M/s, 41.3T total
0B resilvered, 0.05% done, 3 days 04:28:59 to go
config:

NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/4a48b89b-3f98-11ed-b30c-a0423f2d97be ONLINE 0 0 0
gptid/2b250da9-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/b2a6fe0e-4126-11ed-82ef-a0423f2d97be ONLINE 0 0 0
gptid/2c0deac8-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/2c7da563-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/2d23bea8-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/99513bf4-73a0-11ec-a041-a0423f2d97be ONLINE 0 0 0
gptid/2d161187-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/600a2dce-74d7-11ec-89dc-a0423f2d97be ONLINE 0 0 0
gptid/2e0730ad-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/97300069-81c5-11ec-9c80-a0423f2d97be ONLINE 0 0 0
gptid/2e69d8ad-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0

Odd, maybe someone who knows more about this result could offer some advice. My first thought was SMR. Did you burn in the hard drives before performing the drive replacement? Are the drives used? I'm trying to think of what could be going on.

But, I now have no choice but to spend another 24 hours waiting for this "PHANTOM" resilver to progress to completion.

Or over 3 days according to the zpool status message.

You could run a SMART Short test, see what the results are. If feeling froggy, do a SMART Long test if the Short test is okay. Post the results.

 

[eewiz]

Hello All

After posting the last post, I did another zpool status command.

root@shed[~]# zpool status
pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:19 with 0 errors on Thu Sep 29 03:45:19 2022
config:

NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
ada0p2 ONLINE 0 0 0

errors: No known data errors

pool: pool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sat Oct 1 18:19:33 2022
1.61T scanned at 907M/s, 892G issued at 490M/s, 41.3T total
63.9G resilvered, 2.11% done, 1 days 00:03:06 to go
config:

NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/4a48b89b-3f98-11ed-b30c-a0423f2d97be ONLINE 0 0 0
gptid/2b250da9-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/b2a6fe0e-4126-11ed-82ef-a0423f2d97be ONLINE 0 0 0 (resilvering)
gptid/2c0deac8-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/2c7da563-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/2d23bea8-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/99513bf4-73a0-11ec-a041-a0423f2d97be ONLINE 0 0 0
gptid/2d161187-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/600a2dce-74d7-11ec-89dc-a0423f2d97be ONLINE 0 0 0
gptid/2e0730ad-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0
gptid/97300069-81c5-11ec-9c80-a0423f2d97be ONLINE 0 0 0
gptid/2e69d8ad-4f6a-11ec-9072-a0423f2d97be ONLINE 0 0 0

errors: No known data errors
root@shed[~]#

NOW, it says which drive is being resilvered but, at this point, I don't know how to identify "gptid/b2a6fe0e-4126-11ed-82ef-a0423f2d97be" within the GUI so I can tell which drive is actually being resilvered now.

Software that works slower than human cognition is a recipe for confusion.

If anyone knows how to identify a drive by "gptid/b2a6fe0e-4126-11ed-82ef-a0423f2d97be" within the GUI?
Please reply.

Thank You

 

[joeschmuck]

If anyone knows how to identify a drive by "gptid/b2a6fe0e-4126-11ed-82ef-a0423f2d97be" within the GUI?

You should learn how to do searches. The command you seek is glabel status and you will have your answer.

 

[eewiz]

Thank You

root@shed[~]# glabel status
........
gptid/b2a6fe0e-4126-11ed-82ef-a0423f2d97be N/A ada2p2

root@shed[~]# zpool status
.......
gptid/b2a6fe0e-4126-11ed-82ef-a0423f2d97be ONLINE 0 0 0 (resilvering)

ada2 is the drive that I replaced 28 hours ago, and the resilvering was completed on, 4 hours ago, while the Pool Dispaly was not auto-refreshing.
At least I thought that ada2 was the drive that was being resilvered yesterday.
Now, I don't know.
If ada2 was being resilvered yesterday when the display did not work, then it is now being resilvered again while the display does work.
But this resilver is not proceeding anything like the resilver of the 1st drive (ada1).
ada1 put up a time display that started with about 24 hours to go and then proceeded to reduce time throughout the resilver to end about 20 minutes earlier than predicted.

Also, the second "PHANTOM" resilver, which should have been a resilver of ada2, took about 24 hours. Ending about 4-5 hours ago.
Now this resilver, that I have identified as actually being a resilver of ada2, responds quite differently.

As can be seen, the percentage is growing as well as the time to completion.
From about 1 day at the start to now over 2 days and still climbing.
This is nothing like the 1st resilver of ada1 and as for the 2nd "PHANTOM" resilver I don't know because the time display was nonfunctional, but it did complete in about 24 hours as well.

I just checked again and now the time display is progressing downward but, I will have to wait to see if the resilver is actually done in about 24 hours from when it started, and whether it starts to resilver ada2 again once it is done.
This is the third resilver although I have replaced only two drives so far.

All of the drives I am installing are identical 8TB 2.5 million hour MTTF enterprise class hard drives.
They're used. They were removed from a server farm after about 3-4 years of use.
Their serial numbers range from VJVXXXXX to VLHXXXXX.
Am I foolish to assume that all of them should resilver in about the same amount of time with similar reactions from the display?

All for now

 

[joeschmuck]

They're used. They were removed from a server farm after about 3-4 years of use.

So I ask again, did you test the drives out to ensure they were actually good? If so, what test(s) did you run?

Am I foolish to assume that all of them should resilver in about the same amount of time with similar reactions from the display?

Yes and no.

No, if they are all in good condition and given the same system resources then they should be close to within maybe 5 hours or less. Why 5 hours or less, just personal experience. I have four of the exact same drive models but one was manufactured about a month later, it takes about 3 hours longer to perform a SMART Long/Extended test. Why? I don't know.

Yes, if the drives were not tested and there is a surface error, reads/write will take much longer, or if the system becomes busy then it will take more time.

Used hard drives is a huge risk, unless the price is very right. And then I'd have done a full burn-in test on them to prove they were reliable.

 

[eewiz]

Thank You
The price is quite right.

Multiply by 12 drives and you can see the difference.
All I could afford was to by one drive a month since last October.
Now I have 12 drives for about $1300 instead of $4900 with tax.
$4900 is way beyond my budget.

As for testing, I figured if the drives resilver in a timely manner without error that's a pretty good test.
If an 8TB drive fails to resilver correctly, I will reinsert the 4TB drive that was displaced and resilver again.
Then I will send the 8TB back to the supplier for a replacement on their 5-Year Warranty since it would not resilver without error.
I will keep this up until all 12 drives have been replaced.

Also, the system does SMART tests daily and weekly.

After I get all 12 drives upgraded to 8TB, I will wait as many weeks as possible for SMART to flag any problems.
Then, once I am satisfied that I have all good drives, I will expand the pool from 40TB to 80TB.
The server is now 95% full but, it was last June when it went from 94% to 95%.
I should have 6 months or more, if I desire, before I will have to expand the pool and loose the ability to return to one of the twelve 4TB drives that will sit on the shelf until the expansion is complete.

But, when the display flakes out, or it takes 3 resilvers for 2 drives, with no errors.
I get confused enough to start posting to see if others might be able to shed some clarity.
Although, I am assuming that if a resilver completes successfully, that the drive was punished sufficiently that Errors: 0 means that the drive has no read or write errors.
If my assumption is not correct, then SMART will catch it down the road.

I also see that a short SMART test started at 18:00:00 and this resilver started at 18:19:33.
Maybe it was SMART getting in the way, causing the time display to raise so precipitously in the beginning of the resilver as shown above.
At this point the resilver appears to be progressing normally for a completion about 26 hours after starting.

All for now

 

[joeschmuck]

As for testing, I figured if the drives resilver in a timely manner without error that's a pretty good test.

Nope, it is not.

Then I will send the 8TB back to the supplier for a replacement on their 5-Year Warranty since it would not resilver without error.

Who provides the warranty? Amazon or HGST? Also, resilvering failing is likely not a cause for rejection, it may need to be a SMART failure indication, especially if it's the manufacturer who holds the warranty.

Although, I am assuming that if a resilver completes successfully, that the drive was punished sufficiently that Errors: 0 means that the drive has no read or write errors.

Does it Really? It could mean that the drive overcame some issues. Unless you are reading the SMART data, you just don't know if there were pending sector errors or reallocated sectors. Also you did not write to each location on the physical drive so you would not know. Think about it, if all your data fits on a 4TB drive (looking at a single drive of course) now and you are replacing with 8TB drives, at an absolute minimum half of the physical drive is not storing data. I hope that your drives are good because you did spend a lot of money on them.

If my assumption is not correct, then SMART will catch it down the road.

That is a lot of faith on a system that is only designed to catch media failures up to 24 hours before catastrophic failure (and that is not a guarantee). While it may catch some failure in advance, it also may not. There are several failures it can't predict as well.

As for your SMART Long test, how long does the test take to complete? This can be found in the SMART data where it talks about how many minutes an Extended test takes. I ask because my HGST's that are 6TB take 976 minutes if there are no other operations going on which is just under 16.5 hours, again, if nothing else is going on. SMART tests are low priority and run in the drive. If TrueNAS wants to resilver or scrub or just provide data to a user then the SMART test will take longer to complete. Why am I going over this? Because you have a SMART Short test scheduled 24 hours later. You should find out how long the the extended test takes and adjust accordingly, if needed.

Then, once I am satisfied that I have all good drives, I will expand the pool from 40TB to 80TB.

Huh? As I understand it, when you install that last drive, the pool will automatically expand. There isn't anything you have to do to make it happen. Maybe it's different with Scale?

You might take advantage of a script several of us have written in order to monitor your hard drives. See the link below in my signature.

 

[eewiz]

Hello

As I understand it.
On Core one must hit click the link "Expand Pool" and navigate the subsequent dialog to get the expansion to happen.

Because you have a SMART Short test scheduled 24 hours later.

Are you saying that starting of a SMART short test on Saturday will cancel a running SMART long test that started on Friday, providing no long test results?

According to the SmartMonTools site "https://www.smartmontools.org/ticket/40" that issue was fixed many years ago circa smartctl v-5.45.
Core 12.0-U8.1 uses smartctl v7.2.

root@shed[~]# smartctl
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p14 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

I will try to determine if my SMART long test is actually producing results and not being preempted by the short test starting a day later.
Although, I will have to research how to do that.
All I know for sure is that SMART has offlined 3 drives in the past year that I have replaced.
As for running a script, which I assume is similar to a batch file, I would have to get that file onto the server and I don't know how to do that.
It took me months of trial and error to get my two servers up and running and I never could get Putty to connect to either server.
It also appears that the Core internals cannot be SAMBA shared so I'm out of luck there as well.
After alot of work, I did successfully set up SSH Replication between the two servers as that was all GUI controlled.
I can open a shell and ask a question like smartctl or zpool or glabel but, that's about it for me.
I am a life long DOS/Windows user and know little about running a script in a UNIX/Linux console environment.
Is there a PATH in UNIX, does the file have to be in any certain directory before you run it, how would I get a file into that directory?
I settled on Truenas because it was billed as an enterprise class Appliance with GUI control.
Which all-in-all appears to me to work quite well, until......

My current resilver should be done in about an hour and I will then see if I am in a resilver loop or not.

Thank you for all of your help.

 

[eewiz]

Hello All

Resilver of the 2nd drive was successful.
I have started resilvering the 3rd drive and all looks well.
The Pool Status display is auto-refreshing.
I am beginning to surmise that the display is not flaky but, fails to update on purpose, for some unknown (to me) reason.
I just wish I knew the actual reason for the display not working during that "PHANTOM" resilver.
If it happens again I will be no closer to understanding what it means other than "all is not right"
I get the feeling that the programmers of the GUI did this on purpose to key the user to something.
I just don't know what that something is.
I wish I knew where the script that creates the GUI is stored and how to get into my Windows environment where I could examine it but, I would need a clearly written tutorial on how to do that.
Beyond general installation and setup, Truenas tutorials are few and far between.

All for now

 

[joeschmuck]

Are you saying that starting of a SMART short test on Saturday will cancel a running SMART long test that started on Friday, providing no long test results?

LOL, nope, not at all. It's just a scheduling conflict. Call me anal about things like this but if you do not expect the drive to complete the Log test, then why have another test trying to start.

 

[eewiz]

It's that GUI state of mind.
I've known for years that SMART drives test themselves.
I've never had any of my hardware make me aware of how long SMART testing might take.
I've learned that it was a continuous thing that makes the consumer not care about how long it takes.
On Windows you run a utility that monitors the SMART's of all the drives in the box and if there's trouble the utility warns you of impending drive failure with the user having no consideration of how much time SMART testing takes.

Enter Truenas; the GUI let me do it although, I was somewhat curious as to why I had to set up any kind of schedule for SMART, but then it's UNIX/Linux.
After SMART caught some failing hard drives, I figured that it must be working.
I stick my hand into the hole in the black box and if it feels fuzzy and doesn't bite, I figure it's O.K.

I dislike having to conduct research into how to do something in a UNIX/Linux console.
I settled upon Truenas due to the Internet's opinion that Truenas is an enterprise class, GUI controlled, storage box that just works.
I've tried bringing up Linux boxes on and off since my first attempt with Slackware back in the mid 90's.
I've never been able to just use Linux to do my work like I can on a Windows box.
The last thing I want to do is be forced to open a UNIX/Linux console to have to do some important task.
For me it always results in many hours of research to do what others may consider trivial.
Doing anything in a UNIX/Linux console is purely painful to me.

I can do anything on Windows box.
I'm a retired electrical engineer (embedded systems) with 40 years experience.
I don't want to have to research the equivalents to cut, paste, DIR, CD, RD, attribute etc... or to have to learn someone else's idea of how a directory structure should be laid out.
And honestly, I can't comprehend how anyone can work with any number of monochrome console text editors, all with completely different command structures.
Even back in the DOS days there was WordPerfect for text and Clipper for code, all with clearly written instruction manuals.
I design business systems based on Windows servers and workstations and I've had several customers inquire about Linux because it's free.
I'm forced say, if you want to go Linux, you are on your own, I can't help you. The learning curve is so steep that it's not really free.
Admittedly, Linux today is mostly GUI but, I still can't use it to do my work without wasting too much time researching commands in a console for some reason or another.

I do wish to thank you for all of your help.
I'm resilvering the 4th drive and all looks good.

All for now

 

[eewiz]

Hello joeschmuck

As I understand it, when you install that last drive, the pool will automatically expand.

I must apologize.
You were correct.
As soon as the resilver of the last 8TB drive was completed.
The machine immediately changed from 40TB to 80TB.
How Rude.
It appears to me now that ZFS does this whether you like it or not.
This leaves me mystified as to why the Truenas GUI even has a selection to Expand Pool.
It serves no purpose whatsoever.
If you select "Expand Pool" with mixed size drives it says that you can't expand the pool because you have mixed drive sizes.
Which of course I knew but, I thought I would see if the "Expand Pool" selection did anything useful.
If you select "Expand Pool" with matching size drives it prompts "Cancel" or "Expand Pool."
If you select "Expand Pool" it says:

Which is totally bogus since the pool was already expanded by ZFS the moment it finished resilvering the last drive.
The Truenas programmers should remove this useless menu selection.
It only serves to cause confusion.

All for now

 

[joeschmuck]

The Truenas programmers should remove this useless menu selection.

I suspect this is for an option other than replacing existing drives in a pool, maybe for adding additional drives.

 

[eewiz]

I suspect this is for an option other than replacing existing drives in a pool, maybe for adding additional drives.

That is an astute observation that I would not have considered due to the fact that my servers are full. They hold twelve 3.5 inch drives plus a thirteenth 2.5 inch boot drive. I will also be unable to test that scenario.

Thanks for the reply.

 

[joeschmuck]

I will also be unable to test that scenario.

Well you could if you felt like destroying your pool and recreating stuff. Naw, just kidding. Once you have your system working well, it's not the time to try new things.