Replaced One Drive, Resilvering, Now Multiple Failed AND Data Corruption!?

[dak180]

I had added the e-mail and changed default config from 1 to 0, then setup the cron. I click run now on the cron and that's the e-mail I get.
It created the report.conf file, I see it in the directory. E-mail address still is in there, and defaultconfig is still set to 0.

Can you post the command line in the cron task?

 

[isopropyl]

Can you post the command line in the cron task?

Code:

/root/report-script/report.sh -c /root/report-script/report.conf

 

[dak180]

@isopropyl if you are getting an email with Please edit the config file for your setup then your config file is not passing this check which likely means one of a few things: permission issues or a malformed file (the var name no longer matches). Try deleting your current config, generating a new one and using nano to edit the new config file in place and then try again.

 

[isopropyl]

@isopropyl if you are getting an email with Please edit the config file for your setup then your config file is not passing this check which likely means one of a few things: permission issues or a malformed file (the var name no longer matches). Try deleting your current config, generating a new one and using nano to edit the new config file in place and then try again.

I already tried it, even from scratch. However it could likely be my process order. Here's how I did it.

wget the raw git file
chmod +x report.sh
now I have the report.sh
vim report.sh
add my email, and change defaultconfig from 1 to 0, save
/root/report-script/report.sh -c /root/report-script/report.conf

I see the same "please edit the config file for your setup" print in CLI
And like I said I get it to the e-mail so I know it is executing and sending me the e-mail at minimum, however not properly because that is all I see in the e-mail.

It's probably something stupid I'm doing that I'm missing here.

 

[BarefootWoodworker]

@isopropyl if you are getting an email with Please edit the config file for your setup then your config file is not passing this check which likely means one of a few things: permission issues or a malformed file (the var name no longer matches). Try deleting your current config, generating a new one and using nano to edit the new config file in place and then try again.

Off-topic, and I apologize. . .

I love this report! I just got it set up on my TrueNAS. I've been looking for something like this for a while.

 

[BarefootWoodworker]

I already tried it, even from scratch. However it could likely be my process order. Here's how I did it.

wget the raw git file
chmod +x report.sh
now I have the report.sh
vim report.sh
add my email, and change defaultconfig from 1 to 0, save
/root/report-script/report.sh -c /root/report-script/report.conf

I see the same "please edit the config file for your setup" print in CLI
And like I said I get it to the e-mail so I know it is executing and sending me the e-mail at minimum, however not properly because that is all I see in the e-mail.

It's probably something stupid I'm doing that I'm missing here.

You don't edit the "report.sh" file. You'll edit the "report.conf" file that gets generated after the first run.

So you'll do this:
wget the raw git file
chmod +x report.sh
now I have the report.sh
/root/report-script/report.sh -c /root/report-script/report.conf
vim report.conf
add my email, and change defaultconfig from 1 to 0, save

 

[dak180]

now I have the report.sh
vim report.sh
add my email, and change defaultconfig from 1 to 0, save

You don't edit the "report.sh" file. You'll edit the "report.conf" file that gets generated after the first run.

This is the first thing that came to mind when I saw your steps.

 

[isopropyl]

You don't edit the "report.sh" file. You'll edit the "report.conf" file that gets generated after the first run.

So you'll do this:
wget the raw git file
chmod +x report.sh
now I have the report.sh
/root/report-script/report.sh -c /root/report-script/report.conf
vim report.conf
add my email, and change defaultconfig from 1 to 0, save

Yeah you're right, let me try

 

[isopropyl]

Ok I did it, however now I am not getting the e-mail.
If I just try to run the report script ./report.sh
it prints "please specify a config location, if none exist one will be created"
However I see the report.conf it already made when I ran /root/report-script/report.sh -c /root/report-script/report.conf and I edited the email and set defaultconfig to 0 and saved.

And yes I completely removed both files and started from scratch wget ing the file

 

[dak180]

If I just try to run the report script ./report.sh
it prints "please specify a config location, if none exist one will be created"

Please try to follow the Getting Started directions as closely as possible.

 

[BarefootWoodworker]

Looking at your drive models at the beginning of this thread, you have some SAS drives in there (only the 3 Disk ATA drives come back as models that are SATA).

Code:

Disk HGST HUS726040AL4210 NHGAJWEY is DEGRADED
Disk SEAGATE ST4000NXCLAR4000 Z1Z2428B 0000C4118VDG is DEGRADED
Disk ATA ST20000NM007D-3D ZVT5JR8S is DEGRADED
Disk ATA ST20000NM007D-3D ZVT5J3MY is DEGRADED
Disk ATA ST20000NM007D-3D ZVT5JPF5 is DEGRADED
Disk HGST HUS726040AL4210 NHG9ZP7Y is DEGRADED
Disk HGST HUS726040AL4210 NHG9JAAY is DEGRADED

Make sure you have this set correctly in your "/root/report-script/report.conf" file:

Code:

includeSAS="true"

I believe in the default config file it is "false".

I'm just curious. . .are you running SAS and SATA in the same pool and on the same controller/expander?

 

[help!]

is there not an app for this

/root/report-script/report.sh -c /root/report-script/report.conf

I literaly have no idea how to get access to root.

ive tried making shares and changing the locaton of sytem data set but i cant


so do you need two TNS instances incase one break so you can browse the file system?
thanks

 

[dak180]

I literaly have no idea how to get access to root.

ive tried making shares and changing the locaton of sytem data set but i cant

None of this should be done through shares; all of this should be done via ssh and scp.

 

[isopropyl]

I'm just curious. . .are you running SAS and SATA in the same pool and on the same controller/expander?

Yes, the 3 in that one vdev are 20tb SATA drives. They were running fine zero issues for about a month and half and then one of them started saying it could not read SMART data on da17 (one of those). I ended up RMAing it after a few weeks, then during the resilver process it started giving me this issue and showing a ton of other drives as degraded.

It's been about a month and a half it's been like this, because I've been so busy with work. However I hadn't notced the SMART data on da17 issue since replacing it, despite all the other drives showing degraded. However, about a week ago, it randomly started showing that same error again, alongside all the other new ones.

All the other drives are SAS drives.
I have x2 LSI cards (IT MODE) in my system, there is I think 48 bays. So there are bays on the front of the system, and the back. The 3 20TB drives are on the back, and all the others are currently on the front.
However like I said, everything was working fine for like a month after adding in the 20tb drives, except that SMART could not read on da17 error, which I then replaced that specific drive with RMA, and then that's when it all went to hell.

I will check the include SAS line in the script thanks

Code:

CRITICAL
Pool PrimaryPool state is DEGRADED: One or more devices has experienced an error resulting in data corruption. Applications may be affected.
The following devices are not healthy:

    Disk HITACHI HUS72604CLAR4000 K4K7KU5B is UNAVAIL
    Disk HITACHI HUS72604CLAR4000 K4K6EM8B is UNAVAIL
    Disk HGST HUS726040AL4210 NHG9JAAY is DEGRADED
    Disk HGST HUS726040AL4210 NHGAJWEY is DEGRADED
    Disk ATA ST20000NM007D-3D ZVT06KLB is DEGRADED
    Disk ATA ST20000NM007D-3D ZVT5J3MY is DEGRADED
    Disk ATA ST20000NM007D-3D ZVT5JPF5 is FAULTED
    Disk HGST HUS726040AL4210 NHG9ZP7Y is DEGRADED
    Disk SEAGATE ST4000NXCLAR4000 Z1Z2428B 0000C4118VDG is DEGRADED

2024-02-10 10:35:17 (America/New_York)
Dismiss
notifications_active
CRITICAL
Device: /dev/da17 [SAT], failed to read SMART Attribute Data.
2024-02-10 10:34:36 (America/New_York)
Dismiss

Please try to follow the Getting Started directions as closely as possible.

I did like 3 times however I will look again

 

[isopropyl]

Please try to follow the Getting Started directions as closely as possible.

- I ran wget on the raw git file
- chmod +x on script
- placed in it's own folder right in root dir just called report-script/
- pwd = /root/report-script/
- went back to the cron job menu on TrueNAS
- hit run now
- I get the e-mail stating to edit the config file
- I go back into the dir and look, and I see the conf file is created
- I change defaultFile from 1 to 0, add my email, and enabled SAS. Saved.
- Went back to crong job menu on TrueNAS, hit run now
- No e-mail received anymore

It could be something stupid small I'm missing. Not sure what though, and not seeing it on the docs for the script.

 

[dak180]

It could be something stupid small I'm missing. Not sure what though, and not seeing it on the docs for the script.

And just to make sure: you have set up email notifications in the gui and tested that it will send an email to the address you listed in the config file?

 

[isopropyl]

And just to make sure: you have set up email notifications in the gui and tested that it will send an email to the address you listed in the config file?

Yes, and I get an e-mail's from the cron task and regular emails for scrubs and errors and stuff

TrueNAS: CronTask Run
/bin/sh: /root/report-script/report.sh: Permission denied (this was before I chmod +x it)

TrueNAS: CronTask Run
Please edit the config file for your setup
(this is when I ran it to create the .conf file, I then edited the .conf file and after saved, and then ran the script again and no e-mail since that)

 

[BarefootWoodworker]

Device: /dev/da17 [SAT], failed to read SMART Attribute Data.

I'm starting to wonder if the script isn't horking on this.

If you run the command "./report.sh -c report.conf" from the CLI within the directory, what's the output?

 

[isopropyl]

I'm starting to wonder if the script isn't horking on this.

If you run the command "./report.sh -c report.conf" from the CLI within the directory, what's the output?

No output prints, it just hangs for a second.

Also why quote the da17 line? I don't think that has anything to do with why the script itself isn't working

 

[dak180]

I'm starting to wonder if the script isn't horking on this.

It should just continue even if it cannot read anything.

Yes, and I get an e-mail's from the cron task and regular emails for scrubs and errors and stuff

Check the settings in the gui and make sure that sqlite3 /data/freenas-v1.db 'select em_fromemail from system_email;' gets you the same address as it sends from in the mails you are getting.