After unexpected reboot, rrdcached is not working with error: Connection refused (status=61)

[xhenon]

Hello guys,

I must say that my problem is similar to next thread but seems not to be the same.

I have a problem with my truenas version: TrueNAS-12.0-U8
Truenas was working fine until an unexpected reboot happens, after that, I notice the folloging alert in the notification area of the webui:

Code:

Pool Nas_Pool state is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

So I was checking this pool with

Code:

zpool status -xv

and I saw the follogin:

Code:

  pool: Nas_Pool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 04:47:46 with 0 errors on Sun Mar  6 04:48:11 2022
config:

        NAME                                            STATE     READ WRITE CKSUM
        Nas_Pool                                        ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/9eb5823c-4daf-11eb-93d1-f9319e434a35  ONLINE       0     0     1
            gptid/9ef4e84a-4daf-11eb-93d1-f9319e434a35  ONLINE       0     0     1
            gptid/9f33e646-4daf-11eb-93d1-f9319e434a35  ONLINE       0     0     1
            gptid/9f4ae5ff-4daf-11eb-93d1-f9319e434a35  ONLINE       0     0     1
        spares
          gptid/3ba2edd8-f81a-11eb-970c-8d958920e406    AVAIL   

errors: Permanent errors have been detected in the following files:

        /var/db/system/rrd-c8c2ac65efb5423aa839beddcf09baf6/localhost/df-var-db-system-webui/df_complex-free.rrd

Also, I notice that in the terminal appeared the follogin error:

Code:

rrdcached plugin: Failed to reconnect to RRDCacheD at unix:/var/run/rrdcached.sock: Unable to connect to rrdcached: Connection refused (status=61)

The issue seems to be relatad with file df_complex-free.rrd.

Everiyhing seems to be working fine, but each time I reboot truenas when it finished to reboot I can see previous message several times in the terminal

Checking in the forum and on internet I cannot see anithing that can help me to solve the issue, so, can you help me with it please?

Thank you very much.

​

 

[xhenon]

Hello guys,

I must say that my problem is similar to next thread but seems not to be the same.

I have a problem with my truenas version: TrueNAS-12.0-U8
Truenas was working fine until an unexpected reboot happens, after that, I notice the folloging alert in the notification area of the webui:

Code:

Pool Nas_Pool state is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

So I was checking this pool with

Code:

zpool status -xv

and I saw the follogin:

Code:

  pool: Nas_Pool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 04:47:46 with 0 errors on Sun Mar  6 04:48:11 2022
config:

        NAME                                            STATE     READ WRITE CKSUM
        Nas_Pool                                        ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/9eb5823c-4daf-11eb-93d1-f9319e434a35  ONLINE       0     0     1
            gptid/9ef4e84a-4daf-11eb-93d1-f9319e434a35  ONLINE       0     0     1
            gptid/9f33e646-4daf-11eb-93d1-f9319e434a35  ONLINE       0     0     1
            gptid/9f4ae5ff-4daf-11eb-93d1-f9319e434a35  ONLINE       0     0     1
        spares
          gptid/3ba2edd8-f81a-11eb-970c-8d958920e406    AVAIL  

errors: Permanent errors have been detected in the following files:

        /var/db/system/rrd-c8c2ac65efb5423aa839beddcf09baf6/localhost/df-var-db-system-webui/df_complex-free.rrd

Also, I notice that in the terminal appeared the follogin error:

Code:

rrdcached plugin: Failed to reconnect to RRDCacheD at unix:/var/run/rrdcached.sock: Unable to connect to rrdcached: Connection refused (status=61)

The issue seems to be relatad with file df_complex-free.rrd.

Everiyhing seems to be working fine, but each time I reboot truenas when it finished to reboot I can see previous message several times in the terminal

Checking in the forum and on internet I cannot see anithing that can help me to solve the issue, so, can you help me with it please?

Thank you very much.

​

Hello again.

I have addition information here that I just see.

With a dmesg command I can see the following;

Code:

Failed to fully fault in a core file segment at VA 0x8013c2000 with size 0x25000 to be written at offset 0x42f000 for process rrdcached
pid 1035 (rrdcached), jid 0, uid 0: exited on signal 10 (core dumped)

and in /var/db/system/cores I can see a core dump file attached here.

 

[xhenon]

Hello guys,

I have news about this issue.

I was checking in another truenas environment that I have and I deleted the file I can see in the zpool status command (df_complex-free.rrd) and I noticed that it is recreated automatically so I did the same in my producction environment and seems the error message is gone and now don't appears when I restart truenas.

I doing an scrub in order to check others files.

Do you think guys it can be a good resolution for this issue?