No GUI, No SSH, but can use Plex VM

[ere109]

Yesterday morning I noticed that my GUI was unavailable from my phone - figured it was the phone. I later turned on a Plex jail movie with no issue, so didn't worry. Last night I watched a Plex movie, and this morning another show.
Tonight I attempted to log into my NAS - no luck. The GUI is unreachable, and an SSH command just hangs indefinitely. But I can still access my Plex, which is running on the same machine. This is a gist of my current console output (hand typing, can't copy/paste from java console):

Code:

(ahcich0: AHCI reset: device not ready after 31000ms)
"	  " Poll timeout on slot 17 port 0
"	  " is 0000000 cs 00010000 ss 00000000 rs 00020000 tfd 80...
aprobe0:ahcich0:0:0:0: DM TRIM. ACB 06 01 00 00 00 40 00 00 00...
"	  " CAM status: Unconditionally Re0queue Request
"	  " Error 5. Periph was invalidated
"	  " READ_FPDMA_QUEUED. ACB 60 70 ...
"	  " CAM status: command timeout
"	  " Error 5. Periph was Invalidated
... AHCI reset ...
... Poll timeout on slot 18 port 0

That general error continues to repeat, going up the slot count and then eventually getting to "Retries exhausted." After serious negative impact the last time my machine was improperly shut down, I don't want to randomly pull the cord. Thoughts on the problem - what this means and how I should best address it?

 

[ere109]

I can ping the IP of the Freenas:

Code:

ping 192.168.109.200
PING 192.168.109.200 (192.168.109.200) 56(84) bytes of data.
64 bytes from 192.168.109.200: icmp_seq=1 ttl=64 time=18.6 ms
64 bytes from 192.168.109.200: icmp_seq=2 ttl=64 time=4.68 ms
64 bytes from 192.168.109.200: icmp_seq=3 ttl=64 time=4.00 ms
64 bytes from 192.168.109.200: icmp_seq=4 ttl=64 time=20.7 ms
64 bytes from 192.168.109.200: icmp_seq=5 ttl=64 time=4.64 ms
64 bytes from 192.168.109.200: icmp_seq=6 ttl=64 time=6.44 ms
64 bytes from 192.168.109.200: icmp_seq=7 ttl=64 time=927 ms
64 bytes from 192.168.109.200: icmp_seq=8 ttl=64 time=9.78 ms
64 bytes from 192.168.109.200: icmp_seq=9 ttl=64 time=370 ms
64 bytes from 192.168.109.200: icmp_seq=10 ttl=64 time=331 ms
64 bytes from 192.168.109.200: icmp_seq=11 ttl=64 time=7.56 ms
64 bytes from 192.168.109.200: icmp_seq=12 ttl=64 time=6.59 ms
64 bytes from 192.168.109.200: icmp_seq=13 ttl=64 time=15.4 ms
64 bytes from 192.168.109.200: icmp_seq=14 ttl=64 time=11.9 ms
64 bytes from 192.168.109.200: icmp_seq=15 ttl=64 time=14.2 ms
64 bytes from 192.168.109.200: icmp_seq=16 ttl=64 time=5.25 ms
^C
--- 192.168.109.200 ping statistics ---
16 packets transmitted, 16 received, 0% packet loss, time 15014ms
rtt min/avg/max/mdev = 4.005/109.997/927.380/239.243 ms

 

[ere109]

I'm reading various threads on FreeBSD forum, here, etc. Some hints:
people have suggested:
1. rebuilding and restarting of sysutils/hal
2. adding a hint file to disable MSI reporting
3. downgrading drive speed from 600 to 300
4. installing an older, more stable AHCI driver
5. and an issue related to scrubs - which is hightly interesting: I just set up my first scrub schedule, and the first scrub would have happened Thursday night, so interesting that the machine is non-responsive Friday morning. The thread:
https://forums.freebsd.org/threads/ahci-device-timeouts-while-performing-zfs-scrub.24189/

 

[dlavigne]

That is typically a hardware issue. What is the output of zpool status?

Are there any hardware related messages in /var/log/messages?

What is the build version (from System -> Information)?

 

[ere109]

First of all, even though I could STILL access the Plex jail, I couldn't access the GUI, console or SSH, so I had to hit the power button and reboot.
Build is 11.1-U2
pool status shows no errors, but I'll include it:

Code:

% zpool status
  pool: MAINSPIN
 state: ONLINE
  scan: resilvered 212K in 0 days 00:00:00 with 0 errors on Wed Feb 28 21:29:43 2018
config:

		NAME											STATE	 READ WRITE CKSUM
		MAINSPIN										ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/ead229c8-f655-11e7-ac4f-002590ed9fc4  ONLINE	   0	 0	 0
			gptid/ebe14794-f655-11e7-ac4f-002590ed9fc4  ONLINE	   0	 0	 0
			gptid/ecf07c2d-f655-11e7-ac4f-002590ed9fc4  ONLINE	   0	 0	 0
			gptid/ee066b24-f655-11e7-ac4f-002590ed9fc4  ONLINE	   0	 0	 0
			gptid/ef14066a-f655-11e7-ac4f-002590ed9fc4  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:06 with 0 errors on Wed Mar  7 03:45:06 2018
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  ada0p2	ONLINE	   0	 0	 0

errors: No known data errors

I pared down 157kb of messages file to a handful. I included every drive related error, but there are some that I question. For instance, a "tap0" interface shows up on Mar 7, and that seems to be about the time the system hung/locked. During that time, I could still access media through the Plex jail, but my smb shares were "non existent," and all other resources were unreachable (though I could ping the device IP). I'll leave the tap0 info in, because now I'm slightly paranoid. That interface had never existed before, and doesn't exist now using ifconfig. Here's the messages log:

Code:

Mar  2 16:35:42 FreeNAS Syncing disks, vnodes remaining... 0 0 0 0 0 0 done
Mar  2 16:35:42 FreeNAS All buffers synced.
Mar  2 17:35:22 FreeNAS coretemp10: critical temperature detected, suggest system shutdown (*...this repeated for all cores during a movie on Plex jail, ended 19:59:36)
Mar  2 18:03:18 FreeNAS uwsgi: [reporting.rrd:163] Failed to generate graph: b'ERROR: bad CF: rx\n' (* this error showed up concurrently, about every 10 mins)
Mar  2 19:59:36 FreeNAS coretemp10: critical temperature detected, suggest system shutdown
Mar  4 02:13:20 FreeNAS /alert.py: [system.alert:400] Alert module '<samba4.Samba4Alert object at 0x814cca8d0>' failed: [Errno 48] Address already in use
Mar  5 17:02:59 FreeNAS nologin: Attempted login by root on UNKNOWN
Mar  5 17:02:59 FreeNAS root: /usr/local/etc/rc.d/netdata: WARNING: failed to start netdata
Mar  5 17:03:22 FreeNAS nologin: Attempted login by root on UNKNOWN
Mar  5 17:03:22 FreeNAS root: /usr/local/etc/rc.d/netdata: WARNING: failed to start netdata
Mar  7 03:45:00 FreeNAS ZFS: vdev state changed, pool_guid=4105976122447053721 vdev_guid=9675754553994621839
Mar  7 23:07:11 FreeNAS kernel: tap0: link state changed to UP
Mar  7 23:22:56 FreeNAS mountd[2304]: mount request from 192.168.109.124 for non existent path /Private
Mar  7 23:24:47 FreeNAS dhclient[42288]: Interface tap0 is down, dhclient exiting
Mar  7 23:24:47 FreeNAS dhclient[42190]: connection closed
Mar  7 23:24:47 FreeNAS dhclient[42190]: exiting.
Mar  7 23:26:27 FreeNAS root: /etc/rc.d/dhclient: WARNING: failed to start dhclient
Mar  8 01:14:43 FreeNAS kernel: pid 54383 (Plex Media Scanner), uid 972: exited on signal 11
Mar 12 13:46:53 FreeNAS Copyright (c) 1992-2017 The FreeBSD Project.
Mar 12 13:47:02 FreeNAS smbd: dnssd_clientstub ConnectToServer: connect() failed path:/var/run/mdnsd Socket:36 Err:-1 Errno:2 No such file or directory
Mar 12 13:47:17 FreeNAS daemon[3848]:	 2018/03/12 13:47:17 [ERR] agent: failed to sync remote state: No cluster leader

 

[dlavigne]

Were you able to figure this out?

AHCI errors and signal 11 errors suggest a hardware issue.

 

[ere109]

I'm sending the MB in for testing. I believe the secondary ATA controller is bad. But the drive that faulted is connected to the primary ATA controller. I'll blame U2 (or at least hope that the problem goes away now that I've upgraded to U3).

 

[Bostjan]

Thank you for the information.


I have the same problem. Can’t ping the IP of the FreeNAS, can’t SSH, SMB doesn’t work, GUI doesn’t work. I have several jails on this FreeNAS and they work and are accessible through SSH and web (http).


I think that build version of FreeNAS is 11.1-U2.


@dlavigne How can I access messages in /var/log/messages?

How can I se the output of zpool status?

 

[ere109]

navigate to /var/log, then type:

Code:

more messages

That will let you read your message log. However, pay close attention to the time you think a problem occurred, because those logs get VERY big. Fortunately, they're timed.

 

[Bostjan]

I can't connect to the FreeNAS so I attached the keyboard and access it this way.

I navigated to /var/log, and typed: more messages, but the log is useless because every 2 minutes there was a message

"[WARN] agent: Check 'service:nas-health' is now warning"


there are basically just these lines – A LOT of them.

Is there a way to filter this messages and show other?


-----------

Second problem.


After a while my keyboard stopped responding. I disconnect and reconnect it and it didn’t work. I try different USB port, it didn’t work.

I try second keyboard and it still didn’t work. If I typed something, nothing displayed on the screen.

I tried third keyboard and it didn’t work. If I typed something, nothing displayed on the screen. While I was testing and trying the third keyboard if anything will display on the screen I also pressed Ctrl+Alt+Del and the FreeNAS rebooted.

After the reboot, the problem persists. I can’t connect via SSH, web GUI, samba BUT I can connect to jails on this FreeNAS. They all work.

 

[TB4217]

Having the same problem.

Plex can access all its media, but I cannot get to any other files via my Mac or the Windows VM I have on it as well.

I believe it started when I activated an upgrade that was showing in the status area of the GUI console.

My router knows Freenas is connected to a particular IP address, and I can ping that address, but I just get the following error when I try to connect to it -
------------
An error occurred.
Sorry, the page you are looking for is currently unavailable.
Please try again later.

If you are the system administrator of this resource then you should check the error log for details.

Faithfully yours, nginx.
------------

I don't have sufficient skills to be able to do much else, so would appreciate any assistance that might be available.

 

[Bostjan]

Here’s what I did. I think that it resolved this problem. Basically you refresh the network interface.


Connect keyboard and monitor to FreeNAS box. You should see “Console setup” and 12 options. Something like this. https://doc.freenas.org/11/_images/console-menu.png


Select

1) Configure Network Interfaces

and follow the wizard. You’ll be probably presented with the current settings; you just confirm them all.

 

[TB4217]

Thanks Bostjan

Tried your suggestion, but no difference, so I thought I would try to boot into the initial configuration, and this time it did work, however the console alert Led was flashing red, and when I open it the error says -

• CRITICAL: 27 Mar 2018, 7:14 a.m. - The boot volume state is DEGRADED: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

Any idea how to fix this?

 

[Bostjan]

It says

The boot volume

so probably replace the boot media. Do you have FreeNAS installed on mirrored USB drives?

Did you save configs recetly, did you?

 

[Chris Moore]

Thanks Bostjan

Tried your suggestion, but no difference, so I thought I would try to boot into the initial configuration, and this time it did work, however the console alert Led was flashing red, and when I open it the error says -

• CRITICAL: 27 Mar 2018, 7:14 a.m. - The boot volume state is DEGRADED: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

Any idea how to fix this?

I is going to need a new boot device.
Did you backup the configuration db?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[TB4217]

Thanks Bostjan and Chris Moore

I have downloaded the current config and I can see it is a revision earlier than the previous one I had saved.

So do I need to create a new USB boot drive and load the latest saved config into it, or should I try the previous version?