fan failed

[tlachau]

I'm new to FreeNAS, and am trying to see if I can get some help getting more detail on this alert message, and gui alert

Code:

145709 (boot + 26s/ENCL/CRIT) - Enclosure PD 0a(c Port 0 - 3/p1) fan 1 failed
145713 (boot + 26s/ENCL/CRIT) - Enclosure PD 0a(c Port 0 - 3/p1) fan 3 failed
145711 (boot + 26s/ENCL/CRIT) - Enclosure PD 0a(c Port 0 - 3/p1) fan 2 failed

The red alert also shows this from the web page.

Code:

The boot volume state is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

Wondering what I might need to do here to troubleshoot further.

Here is the result of zpool status -v

Code:

errors: No known data errors													
																			 
  pool: freenas-boot															
 state: ONLINE																
status: One or more devices has experienced an error resulting in data		
	   corruption.  Applications may be affected.							
action: Restore the file in question if possible.  Otherwise restore the		
	   entire pool from backup.												
   see: http://illumos.org/msg/ZFS-8000-8A									
  scan: scrub repaired 0 in 0h5m with 1 errors on Thu Jan 25 03:50:43 2018	
config:																		
																			 
	   NAME		STATE	 READ WRITE CKSUM								
	   freenas-boot  ONLINE	   0	 0	 0								
		 da0p2	 ONLINE	   0	 0	 0								
																			 
errors: Permanent errors have been detected in the following files:			
																			 
	   freenas-boot/ROOT/default@2015-09-16-10:47:51:/usr/local/lib/libicudata.
so.53.1

 

[tvsjr]

Those appear to be two separate errors, although one may be linked to the other. The first error indicates the failure of three fans in the enclosure (assuming BSD is interpreting everything properly). The second indicates a critical error on your pool, leading to damaged data. I hope you have a backup. I say the two may be linked as, if 3 fans failed at once, that may have led to a very high temperature condition that has damaged drives.

Post the output of zpool list and zpool status to start. A smartctl -a /dev/daXX replacing daXX with the path to each of your data drives would also be helpful.

Also, per the rules of the forum, please post your *complete* system configuration.

This is a critical error that you need to address immediately.

 

[tlachau]

freenas-boot is a usb stick. It appears maybe my boot device is needing something a little more stable. I have a backup of the configuration.

Code:

zpool list
datastore	 7.12T  1.16T  5.96T		 -	22%	16%  1.00x  ONLINE  /mnt
freenas-boot  15.1G   527M  14.6G		 -	  -	 3%  1.00x  ONLINE  -

zpool status
  pool: datastore
 state: ONLINE
  scan: scrub repaired 0 in 6h20m with 0 errors on Sun Jan 14 06:20:14 2018
config:

		NAME										  STATE	 READ WRITE CKSUM
		datastore									 ONLINE	   0	 0	 0
		  gptid/1c18a7b9-5d86-11e5-97f7-002590915176  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
status: One or more devices has experienced an error resulting in data
		corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
		entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 0h5m with 1 errors on Thu Jan 25 03:50:43 2018
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da0p2	 ONLINE	   0	 0	 0

errors: 1 data errors, use '-v' for a list

As for my disks. In FreeNAS it only shows up as a single disk - mfid1 7.9TB. This system was inherited so I am not sure how it was set up. A quick search suggests that I should start over and rebuild this thing since it looks like it was setup with a RAID controller.

How can I find out which fans have failed other than the labeling from the error?

 

[tvsjr]

You still need to provide the complete system specs. Also, in the future, please put the output of commands in "code" tags or "cmd" tags, like this:

Code:

Monospaced fonts make things much easier to read!

Your initial estimation is correct... you've got several issues going on, and fixing them will necessitate a fair amount of work. Depending on your system config, it may be worth changing what you have now... or simply tossing it and building something brand new. My first suggestion would be to take a complete backup of the pool.

First, FreeNAS showing your storage as one disk is a Bad Thing. You either have a hardware RAID controller in the middle, or you're really running only one disk with no redundancy. FN relies on having direct access to the disks for several reasons, the most important of which is alerting to possible drive issues.

As for your boot device, USB sticks are slow and unreliable. They were the recommended (least bad) option when SSDs were insane money, but times have changed. You don't need anything super-special to be a boot device... if you'll look at my sig, I'm running a pretty substantial FreeNAS system on a pair of Intel 320 40GB drives. These drives were released in 2011, and I bought them on eBay for $25/ea. with about 90% of their warranteed life remaining. I would suggest backing up your config, replacing your USB sticks with a single SSD (or two, if you're paranoid), reinstalling FreeNAS, then reloading your configuration. You also need to know what version of FN you're running... if you're running an older version, you may not want to jump straight to version 11. Full system specs, like I have in my sig, would really help with this.

As for the fans, I would use the simple method of crack the box open and figure out which fan isn't spinning. This could be a phantom error - maybe the motherboard expects a fan in a certain connector and it's never been populated - or it could be a major problem. It's somewhat unlikely that you would drop 3 fans simultaneously, but it does happen. It could also be a motherboard or power supply issue. Some BIOS screens will show fan speeds... it would be worth checking.

 

[Chris Moore]

It is difficult to do more than guess about your fan situation. You really need to tell us about the hardware first so we have some idea what we are thinking about.

It is actually right there in the rules:
https://forums.freenas.org/index.php?threads/updated-forum-rules-4-11-17.45124/

 

[tlachau]

It's an ixsystems device 12 bay

Build FreeNAS-9.3-STABLE-201509022158
Platform Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
Memory 24541MB
System Time Fri Feb 16 12:36:33 PST 2018
Uptime 12:36PM up 3:28, 1 user
Load Average 0.61, 0.44, 0.36

 

[tlachau]

I took the approach suggested and opened the case. All 3 fans are indeed running, so I'm not sure what the alert was about.

 

[tvsjr]

They may be running but not at an optimal speed. There should be more than 3 fans, but I'm not familiar with exactly what piece of hardware iX used for that box. Make sure you check the CPU fans as well.

The fact that it's an iX box makes it weird that you're only seeing one drive. How about the output of camcontrol devlist?

 

[Chris Moore]

Intel(R) Xeon(R) CPU E5645 @ 2.40GHz

This is a processor from a very long time ago, launched in the first quarter of 2010. I would expect the system to be in the vicinity of 6 (or more) years old and it is likely that it was reconfigured at some time between then and now. I don't think that iXsystems would have configured it with hardware RAID.

I understand that you just came into the management of this system? I think that the boot drive is probably faulted and needs to be replaced.
Beyond that, what is the system used for? It may be time to look into a replacement. I am going through that where I work. I have a system that is pushing 6 years old and I am soliciting quotes for replacement hardware now. If you can tell us about the purpose of this storage, we may be able to make some suggestions for replacement hardware.

 

[Chris Moore]

It appears maybe my boot device is needing something a little more stable.

The popular boot device for many builds right now is to use a small SSD as they are more reliable.

 

[tlachau]

Thank you all for your help so far. My disks are controlled by a megaraid controller it seems. From what you've said before I want to have freenas have direct access to the disks. I can't seem to get my controller to give me direct access.

Any thoughts?

 

[Chris Moore]

My disks are controlled by a megaraid controller it seems.

This system must have been modified by a previous caretaker and it is likely that any change to the current configuration will be destructive to the data.
When you shared the zpool list

Code:

zpool list																									
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT												  
datastore	 7.12T  1.16T  5.96T		 -	22%	16%  1.00x  ONLINE  /mnt
freenas-boot  15.1G   527M  14.6G		 -	  -	 3%  1.00x  ONLINE  -	

it shows that you have about 7TB of capacity, but you are only using a little more than 1TB. Given that the system is quite old to begin with, I would guess that it is using low capacity drives and the whole thing is probably a power hog that generates a lot of heat and noise for the little bit of functionality it provides. If there is a business need for this shared storage, I would start the wheels turning to get a new server to replace this one. That way you not only have something newer that you can have more confidence in, and it will use less power. You could get something like the FreeNAS Mini:
https://www.ixsystems.com/freenas-mini/
Put four 4TB drives in it and have something that will last another 3 to 5 years with minimal intervention.

 

[Chris Moore]

PS. If you want to try to modify this system, it could probably be done, but we need to know more about what hardware is actually in the box and you will need to get that data saved somewhere else because any change to the hardware (specifically that hardware RAID controller) will likely make the data inaccessible.