Need help to save my data....

[Dogger35]

I am running the following:
FreeNAS 9.2.x
Intel CoreDuo
8GB RAM
3- 1TB WD Black HD's in RAIDZ (RAID5)


Ok so I am watching a movie and then all of a sudden it stops and acts as if I lost connection. I try from my Phone to hit the GUI and it will not come up. So first confirm that I have connection else where in the house - Yes. Hmmm so I try a PC that is in the same room and can't bring the GUI up. I connect a keyboard and LCD and it looks like it is running but not able to do any thing. I hard shut it down and reboot, system, comes back up and I can get into the GUI now put I can't hit the PlexMedia page. I check the plug-in and it is not running I try and start it and the system freezes again. So
I hard reboot again and this time I start to see errors

1. boot livecd
2. when you get the menu that is only displayed for 2 seconds, press space bar to pause the countdown
3. press ESC (escape) to 'Escape to loader prompt'
4. You get an OK prompt, type the following and press enter:
set hint.ahci.0.msi=0
5. Now continue booting with: boot

Starting file system checks:
/dev/ufs/FreeNASs1a: File System Clean; skipping Checks
/dev/ufs/FreeNASs1a: clean, 456778 free

/dev/ufs/FreeNASs4:File System Clean; skipping Checks
/dev/ufs/FreeNASs4:clean, 456778 free

/dev/ufs/FreeNASs3:File System Clean; skipping Checks
/dev/ufs/FreeNASs3:clean, 456778 free

Mounting local file system
ahcich1: Timeout on slot 11 port 0

CAM Status: Command TimeOut
Error 5 - Retry was blocked
Timeout on slot 28 port 0
(ADA0:ahcich1:0:0:0) : CAM Status: Command timeout
(ADA0:ahcich1:0:0:0) : Retrying Command
ahcich1: Timeout on slot 24 port 0
(aprobe1:ahacich1:0:15:0): SOFT_RESET. ACB 00 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe1:ahacich1:0:15:0): CAM Status: Unconditionally Re-queue Request
(aprobe1:ahacich1:0:15:0): Error 5, Retry was blocked
(aprobe1:ahacich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahacich1:0:0:0): CAM Status Timeout

Device /dev/ada0. Failed SMART SELF CHECK - BACK UP DATA NOW

This repeats but it eventually boots up and I can access the GUI

So I start to do some looking around and I I find this and it allows me to boot in

Once in I can see the ZFS Volume and they show online and healthy but I am sure 1 drive has errors. I am set up in RAIDZ with the 3-1TB drives.

I check the PLEX plugin and still stopped, I check the Jail and its not showing, Checking the Volume it is there. So I try from the PC to access the share and I can see the Videos folder and I try and access it and I can see the folders though it take some time. So I try to start to back up the folders on my local PC and I get access errors and FreeNAS locks up again.

[root@freenas ~]# cat /etc/fstab
/dev/ufs/FreeNASs1a / ufs ro 1 1
/dev/ufs/FreeNASs3 /cfg ufs rw,noauto 2 2
/dev/ufs/FreeNASs4 /data ufs rw,noatime 2 2
/dev/ada0p1.eli none swap sw 0 0
/dev/ada1p1.eli none swap sw 0 0

So my question is this as I am no Unix expert, From the CommLine I run FSCK and this is what I get
[root@freenas ~]# fsck -y
** /dev/ufs/FreeNASs1a
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
39452 files, 1441028 used, 456778 free (2906 frags, 56734 blocks, 0.2% fragmentation)

***** FILE SYSTEM IS CLEAN *****
** /dev/ufs/FreeNASs3
** Last Mounted on /fusion/jkh/921/freenas/os-base/amd64/_.mnt
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
2 files, 2 used, 2829 free (21 frags, 351 blocks, 0.7% fragmentation)

***** FILE SYSTEM IS CLEAN *****
** /dev/ufs/FreeNASs4 (NO WRITE)
** Last Mounted on /data
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
13 files, 3818 used, 36709 free (37 frags, 4584 blocks, 0.1% fragmentation)

Why is this shown as ** /dev/ufs/FreeNASs4 (NO WRITE)

I am pretty sure I know what drive is bad how can I pull and swap it for a new 1TB drive if I am having issue in the GUI? I have a good backup (freenas-FreeNAS-9.2.1.7-RELEASE-x64 (fdbe9a0)-20140926092554) but no snap shot. Which I should have done. This setup is only 1 wks old.


Any help would be great......

thanks in advanced

 

[cyberjock]

Snapshots aren't required for backups. Many people that do backups will argue that ZFS replication is the best way to do and handle backups, and that requires snapshots, but snapshots aren't required.

Normally you should follow the manual for disk replacement. If you can't follow the initial steps for offlining the bad drive the alternative is to simply remove the bad drive with the system off.

But, the fact that Plex won't start means you probably have more than just a single drive failing. It sounds like you may have more than 1 disk failing. I will tell you that you've broken some faux pas in FreeNAS because your first indicator of a disk failing SMART should have come in the form of emails and not messages on bootup. So you clearly don't have emailing and/or SMART monitoring and/or SMART testing properly setup. So you should make that a priority.

I'm glad you have backups though, because I think it's more than 50% chance you're going to need to use them.

 

[Dogger35]

OK where are the logs located? I would like to tail the active log. I am backing up the moves right now off the FreeNAS to a local box and looks like I have about 4 hrs to wait.

What is the faux pas that you are referring to?

I have email setup an never received any email with the smart checking...I got this though
Subject:
Cron <root@freenas> /etc/ix.rc.d/ix-kinit renew
Body:
/etc/rc.conf: plexmediaserver_1: not found

Subject:
freenas.local security run output
Body:
freenas.local changes in mounted filesystems:
13a14
MediaShare/MediaData/Music /mnt/MediaShare/MediaData/Music zfs rw,nfsv4acls 0 0
16d16
< MediaShare/PluginsJail/minidlna_1 /mnt/MediaShare/PluginsJail/minidlna_1 zfs rw,nfsv4acls 0 0
19d18
< devfs /mnt/MediaShare/PluginsJail/minidlna_1/dev devfs rw,multilabel 0 0
21d19
< procfs /mnt/MediaShare/PluginsJail/minidlna_1/proc procfs rw 0 0

freenas.local kernel log messages:

uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uhub2: 6 ports with 6 removable, self powered
uhub6: 6 ports with 6 removable, self powered
Timecounter "TSC-low" frequency 1163773663 Hz quality 1000
vboxdrv: fAsync=0 offMin=0x2ae offMax=0xb2f
em0: promiscuous mode enabled
epair0a: Ethernet address: 02:a5:87:00:0c:0a
epair0b: Ethernet address: 02:a5:87:00:0d:0b
epair1a: Ethernet address: 02:53:d6:00:0d:0a
epair1b: Ethernet address: 02:53:d6:00:0e:0b
epair0a: link state changed to DOWN
epair0b: link state changed to DOWN
ifa_del_loopback_route: deletion failed
Freed UMA keg (udp_inpcb) was not empty (20 items). Lost 2 pages of memory.
Freed UMA keg (udpcb) was not empty (336 items). Lost 2 pages of memory.
Freed UMA keg (tcptw) was not empty (100 items). Lost 2 pages of memory.
Freed UMA keg (tcp_inpcb) was not empty (30 items). Lost 3 pages of memory.
Freed UMA keg (tcpcb) was not empty (16 items). Lost 4 pages of memory.
hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required

-- End of security output --

Body:
freenas.local kernel log messages:

Limiting closed port RST response from 381 to 200 packets/sec
Limiting closed port RST response from 395 to 200 packets/sec
Limiting closed port RST response from 392 to 200 packets/sec
Limiting closed port RST response from 397 to 200 packets/sec
Limiting closed port RST response from 377 to 200 packets/sec
Limiting closed port RST response from 411 to 200 packets/sec
Limiting closed port RST response from 412 to 200 packets/sec
Limiting closed port RST response from 417 to 200 packets/sec
Limiting closed port RST response from 406 to 200 packets/sec
Limiting closed port RST response from 345 to 200 packets/sec
Limiting closed port RST response from 389 to 200 packets/sec
Limiting closed port RST response from 360 to 200 packets/sec
Limiting closed port RST response from 372 to 200 packets/sec
​