SOLVED FreeNAS boot process loops on MEDIUM ERROR and fail to start

[ralphy]

I am running Freenas 8.3 under ESXI 5.0

Recently, one of my drives started showing some intermittent scrub errors, and this morning I was unable to use the web interface to view the details of the storage. SSH to FreeNAS and cd /mnt and then ‘ls’ hung the shell. Then shutdown FreeNAS and the host to run some disk diagnostics(still in progress)

FreeNAS now fails to boot. After the boot message ‘Mounting Local File System’, the following message repeats and does not time out / abort.

Code:

(da8:mpt2:0:1:0): SCSI status error
(da8:mpt2:0:1:0): READ(10). CDB:28 0 0 40 1 80 0 1 0 0
(da8:mpt2:0:1:0): CAM status: SCSI Status Error
(da8:mpt2:0:1:0): SCSI status: Check Condtion
(da8:mpt2:0:1:0): SCSI sense: MEDIUM ERROR asc:11,4 (Unrecovered read error – auto reallocated failed)
(da8:mpt2:0:1:0): Retrying command (per sense data)
….. (repeats)
(da8:mpt2:0:1:0): Error 5 Retries exhausted
….whole cycle repeats

I have tried removing the drive physically (FreeNAS runs as a VM), but when I did this, FreeNAS wouldn’t even boot.

I can boot to single user mode, but am at a loss to know what I need to do to fix the problem.
First step would be to get FreeNAS running, then replace the drive, and finally attempt to recover what’s on the drive (it’s a back drive, so I’m not too concerned just yet)

Could I just remove the relevant disk from the VM inventory? Will that cause problems with FreeNAS not having the disk removed via the OS? I recall that trying to just replace a drive in a RAID set without properly removing it first will cause issues, but this drive is not part of a RAID.

What I could do in single user mode instead, to remove da8 safely from the devices attempted to be mounted so that the boot will finish?


Configuration:
FreeNAS 8.3 x64 under ESXI 5.0
4 virtual disks, 10GB (FreeNAS OS), 50GB, 100GB, 2 TB
3 x WD20EURS in RAID, RDM (not for performance reasons)
1 x WD15EVDS, RTDM
1 x ST350 GB RDM
ASROCK Z77 Extreme 4
2x8GB RAM
i7-3770

 

[dlavigne]

Were you able to solve this?

 

[ralphy]

Yes, thank you, and I really should have posted my solution as it's over a month now and the grey matter has failed me a little.

If I recall correctly, I ended up working on a clone copy of the VM and testing if I could boot FreeNAS by removing the defective drive from the VM inventory. Once this was confirmed as working, I then replaced the defective drive in the VM host with a new one and recreating the Raw Device maps and added it to the FreeNAS VM inventory. I think I then needed to make to changes to the FreeNAS volumes, to get the new drive 'seen' in FreeNAS. Fortunately, the defective volume was only one drive.

I wasn't concerned about losing the old drive, as it was a back drive, asI still hadthe original files.