IO Failures reported. Now Freenas wont boot from USB flash drive

[robdeep]

Hi All,

I recently put together a FreeNas system with some old, non server parts, I know that is not ideal, but I figured it might be worth a shot versus building a new expensive system. The motherboad is a Nvidia based chipset, and has an Intel QuadCore Q6600 cpu, paired with 8gb of DDR2 ram (Non-ecc). 4, 3TB WD red drives in a Raid Z1 configuration.

I installed Freenas 9.1.1 onto an 8gb usb drive, and it was working fine for a few weeks. Then the pool went into an "Unavailable" state. The last email report I got is below... I see the errors and the disks that went offline. When this happened it seems to was logging a ton of messages to the usb drive, filling it up? Upon attempting to restart the system the usb drive is no longer detected as a boot drive.

I've ran SMART tests on all the disks, and the all passed. I'm in the process of running Memtest.

Even if I had a disk problem, why would the USB drive become unusable?

Removing stale files from /var/preserve:
Cleaning out old system announcements:
Backup passwd and group files:
Verifying group file syntax:
/etc/group is fine
Backing up mail aliases:
Backing up package db directory:

Disk status:
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/FreeNASs1a 926M 654M 198M 77% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/md0 4.6M 3.8M 403k 91% /etc
/dev/md1 823k 2.0k 756k 0% /mnt
/dev/md2 149M 149M -11M 109% /var
/dev/ufs/FreeNASs4 19M 1.9M 16M 10% /data
CTS 7.8T 4.0T 3.8T 51% /mnt/CTS
/dev/md3 1.9G 800k 1.7G 0% /var/tmp/.cache
CTS/Storage 3.7T 1.7G 3.7T 0% /mnt/CTS/Storage

Checking status of zfs pools:
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
CTS 10.9T 5.51T 5.37T 50% 1.00x UNAVAIL /mnt

pool: CTS
state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see: http://illumos.org/msg/ZFS-8000-JQ
scan: scrub repaired 0 in 8h36m with 0 errors on Sun Dec 15 08:37:00 2013
config:

NAME STATE READ WRITE CKSUM
CTS UNAVAIL 0 0 0
raidz1-0 UNAVAIL 1 0 0
gptid/335bca3f-47b9-11e3-be8f-00044b04fcd8 ONLINE 0 0 0
10502012699523115976 REMOVED 0 0 0 was /dev/gptid/33ec347d-47b9-11e3-be8f-00044b04fcd8
13618908694853871279 REMOVED 0 0 0 was /dev/gptid/347e455d-47b9-11e3-be8f-00044b04fcd8
gptid/350e7e28-47b9-11e3-be8f-00044b04fcd8 ONLINE 0 0 0

errors: 32063789 data errors, use '-v' for a list

Checking status of ATA raid partitions:
Checking status of gmirror(8) devices:
Checking status of graid3(8) devices:
Checking status of gstripe(8) devices:
Network interface status:
Security check:
(output mailed separately)

Checking status of 3ware RAID controllers:
Alarms (most recent first):
No new alarms.


-- End of daily output --

-- Security Run Output --

Checking for uids of 0:
root 0

Checking for passwordless accounts:

Checking login.conf permissions:

Checking for ports with mismatched checksums:

nas1.CTS.LAN kernel log messages:
+++ /tmp/security.t6W4v9WE 2013-12-26 03:01:00.000000000 -0600
+pid 1665 (syslogd), uid 0 inumber 6205 on /var: filesystem full pid
+1665 (syslogd), uid 0 inumber 6186 on /var: filesystem full

nas1.CTS.LAN login failures:

nas1.CTS.LAN refused connections:


-- End of security output --

 

[robdeep]

I also got this via e-mail..

Cron <root@nas1> newsyslog

bzip2: I/O or other error, bailing out. Possible reason follows.
bzip2: No space left on device
Input file = /var/log/messages.0, output file = /var/log/messages.0.bz2
bzip2: Deleting output file /var/log/messages.0.bz2, if it exists.

newsyslog: `bzip2 -f /var/log/messages.0' terminated with a non-zero status (1)