jpi
Dabbler
- Joined
- Apr 21, 2019
- Messages
- 14
Hello,
My FreeNAS install has been running great up until recently. The most recent change I made was by adding the 6x 3TiB in bays 00-05. These are Western Digital Red 5400 RPM drives configured in a raidz-1 (slow non-critical storage). Let's call this volume "media".
The 600GiB drives in bays 06-11 are three mirrors striped (I think this is equivalent to a RAID10). These drives are Hitachi 15k SAS drives. This volume is fast(ish) and considered critical as it has a zvol on it that is presented over iSCSI to a VMware cluster. The disks in bays 12 and 13 operate standalone and have individual zvols created on each that are also presented over iSCSI to VMware (non-critical). iSCSI is on its own dedicated 2x 1Gbps NICs with isolated VLANs to the ESXi hosts. All of these have been working flawlessly up until recently. The internal USB to SATA is an SSD that has the FreeNAS os installed on it.
On the previously mentioned media volume, I have created a zvol with two SMB shares that are accessed by a Windows Plex server and also an Ubuntu Docker host that mounts the SMB shares locally and passes them into containers like Sonarr, Radarr, and SABnzbd.
So that explains a bit about my setup and the change I made recently. So beginning about 2 days after adding the 6x disks that make up my media volume FreeNAS has been intermittently going unresponsive.
1st incident:
Web UI: timed out
SSH: timed out
SMB shares: unaccessible
iSCSI LUNs: unaccessible (all VMware VMs down)
Resolution: selected reboot option on r510 console. the system successfully rebooted.
2nd incident:
Web UI: allowed login but the page would only load about 50% and nothing was clickable
SSH: authenticated but never gave a prompt
SMB shares: unaccessible
iSCSI LUNs: unaccessible (all VMware VMs down)
Resolution: selected shell option on r510 console. shell would not accept input. ended up hard resetting r510.
3rd incident:
Web UI: timed out
SSH: timed out
SMB shares: unaccessible
iSCSI LUNs: unaccessible (all VMware VMs down)
Resolution: selected reboot option on r510 console. the system failed to reboot. the console showed multiple
I have noticed lots of
Chassis: Dell R510
Controller: Dell PERC H200 flashed to IT mode
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
Memory: 16GB
Build: FreeNAS-11.2-U2
My FreeNAS install has been running great up until recently. The most recent change I made was by adding the 6x 3TiB in bays 00-05. These are Western Digital Red 5400 RPM drives configured in a raidz-1 (slow non-critical storage). Let's call this volume "media".
The 600GiB drives in bays 06-11 are three mirrors striped (I think this is equivalent to a RAID10). These drives are Hitachi 15k SAS drives. This volume is fast(ish) and considered critical as it has a zvol on it that is presented over iSCSI to a VMware cluster. The disks in bays 12 and 13 operate standalone and have individual zvols created on each that are also presented over iSCSI to VMware (non-critical). iSCSI is on its own dedicated 2x 1Gbps NICs with isolated VLANs to the ESXi hosts. All of these have been working flawlessly up until recently. The internal USB to SATA is an SSD that has the FreeNAS os installed on it.
On the previously mentioned media volume, I have created a zvol with two SMB shares that are accessed by a Windows Plex server and also an Ubuntu Docker host that mounts the SMB shares locally and passes them into containers like Sonarr, Radarr, and SABnzbd.
So that explains a bit about my setup and the change I made recently. So beginning about 2 days after adding the 6x disks that make up my media volume FreeNAS has been intermittently going unresponsive.
1st incident:
Web UI: timed out
SSH: timed out
SMB shares: unaccessible
iSCSI LUNs: unaccessible (all VMware VMs down)
Resolution: selected reboot option on r510 console. the system successfully rebooted.
2nd incident:
Web UI: allowed login but the page would only load about 50% and nothing was clickable
SSH: authenticated but never gave a prompt
SMB shares: unaccessible
iSCSI LUNs: unaccessible (all VMware VMs down)
Resolution: selected shell option on r510 console. shell would not accept input. ended up hard resetting r510.
3rd incident:
Web UI: timed out
SSH: timed out
SMB shares: unaccessible
iSCSI LUNs: unaccessible (all VMware VMs down)
Resolution: selected reboot option on r510 console. the system failed to reboot. the console showed multiple
sonewconn: pcb 0xfffff800afcdacb0: Listen queue overflow: 193 already in queue awaiting acceptance (80 occurrences)
. ended up hard resetting r510.I have noticed lots of
CIFS VFS: no writable handles for inode
and task unrar blocked for more than 120 seconds
on my Ubuntu Docker host. I started to think that maybe I was causing too much IO on the SMB share so I stopped the SABnzbd container that downloads and unrars stuff on the share. However, even after stopping that high IO container after the 2nd incident FreeNAS still hung a few days later. Not sure where to go from here other than disabling everything that accesses the shares (Plex, Sonarr container and Radarr container) to try and rule out some weird IO issues causing FreeNAS to freak out. Not sure where to go from here, thanks for the help and reading a rather long post.Code:
R510 Bay\Drive Layout Bay 00 3TiB | Bay 03 3TiB | Bay 06 600GiB | Bay 09 600GiB Bay 01 3TiB | Bay 04 3TiB | Bay 07 600GiB | Bay 10 600GiB Bay 02 3TiB | Bay 05 3TiB | Bay 08 600GiB | Bay 11 600GiB Internal 2.5" Bay 12 1.2TiB Internal 2.5" Bay 13 800GiB Internal USB to SATA: 250GB SSD
Chassis: Dell R510
Controller: Dell PERC H200 flashed to IT mode
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
Memory: 16GB
Build: FreeNAS-11.2-U2