Whoa, this is one long thread for a very basic problem. I'm glad to see a lot of people offering assistance.
@arameen Please take the advice of the group who have posted that you should be doing some more reading up on how FreeNAS works and how to maintain it. Unfortunately there is not a nice single guide on each and every step you must do to configure your FreeNAS system properly to make it a virtually hands-free system. FreeNAS was never originally intended to be used buy the typical novice and required the user to do a lot of work investigating how to make the system work for you. Thankfully our forum members are here to help us all out when we are in need. Also I know that some of the postings here read as if the poster is geting a little frustrated, both them and you, I can relate to be honest, I think we have all been on both sides of a problem. I am glad to see it looks like you are getting your problematic pool recovered.
If you do not know how to setup routine SMART tests then read the User Guide, it has a section on it. Once you have your pool issues solved then I'd establish those settings as follows: Smart Short Test = Once a Day on Sunday through Friday, Smart Long Test = Once on Saturday. Do not run a long test on Sunday because the automatic scrub runs on a sunday and then the hard drive is fighting for time between running both tests. And as previously stated, ensure your email notification is setup properly becasue you will get an email when a drive experiences a problem that you should take care of.
Other things to look at while you are troubleshooting, ensure all the fans in your computer are spinning, including your power supply.
You have several drives which have UDMA errors (ID 199). These are caused by communications errors, typically a bad cable or controller. Unfortunately these values never reset back to zero so my advice is to just print out the SMART data and keep it on hand. If you notice the UDMA CRC errors increase then you have a problem with a cable or controller. While the problem "could" be in the hard drive itself, it's very unlikely.
So lets see if I understand your present hardware configuration...
1) The "Main" pool has been disconnected from the motherboard SATA connectors.
Q1) Are the drives still plugged in to power or are they disconnected?
2) The pool having issues is not disconnected from the HBA card and plugged directly into the motherboard SATA connectors.
Q2) Is the HBA still plugged into the motherboard?
If you are not using the HBA and your pool is fine once you have completed all the testing, then you can validate if the HBA is causing it by powering down your system and using the HBA SATA ports and powering back up, test out your system, run a scrub, after that check the SMART data for any changes. Obvioulsy you can't use just a single data point, you need to give the system some time to fail or pass but this is a good quick check. If you have new failures then it's highly suspect that the HBA is your problem. You can rule out the power supply too by connecting the other main pool hard drives up to the power, assuming they were disconnected. If they have been connected this entire time then I doubt it's the power supply at fault.
My last piece of advice is to take your steps slow, do not rush into anything. While you can check SMART data at anytime, you shouldn't be resilvering one drive while running a smart long test on another drive at the same time, it will slow down the resilvering and smart test and the drive under the smart test will hate you. I'm not telling you to stop any test if you have it running, but it will take a lot longer during a resilver or scrub operation.
Stick with it and best of luck to you,
-Joe
