volume degraded

[quik]

I am quite new to freenas. I have the server running for about 1 week now with a few restarts in between.
A few hours ago i got this message:
The volume media state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.

I have 2 pool , 1 raidz 1 pool with 4 HDD and 1 mirror with 3 HDD. 2 of the mirror pool are running thru a PCIe sata card. all othere SATAs are onboard.
The message happened for 1 HDD from each pool (means it should be the PCIe card) at the same time.
after a reboot freenas sees all HDD for a moment, but after a few minutes 1 drive from each pool disappears again.
The temps for the raidz1 are under 40°, the mirror are a bit higher but under 50° (havent come around yet to install a case fan)
I am at a loss why the HDDs suddenly seem to disappear... maybe a problem with the motherboard?

 

[Redcoat]

Likely this is just the typical experience people have with using PCIe sata cards with FreeNAS. One post by @joeschmuck does, I recall, describe a unit that is successful.
Standard advice here is to get an HBA if you are out of onboard SATA ports.

 

[quik]

I am using this card, seems like a HBA?
all the HDDs connected to the PCIe card work fine, both HDDs that seem to fail are connected directly to the motherboard
I just switched one of the failing HDDs SATA cable, still the same issue. The same drive just audible powered down /disconnected again (actually both, one from each pool)

 

[Chris Moore]

The temps for the raidz1 are under 40°, the mirror are a bit higher but under 50° (havent come around yet to install a case fan)

That is way too hot. I have seen disk controllers stop working for being too hot. Get fans going.

 

[gpsguy]

Please provide detailed hardware information.

If this is a desktop motherboard, one might have 2 different SATA disk controllers. As @Redcoat said, you'd be better off with a supported HBA.

both HDDs that seem to fail are connected directly to the motherboard

 

[Redcoat]

Ah - I focused on this statement "The message happened for 1 HDD from each pool (means it should be the PCIe card) at the same time. " (my emphasis).

I agree with Chris about temperature although it seems that acceptable HDD temps are creeping upwards in manufacturer specs. But your situation may well indicate that mb and/or add-on cards are too hot if you have no case fan(s).

 

[Chris Moore]

I am using this card, seems like a HBA?
all the HDDs connected to the PCIe card work fine, both HDDs that seem to fail are connected directly to the motherboard
I just switched one of the failing HDDs SATA cable, still the same issue. The same drive just audible powered down /disconnected again (actually both, one from each pool)

No, that is not an HBA. It will need to be replaced with something that is compatible.

 

[quik]

I know the 3 mirrored drives are too hot, but one of the "hot" and one of the "cool" drives get ejected. So it shouldnt be a temp problem for now.

I use a dekstop PC (old one, Motherboard is a: gigabyte GA-H77-DS3H i5-3570K with 4x4GB DDR3 RAM, 600W PSU)

I do have a case fan, but its in front of the 4 raidz1 drives, which are cooler. I need to install another one for the 3 mirrored drives tomorrow.

No, that is not an HBA. It will need to be replaced with something that is compatible.

I dont think that is the issue since all the drives connected to it work fine!?

I think i found the issue. The motherboard has 2 6gb and 3 3gb Sata connectors. I think the drives connected to the 2 6gb connectors always seem to disconnect. Just trying to confirm it now.
YEP this seems to be the issue. I connected 2/3 mirrored drives to the 2 6GB sata and 1 to the PCIe card. now 2 of the drives dont show up anymore. Before 1 drive of each pool was connected to one of the 6GB sata each.

 

[gpsguy]

I just looked up the spec's on your motherboard and was going to ask if the failed disks were on the same set of connections (6Gb/s or 3Gb/s).

• 2 x SATA 6Gb/s connectors (SATA3 0/1) supporting up to 2 SATA 6Gb/s devices
• 3 x SATA 3Gb/s connectors (SATA2 2/3/4) supporting up to 3 SATA 3Gb/s devices

I think i found the issue. The motherboard has 2 6gb and 3 3gb Sata connectors. I think the drives connected to the 2 6gb connectors always seem to disconnect. Just trying to confirm it now.

You're still better off on getting a supported HBA (to replace your card) and flash it to IT mode.

 

[joeschmuck]

Likely this is just the typical experience people have with using PCIe sata cards with FreeNAS. One post by @joeschmuck does, I recall, describe a unit that is successful.
Standard advice here is to get an HBA if you are out of onboard SATA ports.

I use a PCI-E card (see my system specs in my signature for specific model) and it works fantastic. HBA's work too but I tend to think outside the box so I don't use HBA's because for one simple reason, updating the HBA firmware use to be something that occured when FreeNAS changed and I didn't want to jump through those hoops. To be honest I'm not sure how often HBA fimrware needs to be changed these days, probably not as often. And HBA's tend to generate more heat, thus use more power. You are getting great advice from everyone in this thread, and HBA would be fine and most of them are heavily tested so compatibility issues would be reduced.

 

[Chris Moore]

To be honest I'm not sure how often HBA fimrware needs to be changed these days

It has been a long time now. Last time the driver changed there was no new firmware to go with it. Still on 20.00.07.00 and holding for around two years now.

 

[Apollo]

I don't really believe drive temperature would be the reason for disconnection. To me this is complete non-sense.
I understand running HDD at higher than what might be construde as normal should cause any harm to the system. Sure, it will force premature wear on the mechanical parts, but the electronics is expected to work in warmer temperature conditions.
I tend to believe the issue lies more on the software/firmware/driver side rather than hardware failure.

 

[Chris Moore]

I don't really believe drive temperature would be the reason for disconnection. To me this is complete non-sense.

Temperature of the controller. I have seen a controller overheat and fail. One of mine did and thereafter it only worked intermittently.
My point was that if the hard drives are that hot, the entire system is probably too hot.

I tend to believe the issue lies more on the software/firmware/driver side rather than hardware failure.

It is very likely that the controller being used is not fully supported under FreeBSD.

I use a dekstop PC (old one, Motherboard is a: gigabyte GA-H77-DS3H i5-3570K with 4x4GB DDR3 RAM, 600W PSU)

https://www.gigabyte.com/Motherboard/GA-H77-DS3H-rev-10#sp
Also at issue with the build is the integrated NIC.

think i found the issue. The motherboard has 2 6gb and 3 3gb Sata connectors. I think the drives connected to the 2 6gb connectors always seem to disconnect. Just trying to confirm it now.
YEP this seems to be the issue. I connected 2/3 mirrored drives to the 2 6GB sata and 1 to the PCIe card. now 2 of the drives don't show up any more. Before 1 drive of each pool was connected to one of the 6GB sata each.

The solution for all of that is to put all the storage drives on a single SAS controller like this:
https://www.ebay.com/itm/HP-H220-6G...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162862201664
You can get the same controller for a bit less if you get one that has not already had the firmware upgraded, but you would then need to upgrade the firmware yourself. Here is an example:
https://www.ebay.com/itm/HP-SAS9205...Controller-660088-001-638834-001/273685912742
We have a guide to upgrading the firmware in the resources section of the forum.
You would use a set of cables like this to connect your SATA drives to the SAS controller:
https://www.ebay.com/itm/Lot-of-2-M...-Forward-Breakout-Internal-Cable/371681252206
Also an option:
https://www.ebay.com/itm/Mini-SAS-S...reakout-Internal-Cable-3-28-Feet/283446003242

 

[quik]

Thank you for the explanation!
I was able to get my hands on a Dell H310, and i will be using that in the future. Since i live in Switzerland, its not easy to source such products sometimes ;)