Six drives failing?

[Arwen]

Actually, 12 disks in RAID-Zx is right at the large size. We have seen larger here, but usually they come to complain about performance after using it for a while.

 

[Davvo]

Yup, I'm just saing that's not common.

 

[ITGuy1024]

@ITGuy1024 you don't see very often 12 spinners in a vdev on this forum, mostly because the use cases don't require it: those who have such a number of disks usually started with 6 in a vdev and then doubled their capacity, so 2 vdevs of 6 disks each.
Anyway, please post the smart data.

Well I started with a 4 disk Z1 vdev. Then another 4 disk Z1 vdev when that ran out.
Figured I would just make a big Z2 at this point and have the extra space.

SMART for da0 and da1 failed. da10 never finishes the test.

I looked at this a bit more closely over my vacation.
da0
da1
da10
da11
These are all on the same port on the sas card, same sas breakout cable. da11is the only one that hasn't had an issue yet and these drives I actually bought new around...2018 I think? I'm wondering if there's a problem with the sas breakout cable. I have some new ones on the way.
I did replace da1. The pool is reslivering now. I'll post updates.

 

[ITGuy1024]

Actually, 12 disks in RAID-Zx is right at the large size. We have seen larger here, but usually they come to complain about performance after using it for a while.

Actually performance has been GREAT so far. Much better than the 4 disk Z1 vdevs I had.
I'm only using this for my Emby media storage so I'm mostly doing reads and not writes to it. Assuming the people with speed issues were doing writes to it?

 

[MyRequiredName]

I am new to all of this. But, is it possible to replace a "failing" disk, then test it outside of the system on its own, using a simple sata connector as opposed to the breakout cable? Wouldn't that be a good test?

 

[morganL]

Well I started with a 4 disk Z1 vdev. Then another 4 disk Z1 vdev when that ran out.
Figured I would just make a big Z2 at this point and have the extra space.

SMART for da0 and da1 failed. da10 never finishes the test.

I looked at this a bit more closely over my vacation.
da0
da1
da10
da11
These are all on the same port on the sas card, same sas breakout cable. da11is the only one that hasn't had an issue yet and these drives I actually bought new around...2018 I think? I'm wondering if there's a problem with the sas breakout cable. I have some new ones on the way.
I did replace da1. The pool is reslivering now. I'll post updates.

My bet would be on either the SATA port or cable... not on multiple drives failing simultaneously.

 

[Arwen]

Actually performance has been GREAT so far. Much better than the 4 disk Z1 vdevs I had.
I'm only using this for my Emby media storage so I'm mostly doing reads and not writes to it. Assuming the people with speed issues were doing writes to it?

Wider sized RAID-Zx tend to do better with larger files. So, music and videos likely perform well on a 12 disk RAID-Zx. It is likely smaller files, or worse, zVols, that do poorly on wide RAID-Zx. Oh, and datasets that have a lot of churn, like backups that include small files, may not like wide RAID-Zx.

 

[ITGuy1024]

So while I was waiting for my new breakout cables to show up I replaced some of the drives. The errors have stopped. It may well be the case that I was having some bad luck and multiple drives on the same cable start to go bad at the same time.

 

[jgreco]

So while I was waiting for my new breakout cables to show up I replaced some of the drives. The errors have stopped. It may well be the case that I was having some bad luck and multiple drives on the same cable start to go bad at the same time.

It could also be that the SFF-8087 end was not fully or properly seated. Happens.

 

[ITGuy1024]

It could also be that the SFF-8087 end was not fully or properly seated. Happens.

That would have been too easy. I checked the cables a while ago and the errors kept happening. They only stopped once the drives were replaced.