SOLVED 7 (random) disks work, add an 8th one and... All goes awry :(

[devnullius]

At boot, in no particular order...

CAM status: CCB request completed with an error
Retrying command
WRITE(10). CDB: <EDIT>

SCSI sense: ILLEGAL MEDIUM ERROR asc:11,0 (unrecovered read error)

Error 5, Unretryable
Info: 0x3df93a48

SCIS sense: HARDWARE FAILURE asc:44,0 (internal target failure)

(then it reboots)

Supermicro X8DA3/X8DAI
Avago MPT SAS2

Background:
I have 1 ZFS pool with 3 disks.

Today, I added 5 more and created a new pool to test it all.

But with 8 disks total, I encounter all kinds of problems (see above). Unplug one (no matter which one) and all is ok. BUT with all 8 disks plugged in, my first ZFS pool suddenly needs data redundancy checks if I boot with 7 disks after the reboot (and after unplugging 1 of the 5 new ones). After that, the first pool is ok again.

Where should I start looking for this problem? :) I'll be testing a bit more to see if I learn more: 1 bay always seems to be in the middle of the problems... But if I double-check that bay, all seems ok. So for now the main problem = 7 is company, 8 is a crowd ;p


Post-edit: I now got this in the console log (I've seen it before with the 8 disks attached) --> The volume ZFSPool1 state is ONLINE: One or more devices are faulted in response to IO failures.
Which sounds like something is wrong with 1 of my 3 primary disks. This is not the case. Remove of of the 5 new disks, and the first ZFS Pool with the 3 disks works like a charm again.

 

[Nick2253]

It helps legibility a lot if you use code tags for your errors.

It would also help if you posted your FreeNAS version and your detailed hardware. The X8DA3 and X8DAi are different motherboards, so which one do you have? How are your drives connected? You mention 1 bay, so are you using a case with a backplane? Which case?

My immediate thought is a power issue.

Have you successfully completed burn-ins on all your new drives? Are you getting anything in SMART tests? Have you run SMART tests on all your drives? If you plug drives into different ports, does it fix the problem? Does it matter how many drives you have on each controller?

 

[BigDave]

What type of chassis is this?
Try taking the 3 Drive pool and moving it all around in the drive bays making sure
it boots and comes up using all the different bays. This will eliminate slot issues.

 

[BigDave]

It helps legibility a lot if you use code tags for your errors.

Please and thank you!

 

[devnullius]

(I quickly typed some errors that stood out)

3rd slot doesn't seem to be a problem, though it keeps being suspect. Hopping around disks doesn't change anything either. Plugging in the 8th disk... Bwergh.

Lack of power is a good one; while I have a 1200W PS, I think the backplane is connected with just 1 Power Adapter. I don't think I have an option to add a 2nd one. I'll look again.

Post-edit: I might have spoken too soon when it came to Bay 3. At reboot, while all looked good, I now get:

Logical Block Address Out Of Range & Medium Error asc:11,0 (unrecovered read error).
I'll put Bay 3 disk in (now empty) bay 4 - see what gives :)

Thank you!
(updated my signature)​

 

[devnullius]

Aaaaah thank you whomever said 'probably lack of power' - you were correct. The backplane had 2 power adapters, true. BUT they were both split from the same PS Power Cable... With 8 disks I probably overloaded that cable. I now have re-arranged it all, now both backplane power ports have their own connection to the PS. No more problems, no more mystery, no more red lights - all is smooth. I probably bought 5 disks too many a few days back when I noticed my previous 4 weren't working as expected (I thought overheating was the cause).

Thank you for the pointer. One step closer to my new NAS - only 2 years underway ;p

xx Nick

 

[BigDave]

Lack of power is a good one; while I have a 1200W PS, I think the backplane is connected with just 1 Power Adapter. I don't think I have an option to add a 2nd one. I'll look again.

I can say that my experience has been that backplanes in general will typically run four 3.5in drives off of
a single four pin molex plug. If your backplane has say 12 bays, there would in most cases be four power ports.