Possible Failure on brand new setup

[{HD}]

I have a build that is only a few months old built with all new hardware since my last build was slowly failing. All drives went through a smarttest(long) and badblocks(short). The ram was tested for 4 days with memtest. I get the following errors in my console but only when I am copying or deleting large amounts of data. I have searched around and everything says loose wires but I have tested and double checked and replaced all my sata cables.

Does anyone know what this means?

Code:

Jan  1 00:00:00 freeNas newsyslog[11769]: logfile turned over
Jan  1 00:00:00 freeNas syslog-ng[1100]: Configuration reload request received, reloading configuration;
Jan  1 00:00:00 freeNas syslog-ng[1100]: Configuration reload finished;
Jan  1 20:59:44 freeNas (ada4:ahcich4:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 90 5f 20 40 4d 02 00 00 00 00
Jan  1 20:59:44 freeNas (ada4:ahcich4:0:0:0): CAM status: ATA Status Error
Jan  1 20:59:44 freeNas (ada4:ahcich4:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan  1 20:59:44 freeNas (ada4:ahcich4:0:0:0): RES: 41 10 90 5f 20 00 4d 02 00 00 00
Jan  1 20:59:44 freeNas (ada4:ahcich4:0:0:0): Retrying command
Jan  2 00:00:00 freeNas syslog-ng[1100]: Configuration reload request received, reloading configuration;
Jan  2 00:00:00 freeNas syslog-ng[1100]: Configuration reload finished;
Jan  2 12:49:53 freeNas (ada8:ahcich11:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 20 30 34 8b 40 3c 02 00 00 00 00
Jan  2 12:49:53 freeNas (ada8:ahcich11:0:0:0): CAM status: ATA Status Error
Jan  2 12:49:53 freeNas (ada8:ahcich11:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan  2 12:49:53 freeNas (ada8:ahcich11:0:0:0): RES: 41 10 30 34 8b 00 3c 02 00 00 00
Jan  2 12:49:53 freeNas (ada8:ahcich11:0:0:0): Retrying command
Jan  2 12:50:18 freeNas (ada9:ahcich12:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 48 d9 91 40 3c 02 00 01 00 00
Jan  2 12:50:18 freeNas (ada9:ahcich12:0:0:0): CAM status: ATA Status Error
Jan  2 12:50:18 freeNas (ada9:ahcich12:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan  2 12:50:18 freeNas (ada9:ahcich12:0:0:0): RES: 41 10 48 d9 91 00 3c 02 00 00 00
Jan  2 12:50:18 freeNas (ada9:ahcich12:0:0:0): Retrying command
Jan  2 12:50:25 freeNas (ada9:ahcich12:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 48 da 91 40 3c 02 00 01 00 00
Jan  2 12:50:25 freeNas (ada9:ahcich12:0:0:0): CAM status: ATA Status Error
Jan  2 12:50:25 freeNas (ada9:ahcich12:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan  2 12:50:25 freeNas (ada9:ahcich12:0:0:0): RES: 41 10 48 da 91 00 3c 02 00 00 00
Jan  2 12:50:25 freeNas (ada9:ahcich12:0:0:0): Retrying command
Jan  2 12:50:36 freeNas (ada9:ahcich12:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 20 08 28 96 40 3c 02 00 00 00 00
Jan  2 12:50:36 freeNas (ada9:ahcich12:0:0:0): CAM status: ATA Status Error
Jan  2 12:50:36 freeNas (ada9:ahcich12:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan  2 12:50:36 freeNas (ada9:ahcich12:0:0:0): RES: 41 10 08 28 96 00 3c 02 00 00 00
Jan  2 12:50:36 freeNas (ada9:ahcich12:0:0:0): Retrying command
Jan  2 12:50:38 freeNas (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 20 60 37 96 40 3c 02 00 00 00 00
Jan  2 12:50:38 freeNas (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan  2 12:50:38 freeNas (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan  2 12:50:38 freeNas (ada0:ahcich0:0:0:0): RES: 41 10 60 37 96 00 3c 02 00 00 00
Jan  2 12:50:38 freeNas (ada0:ahcich0:0:0:0): Retrying command
Jan  2 15:35:16 freeNas ahcich11: Timeout on slot 27 port 0
Jan  2 15:35:16 freeNas ahcich11: is 00000000 cs 08000000 ss 00000000 rs 08000000 tfd d0 serr 00000000 cmd 0004db17
Jan  2 15:35:16 freeNas (ada8:ahcich11:0:0:0): DSM TRIM. ACB: 06 01 00 00 00 40 00 00 00 00 01 00
Jan  2 15:35:16 freeNas (ada8:ahcich11:0:0:0): CAM status: Command timeout
Jan  2 15:35:16 freeNas (ada8:ahcich11:0:0:0): Retrying command
Jan  2 18:59:48 freeNas ahcich8: Timeout on slot 24 port 0
Jan  2 18:59:48 freeNas ahcich8: is 00000000 cs 01000000 ss 00000000 rs 01000000 tfd d0 serr 00000000 cmd 0004d817
Jan  2 18:59:48 freeNas (ada6:ahcich8:0:0:0): DSM TRIM. ACB: 06 01 00 00 00 40 00 00 00 00 01 00
Jan  2 18:59:48 freeNas (ada6:ahcich8:0:0:0): CAM status: Command timeout
Jan  2 18:59:48 freeNas (ada6:ahcich8:0:0:0): Retrying command
Jan  2 21:48:53 freeNas (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 e0 8b cb 40 38 02 00 00 00 00
Jan  2 21:48:53 freeNas (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jan  2 21:48:53 freeNas (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jan  2 21:48:53 freeNas (ada0:ahcich0:0:0:0): RES: 41 10 e0 8b cb 00 38 02 00 00 00
Jan  2 21:48:53 freeNas (ada0:ahcich0:0:0:0): Retrying command

 

[SweetAndLow]

Check your sata cables. Something probably isn't connected correctly or the cable is bad.

 

[{HD}]

@SweetAndLow What is the Rolls Royce of sata cables? I am prepared to just buy 10 new ones to rule out cables as the issue.

 

[SweetAndLow]

@SweetAndLow What is the Rolls Royce of sata cables? I am prepared to just buy 10 new ones to rule out cables as the issue.

No clue I have never had a bad one. I just use whatever comes in the motherboard box. The good ones usually have a clip that locks them in place.

 

[Alecmascot]

I think we need a hardware list

 

[joeschmuck]

I have a build that is only a few months old built with all new hardware since my last build was slowly failing.

You have been a long time member so you should know what I'm going to ask you... Post your hardware configuration and what version of FreeNAS/TrueNAS you are using. If it's a VM, maybe post the config file for it (.vmx for ESXi).

At face value it appears to be the controller is the problem and I say that becasue there are too many issues across the platform to be a handful of cables. If you are using some sort of drive backplane then that could be the issue. Without ab accurate hardware list I'm guessing.

EDIT: Are you sleeping the drives?

 

[{HD}]

I have tried many different configurations of sata cables and I am beginning to think it is the mobo. Let me know what you people think.

MOBO: C3758D4I-4L
- I did all the available updates for bios and bmc
HDD: IronWolf 6TB x 6, WD Red 6TB x 4
- RAIDZ2
- No read errors, write errors, or checksum errors listed in pool status.
RAM: Samsung M393A4K40CB1-CRC 32GB x 2

BUILD: FreeNAS-11.3-U5

Thanks.

 

[Alecmascot]

HDD: IronWolf 6TB x 6, WD Red 6TB x 4

What are the exact model numbers ?
Are drive 0,4,6,8,9 related ?

 

[{HD}]

What are the exact model numbers ?
Are drive 0,4,6,8,9 related ?

The WD Reds I believe are the shingled ones. I don't have models handy. They are not related. 0, 4, 6, 8, and 9 are both WD and IW. I would have to physically open the case to see if they are related on the controller side. If you look at the MOBO there are 4 drives on a mini-sas another 4 on the other mini-sas and the rest as straight sata ports.

 

[Redcoat]

If you look at the MOBO there are 4 drives on a mini-sas another 4 on the other mini-sas and the rest as straight sata ports.

And "the rest" is 5. I strongly suggest you do open the case and determine which drive is connected to what in order to answer @Alecmascot 's point.

 

[Alecmascot]

FYI : https://community.wd.com/t/wd40efax-drives-idnf-when-resilvering-zfs-array/248785

 

[{HD}]

And "the rest" is 5. I strongly suggest you do open the case and determine which drive is connected to what in order to answer @Alecmascot 's point.

Well it is true there may be 5 sata ports on the board but I am using 4 from mini-sas #1 and 4 from mini-sas #2 and of my 10 drives that only leaves 2 on the straight sata. I will indeed open my case when I am near it next and see what is plugged into where but I think @Alecmascot was originally thinking are the drives that are failing all WD or all IW.

 

[Redcoat]

Yes - drive serial numbers and x-ref to drive numbers, plus mapping the drives to the ports, will be good base information.

 

[joeschmuck]

OMG, I need to take a step back. I missed the SMR questions again, I was making assumptions again. Good catch!

Well it is true there may be 5 sata ports on the board but I am using 4 from mini-sas #1 and 4 from mini-sas #2 and of my 10 drives that only leaves 2 on the straight sata. I will indeed open my case when I am near it next and see what is plugged into where but I think @Alecmascot was originally thinking are the drives that are failing all WD or all IW.

Why not just port the output of dmesg and then we should be able to make heads and tails of the data. You could look at the physical connections later.

 

[Heracles]

OMG, I need to take a step back. I missed the SMR questions again

Are you making honor to the description you gave yourself as "Old Man" @joeschmuck