CCB request was invalid

[pfunke]

Hello,

I'm building a new machine as a home server. Consistently, I am getting the following error which makes it impossible to get any storage pool to be stable. It happens with any disk on any SATA port after at most a few Gigabytes of writing data.
It happens with a RAID-Z with 3 disks as well as with just a single disk in striped mode. I created new pools for each test over and over again.

Code:

(ada0:ahcich5:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 68e4 0a 40 1c 00 00 01 00 00
(ada0:ahcich5:0:0:0): CAM status: CCB request was invalid
(ada0:ahcich5:0:0:0): Error 22, Unretryable error
(ada0:ahcich5:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 f8 93 4b 40 01 00 00 01 00 00
(ada0:ahcich5:0:0:0): CAM status: CCB request was invalid
(ada0:ahcich5:0:0:0): Error 22, Unretryable error
(ada0:ahcich5:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 48 cf b8 40 01 00 00 01 00 00
(ada0:ahcich5:0:0:0): CAM status: CCB request was invalid
(ada0:ahcich5:0:0:0): Error 22, Unretryable error

Hardware Configuration:
- AMD Ryzen 5 3400G
- ASRock X570 Pro4 with BIOS version 3.20 (AMD AGESA Combo-AM4 V2 1.0.8.0 Patch A)
- 32GB G.Skill Fortis DDR4-2400 DIMM CL15 Dual Kit
- Harddrives (connected to onboard SATA controller)
3x brand new 8000GB WD Red Plus WD80EFAX, older Samsung HD502HJ 500GB for testing
- old Crucial M4 SATA SSD as boot device (no problems with this drive)

I already ruled out the following possible reasons for the issue:
- operating system: it happens on FreeNAS 11.3-U5 as well as TrueNAS CORE 12.0-RELEASE
- RAM was tested with multiple passes of memtest86 without errors
- changed SATA cables, tried multiple new ones and an one from another working machine
- happens with all 3 new harddrives and with an older one that I know is working fine. SMART readings for all drives are fine and I suspect that it's not 4 drives failing at the same time. I even had one of the new drives run a long SMART test that completed without errors as well to be sure on this.
- swapped the PSU including the cables with one from another working machine
- went through the BIOS many times and tried different explicit and AUTO settings for the onboard SATA controller. All power saving features are off and it is definitely running in AHCI mode.

By now, I am a bit lost and the only thing I can think of is that either the mainboard (SATA controller) is broken out of the box or there is some major hardware incompatibility going one. But since the machine is running fine apart from the issue described, I would suspect the board is fine.
Unfortunately, I do not have a spare SATA controller lying around to check on this theory. What do you think? Send in the mainboard? Try a BIOS upgrade (newest BIOS uses AMD AGESA ComboAM4v2 1.1.0.0 Patch C)?

Thank you for any input!

 

[pfunke]

Quick update, I tried this out on FreeBSD as well as on Debian. Debian's more verbose syslog messages show unaligned writes and hard resets on the link.
This seems to be a broken controller on the mainboard and is definitely not related to FreeNAS/TrueNAS.

 

[panosxp]

Hello,

I actually have the exactly the same problem with my TrueNas server.
I use it as home file server. I was using FreeNas for over a year without any problem.
As soon as i installed TrueNas and when i tried to load the first files on the new server i started to see the same messages you mention above and then server is crashes.
Do you believe that these messages are definitely related with the SATA controller?