TrueNAS12.0U4 ZFS Pools after several weeks "not usable"

[Starion87]

Hello,

we have 2 HPE Apollo 4510 GEN10 with the HPE P408i controllers,
2 Pools, each with 30 Disks (3x vDev with 10 12TB Disks in RAIDZ2)
Each pool has a (mirrored) slog device and 2 cache devices.
The P408i is runnung in "mixed Mode" wich means if the drive isn unconfigured,act as an HBA. All Drives are unconfigured.
We present the data (ZVOLS) with iSCSI over Qlogic FC to ESXi.
Firmware is the newest available.

We have done burn-in- and performance-tests with good results, 3 weeks long.
After running 3 weeks in "production" (as a Backup-Device) we cannot access the data on all pools.
The whole system is very slow accessing the disks/pool (creating a file with "touch" let the ssh-session hung).
Using another controller (eg. LSI 3008) the pool is OK. But this controller has problems with SAS Expander... not showing all drives for all pools.
This is the same an both servers.

The Driver for the P408i is the smartpqi from microsemi.
Already tried a fresh install. nothing helps..

Any ideas?

Console messages looks like this... but no other error messages:

Thanks a lot!

 

[Starion87]

Oh. some more infos:
RAM is 192GB, CPUs are 2x Intel(R) Xeon(R) Gold 5218 CPU
Booting from NVMEe M.2 SSDs (ZFS Mirrored)
Server is running in UEFI Mode

Also some other messages...:

 

[Starion87]

So, we updates the smartpqi with the newer one from microsemi and the problem is solved.
When the System is comming up with the new driver, it finds checksum errors on many disks (10 in my case) and started a resilver with a very good speed of 1GByte/sec Read und 100BMyte/sec Write.
All data is online and healthy.
Has anyone else this problem?
Is it OK to use a newer driver?
Any known side-affects?

Thomas

 

[NugentS]

You are using a RAID controller - which is not advised.