Unretryable error (One or more devices has experienced an unrecoverable error.)

[m3ki]

So here is what I got so far

Link

Each drive had error about a day apart.
2 are on same controller, one on another controller. 2 Drives Were purchased at the same time.
3rd drive a year ago.

 

[m3ki]

So I got some more errors this time they didn't cause any issues and I think retry worked on for the commands.
This time it was on my 2 ssds:

Code:

Jul  3 17:11:39 anomaly ahcich1: Timeout on slot 5 port 0
Jul  3 17:11:39 anomaly ahcich1: is 00000000 cs 00000020 ss 00000000 rs 00000020 tfd c0 serr 00000000 cmd 0004c517
Jul  3 17:11:39 anomaly (ada1:ahcich1:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
Jul  3 17:11:39 anomaly (ada1:ahcich1:0:0:0): CAM status: Command timeout
Jul  3 17:11:39 anomaly (ada1:ahcich1:0:0:0): Retrying command
Jul  3 17:36:29 anomaly usbhid-ups[2254]: Got disconnected by another driver: Device busy
Jul  3 18:00:00 anomaly autosnap.py: [tools.autosnap:61] Popen()ing: /sbin/zfs snapshot -r "zroot/homes@auto-20160703.1800-72h"
Jul  3 18:00:01 anomaly autosnap.py: [tools.autosnap:61] Popen()ing: /sbin/zfs destroy -r -d "zroot/homes@auto-20160630.1800-72h"
Jul  3 18:39:19 anomaly ahcich0: Timeout on slot 22 port 0
Jul  3 18:39:19 anomaly ahcich0: is 00000000 cs 00400000 ss 00000000 rs 00400000 tfd c0 serr 00080000 cmd 0004d617
Jul  3 18:39:19 anomaly (ada0:ahcich0:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
Jul  3 18:39:19 anomaly (ada0:ahcich0:0:0:0): CAM status: Command timeout
Jul  3 18:39:19 anomaly (ada0:ahcich0:0:0:0): Retrying command
Jul  3 20:13:21 anomaly usbhid-ups[2254]: Got disconnected by another driver: Device busy
Jul  4 00:00:00 anomaly syslog-ng[5387]: Configuration reload request received, reloading configuration;
Jul  4 00:03:49 anomaly ahcich1: Timeout on slot 15 port 0
Jul  4 00:03:49 anomaly ahcich1: is 00000000 cs 00018000 ss 00000000 rs 00018000 tfd c0 serr 00000000 cmd 0004cf17
Jul  4 00:03:49 anomaly (ada1:ahcich1:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
Jul  4 00:03:49 anomaly (ada1:ahcich1:0:0:0): CAM status: Command timeout
Jul  4 00:03:49 anomaly (ada1:ahcich1:0:0:0): Retrying command
Jul  4 00:54:27 anomaly ahcich0: Timeout on slot 14 port 0
Jul  4 00:54:27 anomaly ahcich0: is 00000000 cs 0000c000 ss 00000000 rs 0000c000 tfd c0 serr 00000000 cmd 0004ce17
Jul  4 00:54:27 anomaly (ada0:ahcich0:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
Jul  4 00:54:27 anomaly (ada0:ahcich0:0:0:0): CAM status: Command timeout
Jul  4 00:54:27 anomaly (ada0:ahcich0:0:0:0): Retrying command
Jul  4 01:16:20 anomaly usbhid-ups[2254]: Got disconnected by another driver: Device busy
Jul  4 08:18:28 anomaly usbhid-ups[2254]: Got disconnected by another driver: Device busy
Jul  4 08:44:00 anomaly notifier: Stopping smartd.
Jul  4 08:44:00 anomaly notifier: Waiting for PIDS: 48302.
Jul  4 08:44:00 anomaly notifier: smartd not running? (check /var/run/smartd.pid).
Jul  4 08:44:00 anomaly notifier: Starting smartd.
Jul  4 09:00:01 anomaly autosnap.py: [tools.autosnap:61] Popen()ing: /sbin/zfs snapshot -r "zroot/homes@auto-20160704.0900-72h"
Jul  4 09:00:01 anomaly autosnap.py: [tools.autosnap:61] Popen()ing: /sbin/zfs destroy -r -d "zroot/homes@auto-20160701.0900-72h"

ada1 and ada0 are my SSDs.

All this time I was writing heavily to the array.

smart tests show no errors on any of the drives... they are under 1y old.

This makes me think, i saw somewhere on the forums smart can cause issues on freenas andif it invokes the command it would "freeze" the device for few seconds?

My config:

Should I change power mode to something else?
What if device was busy with smart or vice versa and command failed and that's what caused the error (obviously all my devices are set to never sleep)?