Sudden performance reduction

[ryanpr]

I have an old, home built system running TrueNAS 12 U8.1 (more details below). I'm looking for direction on what to look at next.

Until yesterday it was more than fast enough for me. I'm not sure what changed, but the system responsiveness is horrible. Running zpool status takes > 5 seconds. Loading and navigating the UI is painful. Connecting to shares sometimes times out. I don't see any alerts or log errors which would lead a direction. I did have a few errors over the previous ~3 days about replication tasks "Timeout in head()". So it's possible this started a couple days before I noticed. But that just seems to confirm it's slow, not why it's slow. There are no smart failures or drive failures.

System has a SuperMicro X9SCM board with a Xeon E3-1240v2 and 16 GB of RAM. Pool has 4x 4TB drives split into 2 mirrors. Boot is a SSD mirrored with a USB thumb drive. I have a about a dozen SMB shares, a few iSCSI shares (to ESXi) and one AFP share (time machine). The VMs aren't very active and I tried disabling them to validate they're not causing the slowness. Only jail is syncThing. CPU is almost idle. RAM is near full, but 10GB is ZFS cache. Drive latency/temps/busy all look normal.

Thanks for any help

 

[Alecmascot]

How full is the pool ?

 

[ryanpr]

57% used on the pool. Likewise, the datasets seem to have plenty of room as well.

 

[Jessep]

Scheduled SMART tests?
SMART stats?

 

[ryanpr]

I checked smart tests and none were running at the time. When this started I did run the long tests on two disks, second is still running.

As an update, this morning the performance was the same. However, now, 3 hours later, the system seems to be fully back to normal. So, obviously something was happening, but I have no idea what it could have been.

 

[pschatz100]

Boot is a SSD mirrored with a USB thumb drive.

What is the purpose of mirroring an SSD with a thumb drive for boot? Thumb drives are the source for many headaches: they can create weird and unexplainable problems because they degrade before they fail completely. And they do fail more often than you would like - which is why the current recommendations are for booting from SSD's.

If you keep current backups of your system configuration, then you don't need the thumbdrive. In a worst case scenario, you can reinstall TrueNAS and restore the configuration. If you want mirrored boot drives, then use two SSD's. It will be much more reliable.

 

[ryanpr]

What is the purpose of mirroring an SSD with a thumb drive for boot? Thumb drives are the source for many headaches: they can create weird and unexplainable problems because they degrade before they fail completely. And they do fail more often than you would like - which is why the current recommendations are for booting from SSD's.

If you keep current backups of your system configuration, then you don't need the thumbdrive. In a worst case scenario, you can reinstall TrueNAS and restore the configuration. If you want mirrored boot drives, then use two SSD's. It will be much more reliable.

It used to be the norm to run thumb drives for boot. I had two of them mirrored for safety. One died about 3 years ago and I replaced it with an SSD in the boot pool. I like having a mirror of the boot drive to keeps the system running until a replacement device can be installed even though I keep backups. That said, I get your point, SSDs are much more reliable.

Oddly the performance came back. But at this point I'm using it as an excuse to upgrade to a new system, so it's a bit moot.

 

[pschatz100]

It used to be the norm to run thumb drives for boot. I had two of them mirrored for safety. One died about 3 years ago and I replaced it with an SSD in the boot pool. I like having a mirror of the boot drive to keeps the system running until a replacement device can be installed even though I keep backups. That said, I get your point, SSDs are much more reliable.

Thumb drives were the norm many years ago. FreeNAS/TrueNAS has been through many updates since then and the recommendations have changed because the way the system uses the boot drive has changed.

I still think you should remove the thumb drive and just boot from the SSD.

Good luck with your new system.