New System Performance Issues

[nwillia09]

I am having performance problems with my new Thinkmate pre-configured 60 drive truenas server. Everything was going great intially. I was able to get very impressive reads and writes on an empty pool. Last week i migrated 18TB of data to the new system. This data is mainly small tiny files but there are an occasional 50GB file here and there. This week I attempted to add additional data to the system and noticed that the performance was not as good getting short bursts of great performance followed by stalls that would last up to a minute. I decided to reboot the system which didnt go as smoothly has I expected. The reboot took about 30 minutes and while watching the output i noticed it took a very long time to import the pool showing lots of spa_load importing and mentioning of metadata syncing. Now that the system is online its performing worse than before the restart. Navigating the folders via SMB share is extremely slow taking 20+ seconds to list sub folders. I noticed that the ZFS Cache size is not growing very fast in the GUI.

At this point I dont know where to begin with troubleshooting the issue. I have data on the system that hasn't been successfully backed up yet and the poor performance is making it very difficult to capture the data. Until I get this data copied off I cant do anything that would destroy the pool.

I do have one disk in the pool that is showing predictive failures but hasnt actually gone offline or tripped errors in zpool status.

Any help to point me in the right direction is very appreciated.

System
Truenas 12.0U8
60x 18TB Seagate EXOS SAS 7x 8 Wide vdevs 4 hot spares
2x - Intel Xeon 4314 CPUs
1TB of RAM
2x - 25Gb interfaces bonded with LACP

Thanks!