SSD Write speeds died

[BSTAMPER]

So a while back I posted here (link at bottom) about some I/O issues I was having. In that instance I had added some SSD's expecting a huge I/O boost and in the end it was an issue with the network. Upgraded to 10G and everything was fine. Recently, for some reason the performance on the SSD's went through the floor. Using dd in a linux vm I run it and on the SAS drives in my other pool I consistently get 400-600 both read/write. All good for what I need. If I storage vmotion that same linux box over to the SSD pool the read seems OK but the write is like 14-17. I've blown away the pool and set it up as mirror, shut off compression, etc. just to see if I could figure out what was going on. When I originally set it up it was great and then something along the way changed and I'm just not sure what. I've rebooted freenas etc.

I get alarms only from disks in this pool so it seems to be complaining about the drives for some reason:

Device /dev/gptid/2c7c9e2b-a40d-11ec-afad-a0369f4a8344 is causing slow I/O on pool SSD-Pool-01.​

I'm starting to wonder if I didn't essentially burn them out?

They are on a 9200-8i. The disks are all CT1000BX500SSD1.

I ran the SMART Tests on them and they all look to be functioning fine with no issues.


Previous post:

Added SSD's assuming performance? None?

So I have a DL380P G9 and was using what it came with (free) on the HP storage controller. Ran out of space and figured if I'm gonna need more I might as well buy SSD. Bought some 1TB SSD's and threw them on a 9200-8i. Everything I do on this is VMFS. I somewhat assumed I'd see a huge...

www.truenas.com

Any thoughts or anything you all think I could check would be appreciated.

 

[sretalla]

zpool trim poolname ?

Maybe that's slowing you down? (check your pool settings for auto trim too, but I'm not confident it's doing the job you would think it should)

 

[mrpasc]

The CT1000BX500SSD1 is not really good choice:
https://www.tomshardware.com/reviews/crucial-bx500-ssd,5377.html
As they are DRAM-less they have bad sustained write rate. How is fill rate of the pool? Might be there is not enough space left for the pseudo-SLC write cache they use?

 

[BSTAMPER]

Yea I've tried a couple of things. Like I said when they first were deployed they were great. I just noticed the performance issues here in the last weeks and finally started digging into it. I've built and rebuilt the pools and tested but no combination of anything I do seems to work at this point.

 

[BSTAMPER]

I think you are right about the disks. I found this post below which seems to be the same issue I'm having. The SMR drives run out of new space to write to and as a result the performance tanks. I've ordered 4 new WD SA500's to replace them with. Going to guess that'll make all the difference and last longer.

All drives alerted "Device is causing slow I/O"

Note: I'm not physically able to access this hardware right now due to covid so my ability to try different things is limited, just trying to confirm my suspicion that this is the worst case physical hardware degradation I've been recently trying to diagnose really poor dd and smb performance...

www.truenas.com