Write/Read problem - Huge drop down performance after upgrade 9.2.x

[Sasa Petrovic]

Hi guys,
I am facing a huge problem (for me :)) with my NAS after upgrade on 9.2.1.3
My hardware specs:
Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
with 16354MB of RAM
20TB of storage space (18TB of raid6).
Server has 2 NICS (1x4 ports and 1x2ports).
4 clients are connected direct to server via cat.7 cables with 1gbps connections.
After upgrade of OS I am facing huge write and read drop down that I can't even use storage at all.
In the meanwhile my raid controller battery was death and I have to replace it but I don't believe that it can be the cause of this drop down.
My ZFS version is v15.
Here are the stats:

[root@my-storage] /usr/local/www/freenasUI/tools# zpool iostat 1
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
storage 12.0T 6.09T 4 3 585K 330K
storage 12.0T 6.09T 427 18 53.1M 725K
storage 12.0T 6.09T 287 12 34.7M 46.5K
storage 12.0T 6.09T 537 16 66.8M 56.4K
storage 12.0T 6.09T 445 12 55.3M 38.5K
storage 12.0T 6.09T 539 14 67.4M 110K
storage 12.0T 6.09T 307 11 38.5M 717K
storage 12.0T 6.09T 381 14 47.2M 71.9K
storage 12.0T 6.09T 111 16 13.9M 58.9K
storage 12.0T 6.09T 229 6 28.6M 19.5K
storage 12.0T 6.09T 141 22 17.6M 962K
storage 12.0T 6.09T 0 35 0 117K
storage 12.0T 6.09T 359 18 44.6M 38.5K
storage 12.0T 6.09T 149 0 18.4M 0
storage 12.0T 6.09T 60 0 7.50M 0
storage 12.0T 6.09T 152 26 18.9M 792K
storage 12.0T 6.09T 26 29 3.25M 129K
storage 12.0T 6.09T 319 3 39.7M 4.00K
storage 12.0T 6.09T 329 0 40.3M 0
storage 12.0T 6.09T 171 0 21.5M 0
storage 12.0T 6.09T 201 17 24.9M 672K

storage 12.0T 6.09T 297 20 37.0M 193K

Is there anything more what I can check to detect some potential problem?
My raid controller is HP P400 and this battery is for some write cache module. Could be it or?

 

[cyberjock]

oh god.. please tell me you aren't mixing hardware RAID with ZFS.... PLEASE!

If you are, and you happen to have write caching on your RAID controller, YES, your battery is probably your problem.

You should also get rid of that hardware RAID immediately if your data is important.

Hardware RAID + ZFS = fail. (I even use this equation regularly because so many people do it...)

 

[Sasa Petrovic]

Ohh yeah! Damn...I am using this :D
Ok...Yes...data is important :)
I will start right away to backup data from storage to discs and reconfigure everything again.
Can u plz advise what to do next? :)
Just to leave discs without raid and to configure like?

 

[cyberjock]

Your best bet is to get a controller that isn't a RAID controller(like the M1015) and use that. You *want* a dumb HBA. The whole problem with RAID controllers is they do things like prevent SMART monitoring and testing, abstract the disk media, hide errors, etc. You want the dumbest lamest HBA that works(and if it's cheap that's even better!). For the cheapest that also works the best you want the M1015. ebay is the place to go!

 

[Sasa Petrovic]

In that moment when I did this kind of configuration - freenas forum wasn't so much populated and great like these days.
I didn't even had some examples (configurations) to copy. Now, in this case, I will fix everything and will do proper configuration from scratch.
Firstly I will install new battery to be able to operate normally and to be able to do next steps.
I will order ASAP this controller M1015 - price is normal!
Soon will post results of this story!
Thanks a lot.

 

[Sasa Petrovic]

Hmm...but...how many discs I can connect through M1015? I have 12 HDDs...firstly to check specs of it...

 

[Sasa Petrovic]

Ok...I know now=8. But I need at least 12. Something to recommend as good as M1015?

 

[warri]

You could use two M1015s. ;)

 

[Knowltey]

Perhaps a tad off topic, but is there a way I can add a Disk I/O chart to my FreeNAS if it isn't already there or is it only certain hardwares that support that graph?