Slow writes but fast reads

[Dude2k]

Hello together,

I have a very strange problem using FreeNas 9.2.1.2

Hardware:
Mainboard: Gigabyte A-B75N B75 RG SI
CPU: Celeron G1620T
RAM: 8GB
Raid Controller: HP P400
HDD: 5x WD Green 2TB

Software:
ZFS Encrypted

Before:
First everything was fine. I was able to copy ~ 6.1 TB on the raid with a good performance (cifs).

Problem:
Now writing is very slow ~2.8 MB/s (cifs, but same with dd) but reading is at ~80MB/s

Do you have some ideas for me?

 

[ser_rhaegar]

What is your CPU usage when writing?

Also your drive may be past the 80% mark. The more filled the drive, the slower the writes. It has to look for places to put your data due to fragmentation (it is better explained elsewhere if you search for it).

 

[Dude2k]

It's between 2% and 8% (writing) and much higher when reading

My volume as 86%. But you say, that 6% make a difference of >=60MB/s in writing?

 

[cyberjock]

It sure can. Read up on Copy-On-Write and how it messes up files with regards to fragmentation. Also, ZFS changes its behavior when you reach a certain percentage. At one point it flips like a switch, going from performance mode to "fill the drive" mode. Guess which one is slower. The difference is VERY significant too.

 

[ser_rhaegar]

It could be. The more full the drives are, the slower the writes. 80% isn't a hard rule, depends on your system and fragmentation, speed issues could start earlier or later.

 

[cyberjock]

You also need to realize a few things:

1. Your CPU doesn't support AES-NI. Solely based on this, I don't feel you should have any room to complain about with regards to performance. We tell people flat out if you want encryption go with AES-NI or don't complain about your speed.
2. You have the minimum amount of RAM. The manual recommends 8GB minimum, and it makes it pretty damn clear that RAM is a big big factor with regards to performance. More is always better. I've mentioned it at least 4 times in my noobie guide. So again, since you aren't exactly ensuring your pool can perform efficiently by giving it lots of RAM, you really don't have room to complain on this front either.
3. RAID controllers are flat out a bad decision for ZFS. Even with passthrough they can cause their own performance problems(let alone possible zpool corruption without warning).

So you can see how, from my perspective, you've made multiple mistakes, but are upset. Sorry bro, but just because of #3, you probably should be happy you haven't lost your pool. And I can bet you'd care less about your performance if you didn't have a pool. ;)

From my perspective you've done multiple things wrong, it's not working out, and you shouldn't be the least bit surprised about this.

 

[Dude2k]

Hello together,

thank's for your comments.

1. AES-NI is not a problem because at < than ~80% it was blazing fast
2. i understand but as i said in 1. - it worked before
3. same as point 1. before this mark the raid controller performed fine

So in my oppinion I should switch to another distro because it doen't fit to my hardware.

Thank you!

 

[cyberjock]

I disagree with your assessment of all 3 of those comments.

#1: AES-NI is one part of the workload your server has to deal with. As the server goes from 0% full to some higher percentage the work your computer has to perform will slowly increase as well. It's not linear, and it can be like a cliff where you suddenly jump off it and can't really get back on track.

#2: So what. See #2. I had a system that was starved of RAM. Went to bed and it worked fine. Woke up and I couldn't stream a single HD movie without it stuttering. ZFS uses RAM, and its like a cliff in some aspects. As long as your RAM is sufficient to allow prefetch to function efficiently all is fine. But if you go over that edge because your pool has enough data, then things fall apart. So not buying this as a valid justification either.

#3. If your controller is doing read ahead caching(which it can and does support... I checked) and your pool gets to a point where the read-ahead on your RAID controller isn't gaining you anything but is hurting performance because of fragmentation, then your RAID card could actually be completely responsible for your problems.

So sorry, I disagree with all of your comments. Well, except one. Your hardware is inadequate for FreeNAS and you should seek out alternatives or be ready to buy more appropriate hardware for the task.

Good luck in whichever option you choose. Be warned that if your zpool was created in FreeNAS 9.2.1.x you will not be able to import the pool anywhere else since FreeNAS uses the latest ZFS code, which hasn't been incorporated anywhere yet AFAIK.