Help understanding (what I think) are performance problems

[Crispin]

Hi folks,

I'm looking for a "your system is fine - chill" or "actually, something does not look right - go do some more reading".
Not sure if I am expecting too much from my setup.

I've just setup a new NAS on a HP Gen8.
Details are:

tank is made up of:
4 x 4TB reds, RADI1Z
1 x 500GB SSD cache
2 x 120GB SSD mirrored for SLOG

In a nutshell -
I can write across CIFS at 115MB/s so that saturates my NIC.

I have a jail with mysql installed. I noticed while doing a file load into a table that it seemed to perform very poorly. A iostat was suggesting I was getting a write of 10-12M which is pretty odd I think.
File loading was a 16GB single-column word list being loaded into a single column table.

I then started checking around with various bits and not sure if I am understanding it right.
I do a dd to check read and write and the write performs ok - average of 316MB/s over 4 or 5 runs.

The read confuses me - over the same runs it averaged 285MB/s

While doing that I had a iostat running. Both -v and without -v
The with-v shows that during the read the the spinny-disks are reading 40-50M (is bandwidth MB/s?) while the SSD is reading 2.12M and writing 16M
As you can see above, the read performance does not look like it's coming from RAM or from SSD.

The without-v option looks like such

So, am I in the you-need-to-read-more territory? :)
Is something wrong with my setup?

During all of this the CPU was petty bored, never going above 40%

TIA
C

 

[Robert Trevellyan]

Is something wrong with my setup?

Yes, you have L2ARC and SLOG on an under-specced system.

Remove both and repeat your tests. If the bottleneck is lack of ARC, add more RAM. When you've maxed out your RAM, if lack of ARC is still a bottleneck, consider reinstalling the L2ARC.

As for SLOG, unless you need sync writes and you're seeing that as a bottleneck (test by turning them off), it's unnecessary at best and potentially harmful at worst.

 

[Crispin]

Yes, you have L2ARC and SLOG on an under-specced system.

What makes you say that? Anything specific or just 'cause-you-said-it's-slow :)
I can't put anything more than a i3 into the HP so stuck there. Is CPU a bottleneck if it's not busy?

Remove both and repeat your tests. If the bottleneck is lack of ARC, add more RAM. When you've maxed out your RAM, if lack of ARC is still a bottleneck, consider reinstalling the L2ARC.

I can't add more RAM - 16GB is all she can take cap'in

I did do a test over the weekend without the SLOG and it performed worse than now with the mysql loads. I was only seeing 10-12M on the disk writes. Also, during the sql load the usage of the slog went quite high which suggests it was doing what it was meant to do?
I will try again though in a more "testy" way.

As for SLOG, unless you need sync writes and you're seeing that as a bottleneck (test by turning them off), it's unnecessary at best and potentially harmful at worst.

Mysql in a jail would issue sync writes would it not? That's my thinking behind adding the ssd for zil.


I must add, during these tests nothing else is on the NAS. It just services the jails.

 

[Dice]

4 x 4TB reds, RADI1Z

Try 2x vdev of mirrors.

 

[depasseg]

With a 4 disk RAIDZ1, you effectively have the IOPS of a single disk and the the sequential throughput of the aggregate of the 3 data drives. So yes, small read and write mysql transactions are going to appear much slower than a large read or write.

 

[Crispin]

With a 4 disk RAIDZ1, you effectively have the IOPS of a single disk and the the sequential throughput of the aggregate of the 3 data drives. So yes, small read and write mysql transactions are going to appear much slower than a large read or write.

Does this include reads?

In my setup though, why would the SSD not be used for the read test?

 

[Dice]

In my setup though, why would the SSD not be used for the read test?

Do you refer to this?

2 x 120GB SSD mirrored for SLOG

or this?

1 x 500GB SSD cache

 

[Crispin]

the latter - the 500GB cache. Why would the dd read not hit that? 5 write tests then 5 read tests - surely the 2-5th test should have read off that?

 

[Robert Trevellyan]

I can't add more RAM - 16GB is all she can take cap'in

The data structures ZFS uses to track the contents of L2ARC are stored in RAM. In a system with only 16GB of RAM, that RAM would almost certainly be better spent on ARC. The rule of thumb usually quoted in this forum is that 64GB is the minimum before L2ARC is worth considering.

EDIT: I forgot to mention, the bigger the L2ARC, the more RAM it takes to track the contents.

 

[Huib]

This might be the same problem I'm running into?
Big files fly like rockets but once you start using random read/writes on small files (my understanding is this is the case with sql) your performance drops down significantly.

I didn't experiment with SLOG yet because it's generally not recomended in the forums untill you have no other option (ram ram ram).
but I noticed that when copying many small files I don't even get one MB/s where as bigger files (few megs per file) easily give me 100 to 120 MB/s (saturating my nic) even on a 3 mechanical raidz1 vdev.

I intend to experiment with a zpool with just sd drives but that will take a while. I'll get back here if I find out it helps or not.

 

[Stux]

Are you using SMB?