J
jpaetzel
Guest
I've oftentimes talked about how drastic the performance difference is
between a ZFS system on cache and a system off cache, and how quickly it
falls off. I decided to collect some hard data and graph it for
everyone so you could see what I'm talking about.
My test system is a Gen 2.0 FreeNAS mini with 32GB of RAM, and 4 WD Red
6TB hard drives in a RAID 10. There are also 2 Crucial M4 256GB SSDs
that each provide a 4GB ZIL partition and 16GB L2ARC partition to the
pool. (The remainder of the space of the SSDs provide a mirrored 200GB
pool of their own)
I'll note that my ZIL and L2ARC layout is not supported by the FreeNAS
GUI.
For my tests I used iozone running locally. lz4 compression was on and
provided a 1.31x compression ratio of my test data, which is somewhat typical for real
world usage.
I ran 4 iozone read tests of a single thread, starting at 1GB and
working up to 30GB.
arc_max was set to 18GB.
The test results measure read performance only.
Te presence of an L2ARC is mostly irrelevant for these tests. The L2ARC
only comes in to play after a repeated read workload is present for many
hours.
From the 1GB to 13GB tests the ARC hit % is 99.x% and read performance
is single threaded memory speed, or ~ 3.5GB/sec
The 14GB test sees the ARC hit percentage dip to 95% for the first two
runs, with a read speed of 1.4GB/sec. That's 40% of the performance of
the 99% hit rate runs. The second two 14GB runs see ARC hit % recover
and they too finish at 3.5 GB/sec
The 15GB run is much the same as the 14GB test.
The 16GB run only hits 99% ARC hit one of the three times, the other
three times the ARC hit rate is 90% and performance has fallen to 1.2GB/sec
From the 17GB to 20GB tests the ARC hit rate drops for 87% to 83% and
performance goes from 1.2GB/sec to 800MB/sec
From 21GB to 31GB ARC hit rates stabilize at 80% and performance settles
in at ~ 500MB/sec.
Within 7GB the performance drops to 1/7th of the 99% ARC hit rate value.
This is important because it doesn't matter if the system has 32GB of
RAM or 256GB. When your working set outgrows ARC by a few GB
performance is impacted drastically.
I've attached a pretty graph of the results as well as the raw data.
between a ZFS system on cache and a system off cache, and how quickly it
falls off. I decided to collect some hard data and graph it for
everyone so you could see what I'm talking about.
My test system is a Gen 2.0 FreeNAS mini with 32GB of RAM, and 4 WD Red
6TB hard drives in a RAID 10. There are also 2 Crucial M4 256GB SSDs
that each provide a 4GB ZIL partition and 16GB L2ARC partition to the
pool. (The remainder of the space of the SSDs provide a mirrored 200GB
pool of their own)
I'll note that my ZIL and L2ARC layout is not supported by the FreeNAS
GUI.
For my tests I used iozone running locally. lz4 compression was on and
provided a 1.31x compression ratio of my test data, which is somewhat typical for real
world usage.
I ran 4 iozone read tests of a single thread, starting at 1GB and
working up to 30GB.
arc_max was set to 18GB.
The test results measure read performance only.
Te presence of an L2ARC is mostly irrelevant for these tests. The L2ARC
only comes in to play after a repeated read workload is present for many
hours.
From the 1GB to 13GB tests the ARC hit % is 99.x% and read performance
is single threaded memory speed, or ~ 3.5GB/sec
The 14GB test sees the ARC hit percentage dip to 95% for the first two
runs, with a read speed of 1.4GB/sec. That's 40% of the performance of
the 99% hit rate runs. The second two 14GB runs see ARC hit % recover
and they too finish at 3.5 GB/sec
The 15GB run is much the same as the 14GB test.
The 16GB run only hits 99% ARC hit one of the three times, the other
three times the ARC hit rate is 90% and performance has fallen to 1.2GB/sec
From the 17GB to 20GB tests the ARC hit rate drops for 87% to 83% and
performance goes from 1.2GB/sec to 800MB/sec
From 21GB to 31GB ARC hit rates stabilize at 80% and performance settles
in at ~ 500MB/sec.
Within 7GB the performance drops to 1/7th of the 99% ARC hit rate value.
This is important because it doesn't matter if the system has 32GB of
RAM or 256GB. When your working set outgrows ARC by a few GB
performance is impacted drastically.
I've attached a pretty graph of the results as well as the raw data.
