New to FreeNAS, weird behavior on iometer testing

[ZFS Noob]

I'm new to ZFS and FreeNAS, but I like what I've learned over the last week, and I got a test build up on a server for testing.

Server Details

• Dual Xeon processors
• 72G RAM
• 4x 2TB ES.3 SAS drives as a striped mirror/RAID10/RAID1+0/whatever the ZFS terminology is for 2 mirrored vdevs sitting in a pool together.
• 4 Gig-E ports, though only one is used for testing.

The Test
The goal up-front is to benchmark how many IOPS this server/software can manage, and I wanted to do that in three parts:

• Asynchronously
• Synchronously with no SLOG
• Synchronously again, with the SSD SLOG in place.

My expectation is that synch SLOG is going to be near async, but not as performant, which is fine. I run virtual machines which include MySQL servers, and I'm much more concerned with transactions than I am with throughput.
So my test now is IOMETER, 10 workers, running all tests for 5 hours, against a 10 gig data set. The data is hosted on an iSCSI share that my laptop mounted, and my laptop's single GigE connection is plugged into an old 24 port Netgear unmanaged switch I pulled off the shelf to use for the test. I think the switch is good, but I threw out its sister as it was getting iffy after running for a few days.
The Behavior
Async was fast, though apparently I don't know how to read IOMeter results. Overall average IOPS were 6,121, though it also listed 15,367 transactions per second which is weird. I thought IOPS were ~= TPS. Average response time was around 1.5 ms.
Awesome.
Except I looked at the pretty graphs in FreeNAS and saw this:

Lots and lots of gaps in the performance data. I don't know why they're showing up, and that concerns me. Off the top of my head it could be:

• ZFS is poorly tuned and I'm running into that issue where queued writes get backlogged and things grind to a halt until things catch up.
• iometer doesn't run that fast that well on a laptop, so it was timing out.
• The network switch is crappy and was showing its own issues.
• Hell, maybe it was the cables I grabbed.
• Maybe the data collection routines on FreeNAS do this under load.

Since the memory usage graph is gapped too I'm thinking the last explanation is a good one, but I'm running the synchronous test now and I'm not seeing gaps in the logging. Of course, I'm "only" getting ~ 230 IOPS, and server load is quite a bit lower than it was when I was pushing 10,000 IOPS during those portions of the test.
So, what does that look like to y'all? If I rerun the test tomorrow, are there particular diagnostics I can run on the server that will tell me more?
The dataset was 10G, and ARC was 10G, and most everything should have been caching cleanly as every time I checked the server it was only hitting the disks every 5 seconds or so, and for less than a second each time. Maybe one transaction size clogged up the queue while I was away from the computers?

 

[cyberjock]

Well, since you have so much RAM your test sets are going to have to be 72GB of RAM or bigger. Else the entire test set will be in RAM and you'll appear to get freakin amazing performance. My guess is those 10kIOPS is where your RAM is helping you to have a smoking fast server.

Your attempt to force a 10G ARC with a 10G dataset was showing you recognize the problem. But the ARC isn't strictly file data. Some is for the metadata. And there's set limits via tunables for how much of each you can have(as well as minimums). There's ratios that are set(some might not be tunables). You might need a 15G ARC to get your full 10G dataset file in the ARC.

What you are trying to do is VERY complex. you can't just do some simple iometer tests and get results that mean anything(as you've noticed). Just ask my hero jyavenard with his benchmark results. He and I go around and around with benchmarks because there's only a few ways for the benchmarks to be useful, and a billion ways you can totally hose them.

I really can't help you out as much as I'd like because despite knowing alot about ZFS, I'd be stupid to tell you I know anything about tuning ZFS. From my 2 years in this forum I feel like you've gotta have a 4 year degree in ZFS to actually be smart about tuning.

Don't take my post as me trying to discourage you or talk down to you. What you are doing is precisely what I would do. And you're asking the same questions I'd be asking because I don't have solid answers. If anything, you are recognizing one of many problems and looking for insight. Unfortunately, there's not much you're going to get from here. :(

 

[ZFS Noob]

Don't take my post as me trying to discourage you or talk down to you. What you are doing is precisely what I would do. And you're asking the same questions I'd be asking because I don't have solid answers. If anything, you are recognizing one of many problems and looking for insight. Unfortunately, there's not much you're going to get from here. :(​

No worries.​

The goal is to cache everything if possible, so I'm good there. :) I host online forums like this one (only busier), and as I transition to new storage I'd like to get a feel for how many IOPS (and hopefully how many separate database servers) that server can handle. As it is, it looks like the answer is "ZFS can handle more than can be passed over a single GigE connection," which is fine.​

Have you see gappy performance graphs like that before, though? Is that normal in FreeNAS, or is something else happening?​

 

[cyberjock]

Gaps are usually from server downtime where no data is collected. I don't use the graphs much generally(home server) so I don't know when else it might "gap".

I'd figure if you are doing 72GB of RAM, and get a good size L2ARC(160-200GB), and a ZIL(if using ESXi with NFS) you'll probably be dealing with GigE limitations before you care about pool performance.

 

[ZFS Noob]

OK, I've not yet figured this out, but I've got more data. I'm rerunning the test and noticed gapping again. The server itself isn't hitting the drive, and the test laptop is at ~ 100% CPU.

I'm rerunning the test, and this time I specified a max file size that should be 9.5 G on the 10G volume it's accessing. It's possible this was related to being out of space, so we'll see...

 

[cyberjock]

Maybe CPU usage is so high that the data recording service(name slips my mind.. almost bedtime) isn't actually doing anything.

I do remember people complaining about data not being graphed because /data was full. But normally it just stops and never restarts because the old data doesn't expire for like a year or something.

 

[ZFS Noob]

Sorry - I wasn't clear. CPU is peaked on the client machine that's running the testing. I'm not sure what's going on there, so I'm doing some more testing.

Restarting with old data = 9,000+ IOPS due to ARC. Restarting with fresh data starts at ~ 1,400 IOPS and grows as data is cached. I wish I understood testing tools and methodologies better. At this point though, I can say "yep, even at 4 drives this is fast enough for my environment" and the question is how much playtime I want to devote to pretty graphs. ;)