ZIL drives not being accessed

[Chris LaDuke]

Hey all. I am a relateive n00b when it comes to zfs. I have a Supermicro 6025b-3 server with 12 GB RAM, 2 dual intel Gb nics, 6 300 GB 15k SAS drives, and two Intel DC S3700's. I am running it as a datastore for ESXi 5.1. I am connecting via iSCSI using mpio. I have the 6 SAS drives in 3 mirrors and I added the two Intel's as an SLOG. I am currently getting what I would consider poor performance (30MB/sec write). In an effort to determine why I began collecting some data.

Here is a snapshot of my pool
[root@NAS1 ~]# zpool status
NAME STATE READ WRITE CKSUM
SASPOOL ONLINE 0 0 0
mirror-0 ONLINE 0 0
gptid/1d9a8ab3-7e1f-11e3-a5c6-003048d5067c ONLINE 0 0
gptid/1e0188cd-7e1f-11e3-a5c6-003048d5067c ONLINE 0 0
mirror-1 ONLINE 0 0
gptid/1e68aba7-7e1f-11e3-a5c6-003048d5067c ONLINE 0 0
gptid/1ecedb74-7e1f-11e3-a5c6-003048d5067c ONLINE 0 0
mirror-2 ONLINE 0 0
gptid/1f3879fb-7e1f-11e3-a5c6-003048d5067c ONLINE 0 0
gptid/1fa97039-7e1f-11e3-a5c6-003048d5067c ONLINE 0 0
logs
mirror-3 ONLINE 0 0
gptid/1fdeef65-7e1f-11e3-a5c6-003048d5067c ONLINE 0 0
gptid/2006de8a-7e1f-11e3-a5c6-003048d5067c ONLINE 0 0

When reviewing dd I got the following results:
Results of dd if=/dev/zero of=/mnt/sata01/test1 bs=8k count=2000000
2000000+0 records in
2000000+0 records out
16384000000 bytes transferred in 37.685992 secs (434750397 bytes/sec)
When I ran Results of GSTAT while copying a 10 GB file - The percentage use on the 6 SAS drives inconsistently jumped from 0 to 5 to 26 to 50 back to 30 ish. Hit 80% a couple times, but would return to the
20's mostly. Often it went to 0. Was seeing about 25MB/s on the windows VM copying.
dT: 1.001s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 0 0 0 0.0 0 0 0.0 0.0| ada0
0 0 0 0 0.0 0 0 0.0 0.0| da2p1
0 0 0 0 0.0 0 0 0.0 0.0| ada1
0 105 0 0 0.0 103 10973 4.8 45.9| da0
0 101 0 0 0.0 99 10462 4.5 43.0| da1
0 110 0 0 0.0 108 11553 4.6 47.2| da2
0 110 0 0 0.0 108 11553 4.6 47.1| da3
0 115 0 0 0.0 113 12252 4.7 50.6| da4
0 115 0 0 0.0 113 12124 4.6 49.5| da5
0 0 0 0 0.0 0 0 0.0 0.0| cd0
0 0 0 0 0.0 0 0 0.0 0.0| da3p1.eli
0 110 0 0 0.0 108 11553 4.6 47.3| da2p2
0 110 0 0 0.0 108 11553 4.6 47.4| gptid/1d9a8ab3-7e
1f-11e3-a5c6-003048d5067c
0 0 0 0 0.0 0 0 0.0 0.0| da3p1
0 0 0 0 0.0 0 0 0.0 0.0| da4p1.eli
0 110 0 0 0.0 108 11553 4.6 47.1| da3p2
0 110 0 0 0.0 108 11553 4.6 47.2| gptid/1e0188cd-7e
1f-11e3-a5c6-003048d5067c
0 0 0 0 0.0 0 0 0.0 0.0| da4p1
0 0 0 0 0.0 0 0 0.0 0.0| da5p1.eli
0 115 0 0 0.0 113 12252 4.7 50.7
My two Zil drives (ada0 and ada1) are not being touched. Any idea why?

 

[jgreco]

Because you're not writing anything sync?

 

[Chris LaDuke]

jgreco,

Thanks alot for taking the time to respond. Ok, so having now read your explanation of SLOG at http://forums.freenas.org/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/ I realize since I am doing a filecopy over iscsi, that it is an async write and therefore not going to touch the SLOG unless I specify sync=always. But since its an async write, ZIL should not be impacting performance. I was trying to determine my poor performance, and thought maybe it was the ZIL not working properly. But since these are async writes, that shouldn't be it. I have included an iometer report showing I am getting about 30MB/sec read and 40MB/sec write and about 600 IOPS for the array. The iometer test was doing 64K 100% sequential reads and 100% sequential writes. It was done on a VM who's os is stored locally on the host. Is this inline with what your expectations would be? Or does this seem sluggish. My mpio is setup round-robin and the network interfaces have been darn near mirrors of one another over the four, so it seems mpio is working properly. I have read your many posts about iscsi and potential poor performance and inefficiency of zfs and iSCSI, just seems like 30MB/sec is outside the norm.... Here is also a copy of iozone, which makes it look like the local pool is pretty fast....
Results of iozone -s100M -r4K -I -t32 -T -i0 -i2

Throughput test with 32 threads
Each thread writes a 102400 Kbyte file in 4 Kbyte records

Children see throughput for 32 initial writers = 480384.03 KB/sec
Parent sees throughput for 32 initial writers = 175085.85 KB/sec
Min throughput per thread = 9528.23 KB/sec
Max throughput per thread = 25087.94 KB/sec
Avg throughput per thread = 15012.00 KB/sec
Min xfer = 39436.00 KB

Children see throughput for 32 rewriters = 597868.95 KB/sec
Parent sees throughput for 32 rewriters = 244237.71 KB/sec
Min throughput per thread = 14219.66 KB/sec
Max throughput per thread = 26843.38 KB/sec
Avg throughput per thread = 18683.40 KB/sec
Min xfer = 54528.00 KB

Children see throughput for 32 random readers = 1526313.29 KB/sec
Parent sees throughput for 32 random readers = 1493427.91 KB/sec
Min throughput per thread = 161.71 KB/sec
Max throughput per thread = 79624.64 KB/sec
Avg throughput per thread = 47697.29 KB/sec
Min xfer = 208.00 KB

Thanks!

Chris

 

[jgreco]

ok so I may have missed it since I'm on a cell phone, but as I understand it you have a 900GB pool made out of some striped mirrors ("good!"). how full is the pool?

 

[Chris LaDuke]

ok so I may have missed it since I'm on a cell phone, but as I understand it you have a 900GB pool made out of some striped mirrors ("good!"). how full is the pool?

Well its brand new, so as far as raw data, its pretty empty. I created a zvol that is 700G (I had read in another of your posts to keep 10+% open, and more is better). I had tried it with only a 500G zvol as well saw no improvement.

 

[jgreco]

For a new pool, it should be okay then. Over time you need a LOT of pool space free (like 50% or more!) if you have lots of random writes.

I'd say try ditching mpio as a first step. Don't try to run until you can walk! Eliminates several classes of nonobvious failures.

Next step is to get a VM running FreeBSD up on ESXi, then try iperf from the NAS to the VM (make sure to put the VM on the vSwitch you are using for storage). That OUGHT to blast at 100MB/sec +++, otherwise search and destroy your networking issue.

 

[Chris LaDuke]

Got it. Will do and report back.

Thanks again....

 

[Chris LaDuke]

I hate to be a total noob, but I am assuming I should have the FreeBSD install on the local datastore of the esxi server, and not on the datastore I created on the Freenas server? If this is incorrect, please let me know.

Thanks again!

 

[jgreco]

It doesn't matter. You're doing network testing. It might be convenient to do it on a FreeNAS based datastore, not for the current business, but for the next things we might want to do.

 

[Chris LaDuke]

Please see my iperf test below. I also turned off MPIO and decided to do another iometer test. IOPS went from 600 to 1748 and transfer rate went from 30 MB/sec to 109. Network transfer rate on the freenas interfaces was 60 mbit/sec on each interface with MPIO. Without MPIO its 800 mbits on the one. Obviously its a huge improvement. However, I do have 3 other Gbit interfaces sitting there dormant. Any suggestions? Thanks again. Chris
[ 6] local 192.168.10.20 port 10581 connected with 172.22.2.121 port 5001 [ ID] Interval Transfer Bandwidth [ 6] 0.0-10.0 sec 1.09 GBytes 936 Mbits/sec [root@NAS1 ~]# iperf -c 172.22.2.121 ------------------------------------------------------------ Client connecting to 172.22.2.121, TCP port 5001 TCP window size: 32.5 KByte (default) ------------------------------------------------------------ [ 6] local 192.168.10.20 port 49659 connected with 172.22.2.121 port 5001 [ ID] Interval Transfer Bandwidth [ 6] 0.0-10.0 sec 1.09 GBytes 936 Mbits/sec

 

[jgreco]

Okay, so, good. This is probably either a network-configured-wrong issue or MPIO-configured-wrong issue. Could possibly also be a networking issue on a different subnet...

Please describe in some detail your physical networking setup. I'm especially interested in what exactly you did to implement multiple network links.

 

[Chris LaDuke]

Ok, so in my assembling the information you requested I saw that one of my switch links had autonegotiated at 100Mb rather than 1000Mb. I fixed that problem, now I am seeing 2536 IOPS and 158MB/sec transfer. Does this seem inline with what I should expect, given my hardware and configuration (its a GREAT improvement)? I am doing an iometer test 100% sequential write with 64K blocks. Is there a performance testing methodology you prefer? If there is an existing article on it, I am happy to read that too! I am using this for an MSSQL server serving up pdf files (some up to a few MB in size). It doesn't need to be a speed demon, but would like to get the most out of what I've got. I mention it because that can probably effect the testing methodology.

Thanks!

 

[jgreco]

It is within the realm of acceptability, though perhaps a bit disappointingly on the low side.

Basically I like taking things a step at a time when debugging. At this point you could reasonably look at top and gstat to see if there were obvious limits being hit; if so, your journey is pretty close to complete, except for any tuning that might get you further. If not, you want to continue looking for network issues.

 

[Chris LaDuke]

OK. At this point I am seeing about 50% CPU utilization on the FreeNAS box while running iometer from the VM doing a 64K sequential write. I assume this test would push the system most effectively. gstat is showing 0.00 for about 4 seconds then it jumps to about 43% then back down to 0%. I am seeing about 340Mb/sec on each of the FreeNAS network interfaces.

My network consists of an HP procurve 2824. I am attaching a pdf of the port layout. I am also attaching a snapshot of the VM network.

 

[Chris LaDuke]

As I have moved on from the original posted topic I am going to close end this thread and open a new one, per the forum guidelines.

 

[Chris LaDuke]

I wasn't sure if you saw, I created a new thread here that continues this issue.
http://forums.freenas.org/threads/poor-iscsi-performance-when-enabling-sync-always.17990/

Thanks!!