Chris LaDuke
Dabbler
- Joined
- Jan 17, 2014
- Messages
- 18
Hey all. I am a relative n00b when it comes to zfs. I have a Supermicro 6025b-3 server with 12 GB RAM, 2 dual intel Gb nics, 6 300 GB 15k SAS drives, and two Intel DC S3700's. I am running it as a datastore for ESXi 5.1. I am connecting via iSCSI using mpio. I have the 6 SAS drives in 3 mirrors and I added the two Intel's as an SLOG. Each of the network interfaces are on their own vlan.
I am performing my test within a VM on an esxi host that also has two dual intel gb nics, each port on a corresponding VLAN. I am attaching a network diagram. My test consists of iometer 64K bs sequential write.
My iops are consistently approximately 2600 and my transfer rate is 168MB/sec. When I look at the individual network interfaces each is getting about 350 mbit/sec. What I have tested so far:
Disabled MPIO and tested individual network interfaces - I see approx 950 Mbit/sec on each of the individual interfaces. They each push about 116MB/sec and about 1870 IOPS each. There were no standouts of poor performance.
I then created a new pool of the two SSD drives in a stripe (gstat indicated the SAS drives were only 40% utilized at the peak, so I didnt think it was spool but wanted to check it off the list). I wanted to see if maybe I was hitting a drive bottleneck. Saw slightly higher IOPS, 2750, but the same transfer rate, 169 MB/sec.
I then did the test from a different esxi host, thinking maybe it was host specific. Same results. It should be said the hosts are identical IBM X3650's with dual E5345's, so if its a problem specific to that host platform, that doesn't really rule anything out. I also increased the vcpus on one of the vm's to four, to see if it wasn't a CPU issue on the esxi side (though Windows only reported 20% CPU utilization). While the test is running I see about 50% total CPU utilization on the FreeNAS server.
I have also attached iozone results, a snapshot of my pool and a snapshot of gstat when it peaks.
Somewhere I am hitting this 116MB/sec bottleneck, and I am stumped how to better isolate the problem. Or maybe, ask Jack Nicholson asked "What if this as good as it gets?"
Any advice is appreciated
Chris
I am performing my test within a VM on an esxi host that also has two dual intel gb nics, each port on a corresponding VLAN. I am attaching a network diagram. My test consists of iometer 64K bs sequential write.
My iops are consistently approximately 2600 and my transfer rate is 168MB/sec. When I look at the individual network interfaces each is getting about 350 mbit/sec. What I have tested so far:
Disabled MPIO and tested individual network interfaces - I see approx 950 Mbit/sec on each of the individual interfaces. They each push about 116MB/sec and about 1870 IOPS each. There were no standouts of poor performance.
I then created a new pool of the two SSD drives in a stripe (gstat indicated the SAS drives were only 40% utilized at the peak, so I didnt think it was spool but wanted to check it off the list). I wanted to see if maybe I was hitting a drive bottleneck. Saw slightly higher IOPS, 2750, but the same transfer rate, 169 MB/sec.
I then did the test from a different esxi host, thinking maybe it was host specific. Same results. It should be said the hosts are identical IBM X3650's with dual E5345's, so if its a problem specific to that host platform, that doesn't really rule anything out. I also increased the vcpus on one of the vm's to four, to see if it wasn't a CPU issue on the esxi side (though Windows only reported 20% CPU utilization). While the test is running I see about 50% total CPU utilization on the FreeNAS server.
I have also attached iozone results, a snapshot of my pool and a snapshot of gstat when it peaks.
Somewhere I am hitting this 116MB/sec bottleneck, and I am stumped how to better isolate the problem. Or maybe, ask Jack Nicholson asked "What if this as good as it gets?"
Any advice is appreciated
Chris