Few concerns about my iSCSI setup

Status
Not open for further replies.

asheeown

Cadet
Joined
Dec 30, 2013
Messages
4
For the past couple of weeks I've been playing around with a setup to get a small but decently reliable storage server up and going. I started with FreeNAS, and ran through some others just to compare performance, so far FreeNAS has given me the best.

Current NAS Setup:
Asus Mobo
Core i3 3.1Ghz
32GB DDR3 RAM
1x Onboard 1Gbe (Used for all management purposes)
Quad port Gbe Intel Server NIC, PCIe (Used for iSCSI)
ZFS Pool: 2x RaidZ each containing 3x 2TB Seagate Barracudas
(Making it a ZFS RAID 50)

The drives used to be connected via an Adaptec 3805 I had laying around. I found a post here that suggested to connect them directly to the mobo to avoid any double caching issues.

My test server is a Dell PowerEdge 1850 also with the same quad port intel gigabit nic card in it and Window Server 2012 using MPIO and the iSCSI initiator built in.

With the results I'm seeing benchmarking my storage I'm just having some trouble coming to terms with the performance I'm getting over iSCSI using all four NICs on each side. Obviously actual performance will never live up to theoretical but I'm not seeing anything close to that.

My first issue is consistency, I can be copying a file from the mounted drive at say 105 MB/s and it will dip to 30 MB/s for X amount of time and then bounce back up. Even writing does the same, bouncing up and down and I can watch the network traffic show the same results.

Another one is my write performance, since this topic has been covered time and time again I expect to take heat from this, but I have been through the forums for the past two weeks trying different solutions, versions, configurations, everything to change it or figure out why but I can't. With client write cache on I can write to the disk at a maximum of 70 MB/s give or take a few MB/s, this is straight from a RAMdisk or drive, doesn't matter (each NIC can get up to 300-400 mb/s). I usually try to keep my transfers above 10GB to get a nice idea of where my speeds stand. Without client write cache on, I can get as low as 10 MB/s and the network graphs show the same performance drop, never getting over 40-100 mb/s each.

I can live with about 100 MB/s read and 70 MB/s write if it were consistent. But it's not consistent and I would rather keep the client side write cache off in my case. I will post my tests below to show you the speeds I'm getting between the boxes and on the NAS itself. I will obviously continue to look around the forums for more answers but I've pretty much abused the search so far, I'm not sure how much more I will get from it.

DD Tests directly on NAS:
Code:
[root@localhost] /mnt/Pool# dd if=/dev/zero of=ddfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 26.479950 secs (395988664 bytes/sec)
[root@localhost] /mnt/Pool# dd if=ddfile of=/dev/zero bs=1M
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 15.434337 secs (679378717 bytes/sec)


iPerf Tests between NAS and test box:
Code:
[root@localhost] ~# iperf -w 65535 -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[ ID] Interval      Transfer    Bandwidth
[  4] local 10.0.1.10 port 5001 connected with 10.0.1.100 port 50017
[  4]  0.0-10.0 sec  858 MBytes  718 Mbits/sec
[  5] local 10.0.2.10 port 5001 connected with 10.0.2.100 port 50039
[  5]  0.0-10.0 sec  822 MBytes  688 Mbits/sec
[  4] local 10.0.3.10 port 5001 connected with 10.0.3.100 port 50067
[  4]  0.0-10.0 sec  780 MBytes  653 Mbits/sec
[  5] local 10.0.4.10 port 5001 connected with 10.0.4.100 port 50091
[  5]  0.0-10.0 sec  868 MBytes  726 Mbits/sec
 
 
C:\Users\Administrator\Downloads\iperf-2.0.5-cygwin>iperf -w 65535 -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[ ID] Interval      Transfer    Bandwidth
[  4] local 10.0.1.100 port 5001 connected with 10.0.1.10 port 44155
[  4]  0.0-10.0 sec  1.08 GBytes  932 Mbits/sec
[  4] local 10.0.2.100 port 5001 connected with 10.0.2.10 port 15695
[  4]  0.0-10.0 sec  1.09 GBytes  936 Mbits/sec
[  4] local 10.0.3.100 port 5001 connected with 10.0.3.10 port 58551
[  4]  0.0-10.0 sec  1.09 GBytes  931 Mbits/sec
[  4] local 10.0.4.100 port 5001 connected with 10.0.4.10 port 36130
[  4]  0.0-10.0 sec  1.09 GBytes  931 Mbits/sec


Any suggestions or insight would be greatly appreciated. Thank you.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
few things:

1) see bug 1531 (not really a bug?)

2) raidz is kind of bad with iscsi, look for discussions of volblocksize (mirrors are fine)

3) as a cow filesystem, iscsi kills zfs with fragmentation. you really need lots of free space on a pool, like over 50% free (yes, that is correct) to allow the system a reasonable pool of free blocks. if you think that's bad, in pathological fragmentation cases you need to maintain more like 90% free space... i linked a study someone did on this a few months ago.

just some things to help you on your quest.
 
J

jkh

Guest
What version of FreeNAS? I ask because we fixed some fairly significant iSCSI latency / variable performance problems in 9.2.0 (with symptoms very similar - transfer rates would just blast along and then suddenly hit a wall for several seconds before resuming proper behavior).
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I keep hearing about these issues and it would be very nice to know what the actual fixes/changes/etc were. Is there a summary somewhere?
 
J

jkh

Guest
I keep hearing about these issues and it would be very nice to know what the actual fixes/changes/etc were. Is there a summary somewhere?

Via the diffs / commit messages in the FreeNAS and trueos github repositories, you can easily find this information. No other summary list exists or is planned since anyone geeky enough to care is also expected to be geeky enough to know git... :)

We're also going to be reviving the freenas-commits mailing list for those who wish to follow the commits as they go by. Just a last bit of post-sourceforge transition work to do there.
 

asheeown

Cadet
Joined
Dec 30, 2013
Messages
4
few things:

1) see bug 1531 (not really a bug?)

2) raidz is kind of bad with iscsi, look for discussions of volblocksize (mirrors are fine)

3) as a cow filesystem, iscsi kills zfs with fragmentation. you really need lots of free space on a pool, like over 50% free (yes, that is correct) to allow the system a reasonable pool of free blocks. if you think that's bad, in pathological fragmentation cases you need to maintain more like 90% free space... i linked a study someone did on this a few months ago.

just some things to help you on your quest.

This is one of the things I have yet to actually try. Originally I had the Adaptec 3805 running a raid 50, completely initialized but on a Windows based platform. I received similar results but discarded it as overhead issues with Windows. Once I moved to FreeNAS and other BSD based projects I immediately destroyed the raid and started in with ZFS. I have seen other posts around about sticking iSCSI on top of ZFS, just haven't devoted the time to rebuilding the array on the raid card. This will definitely be my next step though, thank you for this information.


What version of FreeNAS? I ask because we fixed some fairly significant iSCSI latency / variable performance problems in 9.2.0 (with symptoms very similar - transfer rates would just blast along and then suddenly hit a wall for several seconds before resuming proper behavior).

I'm actually using 9.1.1 but the issues you say are fixed in 9.2.0 are definitely what I'm experiencing so I will continue further testing on the upgraded platform. I honestly wasn't aware of the new version out, I didn't even think to look since I started this not even three weeks ago. Good to see active updates coming through, although I didn't think that was a problem with this community.

Thanks again guys, I will be posting updates as I do more testing.
 

asheeown

Cadet
Joined
Dec 30, 2013
Messages
4
Alright so I took put both of your suggestions into play. I see better and more stable performance when reading from the iSCSI disk but writing hasn't changed a bit and in some cases is worse.

First I upgraded to 9.2.0 to see how it would play with ZFS and iSCSI together. Saw little to no difference, almost as if I did nothing. Reading was slightly more stable but still dropped from around 110 MB/s to around 40-50 MB/s at any given time during a transfer.

Next I destroyed the ZFS Pool and remade the RAID 50 on my Adaptec 3805. Once I got it optimal I did the basic benchmarks and got in the realm of where I was before, upwards of 350-400 MB/s write and 500-550 MB/s read. Still getting the same iPerf results as nothing has changed there. Reading I'm able to cap at about 115 MB/s (or about 240 Mbit/s on each network adapter) and it will be consistent throughout the transfer, maybe a little slow at first but always picks up. That I'm happy about and can definitely live with, however my write speed has gotten worse in most cases. Most of the time the transfer will start off at 3-8 MB/s and slowly ramp up to 70 MB/s (barely touching the speeds of each NIC). Oh and this is with client write caching turned on, I've tried it with it off and I get even worse results. I've also setup another client machine just to make sure it wasn't the machine I was using, no difference.

I did, however find out that I could maintain around 100 MB/s read from two different iSCSI disks on two different clients. It's nice to be able to know it can sustain that type of concurrency but why am I not receiving 200 MB/s on one client without another transfer going?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
If you created a RAID50 you deserve everything that's coming. It's blatantly and bluntly discussed in the manual that hardware RAID is a recipe for disaster. If you want to lose your data, keep using hardware RAID. It'll go great until one day you'll find your pool not functional.
 

asheeown

Cadet
Joined
Dec 30, 2013
Messages
4
If you created a RAID50 you deserve everything that's coming. It's blatantly and bluntly discussed in the manual that hardware RAID is a recipe for disaster. If you want to lose your data, keep using hardware RAID. It'll go great until one day you'll find your pool not functional.

Not exactly sure where you came into this but nothing was said about this being permanent. I inquired about reasons why I was getting poor performance, I got useful feedback which I used for testing and posted my results. I'm well aware of the recommended setup for FreeNAS, but thank you for your input.
 
Status
Not open for further replies.
Top