I have been doing some tests for using iSCSI without a bare metal hypervisor.
What I'm trying to do:
I'm trying to use my FreeNAS as a small "home SAN". My goal is to convert my desktops into "thin clients" which have a small and fast SSD for the OS and apps, but no more HDDs for data. Instead of HDDs the desktops rely entirely on iSCSI block storage as secondary drives.
Acceptance criteria:
Using 2 NICs I want to archive about 180 MB/s throughput for sequential reads and writes to match a fast desktop HDD.
Compared to physical HDDs I also expect fast random writes (curtsey of the asynchronous behavior of iSCSI and ZFS) and more or less slow random reads (curtsey of sluggish vdev seek times plus network latency).
The problem:
Well, my problem is entirely with the sequential read performance. While sequential write performance is very good and as expected, it seems that I'm stuck hitting a performance barrier for reads. I can hardly exceed 60 MB/s even with multiple concurrent connections.
What I tested:
I tested the new CTL iSCSI target in FreeNAS 9.3 as well as 9.2.1.8 against 3 different iSCSI initiators.
The test setup:
The server:
The client:
Test 1: Windows Server 2008 R2 SP1 with FreeNAS 9.2.1.8 and FreeNAS 9.3
First I wanted to test MPIO performance under Windows. Since Windows 7 does not support MPIO, I used Windows Server 2008 R2 for this test. I set up MPIO with 2 connections to a single target with round robin load balancing.
The resulting random read and random write performance was good.
The sequential write performance was excellent with about 200 MB/s and traffic spread evenly over both NICs.
But unfortunately the sequential read performance was just bad:
Traffic was load balanced over both NICs so neither of them had much to do. Throughput did never exceed 60 MB/s no matter if I used a synthetic benchmark or just copied a big video file off the iSCSI drive.
When I first encountered this phenomenon with 9.2.1.8 I assumed something was wrong with the still experimental CTL. So I decided to give the brand new 9.3 a spin, because I read that lots of improvements went into CTL with 9.3. Setting up a new server with 9.3 was a piece of cake, but unfortunately I could not overcome the 60 MB/s read barrier with it. While IOPS, random reads and random writes improved, sequential reads remained equally poor.
At that point I assumed that there must be something wrong with the Microsoft iSCSI Initiator. So I decided to give a recent Linux a try and kicked the Windows Server in the bin.
Test 2: Gnomebuntu 14.10 with FreeNAS 9.3
I setup open-scsi with MPIO and round robin load balancing according to this excellent tutorial.
The results were not much different to the Windows Server 2008 results, tho:
Sequential writes from FreeNAS 9.3 were fine as ever, but - even though slightly improved - sequential reads remained bad and far from my acceptance criteria threshold.
After the test on Linux it was rather clear that the problem was not the iSCSI initiator. The problem seems to be with FreeNAS and/or CTL.
Test 3: Windows 7 with FreeNAS 9.2.1.8
So I decided to conduct one last test. In order to clear the multi path handling in CTL I connected each of the 2 NICs to a single, independent target. I then created a dynamic stripe in Windows (software raid0) spanning over both targets.
I was a little bit surprised to see that this "multi path raid" did work ok, but nevertheless the read results were the same. Using this pseudo MPIO I hit the 60 MB/s read barrier just the same as with real MPIO or even a single connection target.
Questions:
/bb
What I'm trying to do:
I'm trying to use my FreeNAS as a small "home SAN". My goal is to convert my desktops into "thin clients" which have a small and fast SSD for the OS and apps, but no more HDDs for data. Instead of HDDs the desktops rely entirely on iSCSI block storage as secondary drives.
Acceptance criteria:
Using 2 NICs I want to archive about 180 MB/s throughput for sequential reads and writes to match a fast desktop HDD.
Compared to physical HDDs I also expect fast random writes (curtsey of the asynchronous behavior of iSCSI and ZFS) and more or less slow random reads (curtsey of sluggish vdev seek times plus network latency).
The problem:
Well, my problem is entirely with the sequential read performance. While sequential write performance is very good and as expected, it seems that I'm stuck hitting a performance barrier for reads. I can hardly exceed 60 MB/s even with multiple concurrent connections.
What I tested:
I tested the new CTL iSCSI target in FreeNAS 9.3 as well as 9.2.1.8 against 3 different iSCSI initiators.
The test setup:
The server:
- Hardware:
- Intel Xeon 1230 v3,
- Supermicro X10-something-something with 2 onboard Intel 1GbE NICs
- 16 GB RAM
- IBM M1015 HBA
- 3 mirror vdevs with WD REDs (writes: ~320 MB/s, reads: ~460 MB/s)
- Intel Xeon 1230 v3,
- Software:
- FreeNAS 9.2.1.8 and 9.3
- CTL iSCSI target in both cases
- ZVOL extents with 4kB block size
- LZ4 compression enabled
- Hardware:
- Intel Xeon 1231 v3
- 8 GB RAM
- 256 GB Samsung SSD 850 Pro
- 2 Intel 1GbE NICs (1 onboard, 1 PCIe)
- Software:
- Windows Server 2008 R2 SP1, Windows iSCSI Initiator w/ MPIO
- Gnomebuntu Linux 14.10, open-iscsi w/ multipath-tools for MPIO
- Windows 7 SP1, Windows iSCSI Initiator
First I wanted to test MPIO performance under Windows. Since Windows 7 does not support MPIO, I used Windows Server 2008 R2 for this test. I set up MPIO with 2 connections to a single target with round robin load balancing.
The resulting random read and random write performance was good.
The sequential write performance was excellent with about 200 MB/s and traffic spread evenly over both NICs.
But unfortunately the sequential read performance was just bad:
Traffic was load balanced over both NICs so neither of them had much to do. Throughput did never exceed 60 MB/s no matter if I used a synthetic benchmark or just copied a big video file off the iSCSI drive.
When I first encountered this phenomenon with 9.2.1.8 I assumed something was wrong with the still experimental CTL. So I decided to give the brand new 9.3 a spin, because I read that lots of improvements went into CTL with 9.3. Setting up a new server with 9.3 was a piece of cake, but unfortunately I could not overcome the 60 MB/s read barrier with it. While IOPS, random reads and random writes improved, sequential reads remained equally poor.
At that point I assumed that there must be something wrong with the Microsoft iSCSI Initiator. So I decided to give a recent Linux a try and kicked the Windows Server in the bin.
Test 2: Gnomebuntu 14.10 with FreeNAS 9.3
I setup open-scsi with MPIO and round robin load balancing according to this excellent tutorial.
Code:
root@Mirakulu:~# multipath -ll
cygnus-san-test (36589cfc000000fafb2155bc92e60c776) dm-0 FreeBSD,iSCSI Disk
size=60G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 6:0:0:0 sdb 8:16 active ready running
`- 7:0:0:0 sdc 8:32 active ready runningThe results were not much different to the Windows Server 2008 results, tho:
Code:
root@Mirakulu:/mnt# dd if=/dev/urandom of=testfile.dat bs=2M count=10000 10000+0 records in 10000+0 records out 20971520000 bytes (21 GB) copied, 1541,49 s, 13,6 MB/s root@Mirakulu:/mnt# cd cygnus-san-test/ root@Mirakulu:/mnt/cygnus-san-test# dd if=../testfile.dat of=testfile.dat bs=2M count=10000 10000+0 records in 10000+0 records out 20971520000 bytes (21 GB) copied, 109,907 s, 191 MB/s root@Mirakulu:/mnt/cygnus-san-test# dd if=testfile.dat of=/dev/null bs=2M count=10000 10000+0 records in 10000+0 records out 20971520000 bytes (21 GB) copied, 237,857 s, 88,2 MB/s
Sequential writes from FreeNAS 9.3 were fine as ever, but - even though slightly improved - sequential reads remained bad and far from my acceptance criteria threshold.
After the test on Linux it was rather clear that the problem was not the iSCSI initiator. The problem seems to be with FreeNAS and/or CTL.
Disappointed I put Windows 7 back on my desktop and the USB stick with FreeNAS 9.2.1.8 back into the server.myself said:
Test 3: Windows 7 with FreeNAS 9.2.1.8
So I decided to conduct one last test. In order to clear the multi path handling in CTL I connected each of the 2 NICs to a single, independent target. I then created a dynamic stripe in Windows (software raid0) spanning over both targets.
I was a little bit surprised to see that this "multi path raid" did work ok, but nevertheless the read results were the same. Using this pseudo MPIO I hit the 60 MB/s read barrier just the same as with real MPIO or even a single connection target.
Questions:
- What is determining the sequential read performance of CTL in theory? Thread contention, kernel memory, IO subsystem latencies...?
- Why is the CTL read throughput so low in comparison to other "file shares" accessing the pool? I don't believe it's the ZFS pool as CIFS and FTP work very well and seem only restricted by the 1GbE connection.
- Is there anything that can be done from the user side in terms of configuration? I'd gladly trade some IOPS for throughput.
- What is your experience with CTL? Can you get adequate sequential reads?
/bb
