interpretation of random 4k read tests

[monovitae]

Hi guys, I was hoping to get some insight on how to intrepret some tests I've been running.

Show : Background for those who care

After recently migrating from an old freenas box to a newer better server(see sig) everything is fine except that on XenServer when I am installing a new vm from virtualCD(served by freenas NFS) onto the new vm(virtual disk ISCSI freenas) the install process is slow and jerky etc.

This very well could be an issue with xenserver or something else(I'm leaning that way) but after extensive testing I'm not sure how to intrepet the specific test below, or where to look next.

All of my testing Iperf, DD, IOZone, copying files manually etc. all peg out my gig network at around 112MB/s.

The one exception is when I ran the citrix performance vm, it would peg out the network on both my 8xWD reds raidz2, and the mirrored 850pro SSDs, on every test except 4k read. Which i thought could maybe be the cause, but now I'm thinking that the problem is really somewhere from xenserver up, I'm feeling like the network and freenas are pretty solid. I might just throw in the towel and move over to vmware.

So I ran the following Iozones. These are all preformed from a new Ubuntu Test VM with two extra virtual drives mounted one for spinning disk and one for ssd, over MPIO ISCSI to the freeNAS box.

Am I to assume that since the Mirrored SSDs are so close to the raidz2 spinners that this is mostly hitting ARC. And if so shouldn't the results be higher, is 11086 the expected result for 4k random reads from ARC? If this is nominal great, I'll look further into the xenserver side of it. If its not normal, where should I look for the bottleneck, I don't see anything obvious like high CPU loading on either the VM, dom0, or freenas.

Code:

SSD

        Run began: Wed Dec  2 08:23:10 2015

        File size set to 2097152 kB
        Record Size 4 kB
        Setting no_unlink
        Command line used: iozone -s 2048m -r 4k -i 0 -w -f /mnt/localssd/test
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                   
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
         2097152       4   110908   128934        

        Run began: Wed Dec  2 08:26:50 2015

        File size set to 2097152 kB
        Record Size 4 kB
        Setting no_unlink
        Command line used: iozone -s 2048m -r 4k -i 2 -w -f /mnt/localssd/test
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                   
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
         2097152       4                                      11086   119903       

Spinners

        Run began: Wed Dec  2 08:31:59 2015

        File size set to 2097152 kB
        Record Size 4 kB
        Setting no_unlink
        Command line used: iozone -s 2048m -r 4k -i 0 -w -f /mnt/local/test
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                   
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
         2097152       4   129071   128993

        Run began: Wed Dec  2 08:33:23 2015

        File size set to 2097152 kB
        Record Size 4 kB
        Setting no_unlink
        Command line used: iozone -s 2048m -r 4k -i 2 -w -f /mnt/local/test
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                   
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
         2097152       4                                       9984   119395    

 

[jgreco]

Your filesize is only 2GB and your filer has 48GB RAM. You're not likely to get meaningful results unless you get up to a filesize of a few hundred GB.

It seems clear that there's something amiss with the reads.

Code:

# iozone -s 2048m -r 4k -i 2 -w -f /var/tmp/testfile
        Iozone: Performance Test of File I/O
                Version $Revision: 3.420 $
                Compiled for 32 bit mode.
                Build: freebsd


        Run began: Wed Dec  2 16:50:04 2015

        File size set to 2097152 KB
        Record Size 4 KB
        Setting no_unlink
        Command line used: iozone -s 2048m -r 4k -i 2 -w -f /var/tmp/testfile
        Output is in Kbytes/sec
        Time Resolution = 0.000002 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152       4                                    764664   48174

iozone test complete.

This is just a random FreeBSD VM running on ESXi attached to a FreeNAS iSCSI volume. The read speeds on data recently written is generally quite good.

 

[monovitae]

Thanks for the quick reply. I agree about the file size vs ram, I'm not trying to eek out every bit of performance from the system. In fact if it stays in ARC thats perfectly fine with me, it just shows that that 11086 is way to low especially if its hitting ARC. After seeing your results, I decided to run the same test locally on the freenas box.

- The SSDs Locally

For some reason it looks like my copy/paste formating is messed up. As above, the first test in a set is write/rewrite and thee second is random read and random write.

Code:

   File size set to 2097152 KB
   Record Size 4 KB
   Setting no_unlink
   Command line used: iozone -s 2048m -r 4k -i 0 -w -f /mnt/ultra/cifstest/iozonetest
   Output is in Kbytes/sec
   Time Resolution = 0.000001 seconds.
   Processor cache size set to 1024 Kbytes.
   Processor cache line size set to 32 bytes.
   File stride size set to 17 * record size.
  random  random  bkwd  record  stride  
  KB  reclen  write rewrite  read  reread  read  write  read  rewrite  read  fwrite frewrite  fread  freread
  2097152  4  524839  531244 

  Run began: Wed Dec  2 10:13:24 2015

   File size set to 2097152 KB
   Record Size 4 KB
   Setting no_unlink
   Command line used: iozone -s 2048m -r 4k -i 2 -w -f /mnt/ultra/cifstest/iozonetest
   Output is in Kbytes/sec
   Time Resolution = 0.000001 seconds.
   Processor cache size set to 1024 Kbytes.
   Processor cache line size set to 32 bytes.
   File stride size set to 17 * record size.
  random  random  bkwd  record  stride  
  KB  reclen  write rewrite  read  reread  read  write  read  rewrite  read  fwrite frewrite  fread  freread
  2097152  4  1034974  188804  

Running the same test for some 7200rmp spinners locally gives the same result as above, so definitely hitting ARC.


If I run the same setup as my OP except for change block size to 128k i get the following.

Code:

  File size set to 2097152 kB
  Record Size 128 kB
  Setting no_unlink
  Command line used: iozone -s 2048m -r 128k -i 0 -w -f /mnt/localssd/iozone128
  Output is in kBytes/sec
  Time Resolution = 0.000001 seconds.
  Processor cache size set to 1024 kBytes.
  Processor cache line size set to 32 bytes.
  File stride size set to 17 * record size.
  random  random  bkwd  record  stride  
  kB  reclen  write  rewrite  read  reread  read  write  read  rewrite  read  fwrite frewrite  fread  freread
  2097152  128  129083  128918

  File size set to 2097152 kB
  Record Size 128 kB
  Setting no_unlink
  Command line used: iozone -s 2048m -r 128k -i 2 -w -f /mnt/localssd/iozone128
  Output is in kBytes/sec
  Time Resolution = 0.000001 seconds.
  Processor cache size set to 1024 kBytes.
  Processor cache line size set to 32 bytes.
  File stride size set to 17 * record size.
  random  random  bkwd  record  stride  
  kB  reclen  write  rewrite  read  reread  read  write  read  rewrite  read  fwrite frewrite  fread  freread
  2097152  128  78658  128886

So I guess the next question is what would be the possible bottleneck that would allow for hitting 1GB line speed on all tests except for 4k random read?

 

[monovitae]

K so another update. Going with the isolate and destroy theory here. I decided to do the same test on my laptop to take xenserver out of the mix.

- Laptop over NFS to freenas SSD array.

Code:

  File size set to 2097152 KB
   Record Size 4 KB
   Setting no_unlink
   Command line used: iozone -s 2048m -r 4k -i 2 -w -f /mnt/nfstemp/iozonelap
   Output is in Kbytes/sec
   Time Resolution = 0.000001 seconds.
   Processor cache size set to 1024 Kbytes.
   Processor cache line size set to 32 bytes.
   File stride size set to 17 * record size.
  random  random  bkwd  record  stride   
  KB  reclen  write rewrite  read  reread  read  write  read  rewrite  read  fwrite frewrite  fread  freread
  2097152  4  14235  152187   

The physical topology of the environment is as follows:

For the laptop test(NFS)
FreeNASBox(Physical) ------(IGB1 1GB Nic) ------ TL-SG2424P(Switch) ----- VLAN into PFSENSE Box ----- VLAN Back out to TL-SG2424P(Switch) ---- Laptop

For the Test VM ISCI- Tweak on both freenas and xenservers MTU 9000
FreeNASBox(Physical) ------(IGB3 1GB On a Different physical nic card) -------- 8 port dumb switch only used for ISCSI ------- XenServer

So now that I've actually thought the topology out I'm starting to think it is on the freeNAS side again. The only common element between the laptop test and the VM test is the FreeNAS box, the two tests don't even share the same physical NIC.

 

[monovitae]

More updates. I still have the

Show : old server

ASROCK C2550D4I
Intel(R) Atom(TM) CPU C2550 @ 2.40GHz
16GB ECC DDR3
2x Intel i210 GB
Marvell SE9172 and Marvel SE92230
4x6TB WD Reds

online, its currently used as a replication target for the box listed in my signature. Just to further verify my thoughts I decided to run the above test from my laptop to the old server, with the replicated data set. It does a little bit better scoring 12610 on 4k random reads. Which had me thinking ok maybe the test itself is worthless(maybe it is), but then I remembered jgreco results from above. So there may be an issue there.

Out of curiosity I downloaded the lastest FreeBSD 10.2 release to see if my results had something to do with linux(the test vm was ubuntu, and the laptop is arch). Ran sam iozone command to old server over NFS. This is actually worse over nfs write=59553 rewrite=22863, and I'm still waiting for it to finish doing random read/write. But if I look at the network performance either in freeNAS gui or in xenserver, freeNAS has been transmitting at about 386mbit/s the whole time iozone ran like 42GB, but it never finished!

But then I thought well, the whole symptomatic cause that led me down this damn rabbit hole was installing VM's after i switch to the new system. So I go to the old system and share out my iso library via NFS just like on the new system and try and spin up a new ubuntu vm from each of the nfs iso librarys to local storage on the xen server. On the old server. Blazing fast no delay whatsoever. I could do an install from start to finish in a couple minutes. Go back to the new system and stalling everywhere takes about 20 minutes to finish the install. I also tried this with the freebsd install and interestingly during installation to disk it actually popped an error instead of just stalling like ubuntu/windows, could be another red herring but heres what it said.

Code:

 cam status: command timeout 

So theres definitely some sort of difference between the two systems, its not load, neither box has any sort of load disk, cpu or otherwise.

It seems like the issue has to be somewhere between the harddisks on the server and the servers network port. Not entirely sure how to isolate and test further.

Is there some sort of way that can diff the two boxes? Or further tests I can run on the new box?

 

[jgreco]

What happens if you run it directly on the FreeNAS box?

Why are you using jumbo frames? They're total crap.

 

[monovitae]

I'll read up on jumbo frames, and may change it after I'm more informed, but I think thats unrelated to the issue, as i see the same results over the NFS link on the other physical NIC which doesen't have jumbo frames.

I ran the tests locally on FreeNAS up in post #3

 

[monovitae]

Good write up on Jumbo Frames, thanks. I'll remove it, fortunately i only had it implemented in my isolated iSCSI network.

 

[monovitae]

To further isolate the issue I physically moved my NFS/CIFS/ETC. logical network off the integrated nic on the mobo, and over to the intel quad gb nic. No change in vm install performance. At this point i think it has to be something either on the IBM M1015IT cards, mobo, backplane, or software.

 

[monovitae]

I see that the above posts have become somewhat unwieldy. To summarize there are two seperate issues.

The first is the low random 4k read results. This is odd because it occurs on two completely different servers, occurs regardless of client OS, physical and logical network configurations, and sharing protocols. However, as jgreco inquired, the test seems to run fine locally. As such it seems that from iozone on bsd down through the mobo/cpu and into the hba disks etc things should be fine. Somewhere either in freeNAS or possibly in networking there is an issue. I'm leaning against the network side of it since this occurs on a non routed, physically isolated iscsi network and occurs regardless of configuration as noted above.

The other issue is the painfully slow installation of vm's from virtual cd drive in xenserver. This occurs with either cifs or nfs iso librarys. And occurs on the new server whether I'm using the onboard nic, or the intel add in card. However, if I go back to using the old server as the iso library even over the same networking equipment it runs perfectly fine. Both the new and old system are on the same versions of freenas and are configured identically as far as I can tell.

At this point I'm really at a loss as to what to test or investigate next, if anyone has some ideas please let me know.

 

[monovitae]

More tests.

I tried using universal usb installer on a windows box to install an iso to USB from both the old and new servers over CIFS. Roughly the same performance around 94 mbit/s.

So then I decided to load a LIVECD into ram on a vm from each freenas box over NFS. See results as attached.

 

[cyberjock]

I just read this whole thread. There's a lot of variables and a lot of possible situations that could be to blame. My pfsense box could never handle traffic for VMs going through it, so that could be a problem. The problem I'm having is that you've tried so many things and posted so many results, coupled with the very complex nature of this problem, that I really don't have any advice because I can't figure out what is up and what is down between all of this.

Sorry. :*(

 

[monovitae]

No worries thanks for taking a look. Ultimately, I'm not really worried about the random 4k reads from iozone. I feel like it may be a red herring, and more importantly it occurs on both the old and new system. What the real issue is to me at this point, is the reading of isos to a virtual cd under xenserver. This the real odd one for me because on my old box(hardware specs in post 5) it runs perfect , and in the new box(which i think adheres to reccomended hardware pretty well) its either incredibly slow or fails outright.

 

[monovitae]

I also don't yet have the knowledge to intrepet it, but i feel like the network chart in post 11# has some clue to the whole situation. I think its very odd that for the exact same iso being read into ram, that on the old box(system2) it just spits it out in one nice spike that hits 160MB/s and that on the new one it has those 4 wierd spikes with 0 data transfer inbetween them and only hits 40-50MB/s on the new box(system1).

 

[monovitae]

Ok so long story shorter, I took some time to step away from this over xmas break and then came back to it.

I've resolved half of the issues described in this thread. The real world issue of the slow ISO loads, SOLVED. I was correct about those wierd spikes in network traffic in the previous post, I knew something was wierd but couldn't figure out what. So I ran tcpdump on the xenserver during and ISO load, an then looked at it in WireShark. The whole stream was full of TCP resets, as well as traffic to an IP I didn't immediately recognize.

Apparently even though the GUI didn't show it, xenserver had a stale IP address for the original FreeNAS stuck in its config, as well as the new IP address. This was causing it to "fight" over the two destinations causing the issues with the ISOs. It didn't show up as massive VM failures, because the VMdisks are on a totaly seperate ISCSI network.

So yeah, me or luck or whatever, point is its resolved.

That leaves the odd iozone benchmarks. I've seen posted elsewhere around here that the ideal test is

Code:

 iozone -r 4k -s 4G -i 0 -i 2 -o -f ./iozonessd.file > ./iozoneresultssyncssd.txt 

Exclude the -o if you dont care about sync writes.

When I run this I get the following over ISCSI MPIO 1GBPS, for both of my pools even though one is 8disk raidz2 spinning rust, and the other is 4 disk mirrored 850pro SSDs, the ISCSI is taking the sync out because I'm still set at standard and i see mostly 0's in zilstat.

Writing and Reading for either pool: 16 MBPS, 2400 IOPS, 0.4ms latencey .

Code:

 IOzone Results
  Record Size 4 KB
   File size set to 4194304 KB
   SYNC Mode.
   Command line used: iozone -r 4k -s 4G -i 0 -i 2 -o -f ./iozonessd.file
   Output is in Kbytes/sec
   Time Resolution = 0.000001 seconds.
   Processor cache size set to 1024 Kbytes.
   Processor cache line size set to 32 bytes.
   File stride size set to 17 * record size.

KB 4194304
reclen 4
write 2434
rewrite 8026
random read 9521
random write 7628

I guess the question is now, is this test setup wrong? I still haven't seen anybody post their own results for a similar test. I don't want to be tilting windmills, because a test that is ultimately worthless. Then if anyone is getting better numbers, or can just tell that these are bad, by looking, the next question is whats, ccausing this. I would expect to see much better results from the SSD pool.

Maybe at this point I should just start a new thread that says run this IOZONE command, to seperate whats left of this problem from all the rest of the noise in this thread.