Performance issue FreeNAS/Dell R710

[Pool]

Good evening, good morning and good afternoon wherever you may be :)

As you can see I'am new to FreeNAS and I am having performance issue with my home installation. Write speed is really slow.

Before reading this, you can drop down directly to the picture, it contains almost all what I am going to write after.

So here is my setup :

FreeNAS (last version) :
Dell R710 / 2xE5620 / 64GB RAM / Dell SAS 5E / Dell H200 (IT mode v20) / Qlogic 2462 (4GB) / USBkey 8GB (FreeNAS-9.10.1-U2 (f045a8b) OS)
Dell MD1000 in split mode with 15x1TB SAS 7.2K cabled trough 2 SAS 4x cables to the Dell SAS 5E HBA
1 SSD 260 GB cabled to the Dell H200 SAS HBA (Flashed to IT mode with firmware v20)
5x146 15K cabled to the Dell H200 SAS HBA

ESXi 1 :
Dell R610 / 2xX5670 / 128GB RAM / Qlogic 2462 (4GB) / USBkey 8GB (ESXi 6 OS)


ESXi 2 :
Dell R610 / 2xX5670 / 128GB RAM / Qlogic 2462 (4GB) / USBkey 8GB (ESXi 6 OS)

For my FreeNas I have followed this topic for the fiber channel target configuration : https://forums.freenas.org/index.php?threads/freenas-9-3-fc-fibre-channel-target-mode-das-san.27653/ (btw wonderful post !)

I have made three volumes RAID0-8x1TO-SAS and RAID0-7X1TO-SAS and SSD (This is the real volumes names like that you don't get loss further). Notice I am not using the 5x146 15K cabled to the Dell H200 SAS HBA because the performance where the same, poor... so I am using them for something else.

The problems start here so I will not detail the rest of the installation, but if you need any information don't hesitate to ask.

Ok lets do some tests !

I'll start with the RAID0-8x1TB-SAS write tests (Orange Disks):

[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.081489 secs (992836705 bytes/sec)

[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 5.914924 secs (1815309542 bytes/sec)

[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1 conv=sync
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.263302 secs (849948587 bytes/sec)

[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10 conv=sync
10+0 records in
10+0 records out
10737418240 bytes transferred in 7.284734 secs (1473961606 bytes/sec)

Now the RAID0-7x1TB-SAS write tests (Green Disks) :

[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.077653 secs (996370717 bytes/sec)

[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 5.886430 secs (1824096812 bytes/sec)

[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1 conv=sync
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.233587 secs (870422435 bytes/sec)

[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10 conv=sync
10+0 records in
10+0 records out
10737418240 bytes transferred in 7.317794 secs (1467302641 bytes/sec)

Now on the SSD :

[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.112368 secs (965275629 bytes/sec)

[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 5.883694 secs (1824944997 bytes/sec)

[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=1 conv=sync
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.262673 secs (850372114 bytes/sec)

[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=10 conv=sync
10+0 records in
10+0 records out
10737418240 bytes transferred in 7.294051 secs (1472078820 bytes/sec)


Ok so now you can understand why I am going to cry ! Look at those numbers ! Not really poor performance for this kind of hardware ?

So it's been something like 200 hours spent on this issue ! I have made a lot of tests and some of them where really helpful to understand from where the issue is coming.
Here is the most important of them :

Test 1 :
I have a friend who have almost the same setup, but he don't use a server for FreeNas, he use a gaming computer with core i7/32Gb RAM and he don't use an HBA. He use a PERC 6 E to connect his freenas to his MD1000 (which is full of 15x1TB SAS 7.2K exactly the same as mine, even serial numbers follow themself !).
Just this make me crazy ! No ECC ! Hardware RAID under ZFS !
All the prerequisite of FreeNAS are not respected and even so he have 2.5x my performance.
So I have put a PERC 6/E in my FreeNAS machine, made a RAID 0 of each disk, selected write trough, etc... And I ended up with the exact same performance ! OMG

Test 2 :
I went with my FreeNAS machine to my friend house and I have tested my H200 (We had to put some tape on to pci pin to make it work !) and my SAS 5E on his gaming computer. And OMG we had 2.5x the performance you are seeing upper ! So we started to think that the problem was the R710 which a bit old hardware, but I couldn't believe that ! (I7 + 32GB VS 2xE5620 + 64GB lol?)

Test 3 :
I installed Centos 7 on the R710, made raid with mdam, the same disposition as on FreeNAS. And WOW ! The best performance I ever saw on that machine since I am troubleshooting this. I was hitting 2.9Gb on the SAS5/E and 5.9Gb on the H200 with the SSD.
I installed Windows Server 2012 and :
I tried a copy from RAID0-8x1TB-SAS to RAID0-7x1TB-SAS -> 350 MB/s
I tried a copy from RAID0-7x1TB-SAS to RAID0-8x1TB-SAS -> 380 MB/s
I tried a copy from RAID0-8x1TB-SAS to SSD -> 650 MB/s
I tried a copy from RAID0-7x1TB-SAS to SSD -> 650 MB/s

Same as centos, FAST !

So the problem is not the H200 nor the SAS 5/E (Test 1 and 2) this left us CPU RAM and Motherboard and Test 3 proofs that they can give good performance.

This is making me pointing FreeBsd + R710 = PROBLEM !!!
Notice ! I wrote FreeBsd and NOT FreeNAS

But i can not figure out what it is, my freebsd skills equal 0+0 !

So I came here, hoping that someone have experienced this kind of issue before, and he will tell me change this 0 by a 1 and Abracadabra !!!! Lol ! I know, performance issue are very specific to each hardware which make it hard to debug.

If you have any question regarding this installation, or if you want me to make some other tests and put them here just ask :)

 

[Ericloewe]

I'm kinda grasping at straws here, but try getting rid of the fiber channel card. In fact, try this without anything except for the H200.

 

[Pool]

I'm kinda grasping at straws here, but try getting rid of the fiber channel card. In fact, try this without anything except for the H200.

Already done. Same bad result

 

[Ericloewe]

Some sort of interrupt storm, perhaps? I really don't know what could be the problem. Maybe some of the guys with experience in Dell servers might have a clue, but this one has me baffled.

 

[Pool]

Some sort of interrupt storm, perhaps? I really don't know what could be the problem. Maybe some of the guys with experience in Dell servers might have a clue, but this one has me baffled.

Interrupt storm ? Between what and what ?
Maybe you are speaking about the Qlogic HBA ?
Notice that all the result before are tests made directly from the FreeNAS machine, so the fiber is not in use.
Here is the results with the same test on a Centos 7 Virtual Machine hosted on RAID0-7x1TO-SAS :

[root@reverse-proxy tmp]# dd if=/dev/zero of=/tmp/file bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 7.45637 s, 144 MB/s

[root@reverse-proxy tmp]# dd if=/dev/zero of=/tmp/file bs=1G count=5
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB) copied, 37.2095 s, 144 MB/s

[root@reverse-proxy tmp]# dd if=/dev/zero of=/tmp/file bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 6.97911 s, 154 MB/s

[root@reverse-proxy tmp]# dd if=/dev/zero of=/tmp/file bs=1G count=5 oflag=direct
5+0 records in
5+0 records out
5368709120 bytes (5.4 GB) copied, 35.6937 s, 150 MB/s

 

[Ericloewe]

Interrupt storm ? Between what and what ?

Something on the motherboard with a dubious driver. BMC, superIO, who knows...

 

[depasseg]

I think you are testing the CPU's ability to compress data. Your friend's core i7 is faster than your R710. You are getting between 7-14 Gigabits per second write throughput on 8 7200rpm drives. Max sequential throughput for a 7200rpm drive is ~190MB/s, multiplied by 8 is ~9Gbits/s.

 

[Pool]

I think you are testing the CPU's ability to compress data. Your friend's core i7 is faster than your R710. You are getting between 7-14 Gigabits per second write throughput on 8 7200rpm drives. Max sequential throughput for a 7200rpm drive is ~190MB/s, multiplied by 8 is ~9Gbits/s.

Yes ! you are right about the math !
So I should be able to write at 9 Gbits aprox, why I can't do it on FreeNAS ?
And there is no compression, my CPU have never be used more than 10%.

 

[Ericloewe]

I think you are testing the CPU's ability to compress data.

From /dev/zero? It's a Nehalem at 2.4GHz, I don't see that bottlenecking a compression of an endless stream of zeros.

Though now that you mention it, the values for all types of drives are very close... Hey wait a second, these are Bytes I'm looking at, not bits!

8Gb/s of zeros does NOT sound like a performance problem.

 

[depasseg]

So I should be able to write at 9 Gbits aprox, why I can't do it on FreeNAS?

You are. 190 is a MAX, and don't mix bits and bytes.
Just converting the 1st number gets 8 Gigabits/sec
https://www.google.com/search?q=992...ec&aqs=chrome..69i57&sourceid=chrome&ie=UTF-8

 

[Ericloewe]

Just converting the 1st number gets 8 Gigabits/sec

Yup, right where it should be.

On that note, I'd better go get some sleep before I accidentally tell someone that their J1900 Atom is a good choice for FreeNAS.

 

[Pool]

You are. 190 is a MAX, and don't mix bits and bytes.
Just converting the 1st number gets 8 Gigabits/sec
https://www.google.com/search?q=992836705+bytes/sec&oq=992836705+bytes/sec&aqs=chrome..69i57&sourceid=chrome&ie=UTF-8

ooooook .... I got it ! But this is not explaining to me why those my friend have better performance. you said CPU can you tell me more about that ?

 

[depasseg]

That dd command is not a good measure of comparative performance, because it's not actually testing the transfer speed. It's testing how fast you can write a highly compressible stream of zeros. Here are the results from my system. The pool is an 11 disk RAID-Z2 with an SSD SLOG.

Just a hint the first line is 22 Gigabits/second which isn't possible on 9 data drives.

Code:

[root@freenas1] /mnt/tank/test99# dd if=/dev/zero of=testfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 3.766939 secs (2850435978 bytes/sec)
[root@freenas1] /mnt/tank/test99# dd if=/dev/zero of=testfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 3.614486 secs (2970662579 bytes/sec)
[root@freenas1] /mnt/tank/test99# dd if=/dev/zero of=testfile bs=1G count=1 conv=sync
1+0 records in
1+0 records out
1073741824 bytes transferred in 0.743944 secs (1443310149 bytes/sec)
[root@freenas1] /mnt/tank/test99# dd if=/dev/zero of=testfile bs=1G count=10 conv=sync
10+0 records in
10+0 records out
10737418240 bytes transferred in 4.594007 secs (2337266400 bytes/sec)
[root@freenas1] /mnt/tank/test99# 

 

[depasseg]

Also, factors like available RAM come into play, since a file can be captured in RAM and marked as complete while the writing to disk happens in the background. This is avoided by using the dataset force sync option.

And since datasets by default also compresses data, that means that you aren't really writing 10G worth of data to disk. This is where I was thinking that one CPU could compress faster than another, with the result being a difference in apparent write speed.

 

[Pool]

Also, factors like available RAM come into play, since a file can be captured in RAM and marked as complete while the writing to disk happens in the background. This is avoided by using the dataset force sync option.

And since datasets by default also compresses data, that means that you aren't really writing 10G worth of data to disk. This is where I was thinking that one CPU could compress faster than another, with the result being a difference in apparent write speed.

Thank you very much depasseg, this was very educational for me. Btw I AM AN IDIOT !