Pool
Cadet
- Joined
- Oct 6, 2016
- Messages
- 6
Good evening, good morning and good afternoon wherever you may be :)
As you can see I'am new to FreeNAS and I am having performance issue with my home installation. Write speed is really slow.
Before reading this, you can drop down directly to the picture, it contains almost all what I am going to write after.
So here is my setup :
FreeNAS (last version) :
Dell R710 / 2xE5620 / 64GB RAM / Dell SAS 5E / Dell H200 (IT mode v20) / Qlogic 2462 (4GB) / USBkey 8GB (FreeNAS-9.10.1-U2 (f045a8b) OS)
Dell MD1000 in split mode with 15x1TB SAS 7.2K cabled trough 2 SAS 4x cables to the Dell SAS 5E HBA
1 SSD 260 GB cabled to the Dell H200 SAS HBA (Flashed to IT mode with firmware v20)
5x146 15K cabled to the Dell H200 SAS HBA
ESXi 1 :
Dell R610 / 2xX5670 / 128GB RAM / Qlogic 2462 (4GB) / USBkey 8GB (ESXi 6 OS)
ESXi 2 :
Dell R610 / 2xX5670 / 128GB RAM / Qlogic 2462 (4GB) / USBkey 8GB (ESXi 6 OS)
For my FreeNas I have followed this topic for the fiber channel target configuration : https://forums.freenas.org/index.php?threads/freenas-9-3-fc-fibre-channel-target-mode-das-san.27653/ (btw wonderful post !)
I have made three volumes RAID0-8x1TO-SAS and RAID0-7X1TO-SAS and SSD (This is the real volumes names like that you don't get loss further). Notice I am not using the 5x146 15K cabled to the Dell H200 SAS HBA because the performance where the same, poor... so I am using them for something else.
The problems start here so I will not detail the rest of the installation, but if you need any information don't hesitate to ask.
Ok lets do some tests !
I'll start with the RAID0-8x1TB-SAS write tests (Orange Disks):
[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.081489 secs (992836705 bytes/sec)
[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 5.914924 secs (1815309542 bytes/sec)
[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1 conv=sync
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.263302 secs (849948587 bytes/sec)
[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10 conv=sync
10+0 records in
10+0 records out
10737418240 bytes transferred in 7.284734 secs (1473961606 bytes/sec)
Now the RAID0-7x1TB-SAS write tests (Green Disks) :
[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.077653 secs (996370717 bytes/sec)
[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 5.886430 secs (1824096812 bytes/sec)
[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1 conv=sync
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.233587 secs (870422435 bytes/sec)
[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10 conv=sync
10+0 records in
10+0 records out
10737418240 bytes transferred in 7.317794 secs (1467302641 bytes/sec)
Now on the SSD :
[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.112368 secs (965275629 bytes/sec)
[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 5.883694 secs (1824944997 bytes/sec)
[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=1 conv=sync
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.262673 secs (850372114 bytes/sec)
[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=10 conv=sync
10+0 records in
10+0 records out
10737418240 bytes transferred in 7.294051 secs (1472078820 bytes/sec)
Ok so now you can understand why I am going to cry ! Look at those numbers ! Not really poor performance for this kind of hardware ?
So it's been something like 200 hours spent on this issue ! I have made a lot of tests and some of them where really helpful to understand from where the issue is coming.
Here is the most important of them :
Test 1 :
I have a friend who have almost the same setup, but he don't use a server for FreeNas, he use a gaming computer with core i7/32Gb RAM and he don't use an HBA. He use a PERC 6 E to connect his freenas to his MD1000 (which is full of 15x1TB SAS 7.2K exactly the same as mine, even serial numbers follow themself !).
Just this make me crazy ! No ECC ! Hardware RAID under ZFS !
All the prerequisite of FreeNAS are not respected and even so he have 2.5x my performance.
So I have put a PERC 6/E in my FreeNAS machine, made a RAID 0 of each disk, selected write trough, etc... And I ended up with the exact same performance ! OMG
Test 2 :
I went with my FreeNAS machine to my friend house and I have tested my H200 (We had to put some tape on to pci pin to make it work !) and my SAS 5E on his gaming computer. And OMG we had 2.5x the performance you are seeing upper ! So we started to think that the problem was the R710 which a bit old hardware, but I couldn't believe that ! (I7 + 32GB VS 2xE5620 + 64GB lol?)
Test 3 :
I installed Centos 7 on the R710, made raid with mdam, the same disposition as on FreeNAS. And WOW ! The best performance I ever saw on that machine since I am troubleshooting this. I was hitting 2.9Gb on the SAS5/E and 5.9Gb on the H200 with the SSD.
I installed Windows Server 2012 and :
I tried a copy from RAID0-8x1TB-SAS to RAID0-7x1TB-SAS -> 350 MB/s
I tried a copy from RAID0-7x1TB-SAS to RAID0-8x1TB-SAS -> 380 MB/s
I tried a copy from RAID0-8x1TB-SAS to SSD -> 650 MB/s
I tried a copy from RAID0-7x1TB-SAS to SSD -> 650 MB/s
Same as centos, FAST !
So the problem is not the H200 nor the SAS 5/E (Test 1 and 2) this left us CPU RAM and Motherboard and Test 3 proofs that they can give good performance.
This is making me pointing FreeBsd + R710 = PROBLEM !!!
Notice ! I wrote FreeBsd and NOT FreeNAS
But i can not figure out what it is, my freebsd skills equal 0+0 !
So I came here, hoping that someone have experienced this kind of issue before, and he will tell me change this 0 by a 1 and Abracadabra !!!! Lol ! I know, performance issue are very specific to each hardware which make it hard to debug.
If you have any question regarding this installation, or if you want me to make some other tests and put them here just ask :)
As you can see I'am new to FreeNAS and I am having performance issue with my home installation. Write speed is really slow.
Before reading this, you can drop down directly to the picture, it contains almost all what I am going to write after.
So here is my setup :
FreeNAS (last version) :
Dell R710 / 2xE5620 / 64GB RAM / Dell SAS 5E / Dell H200 (IT mode v20) / Qlogic 2462 (4GB) / USBkey 8GB (FreeNAS-9.10.1-U2 (f045a8b) OS)
Dell MD1000 in split mode with 15x1TB SAS 7.2K cabled trough 2 SAS 4x cables to the Dell SAS 5E HBA
1 SSD 260 GB cabled to the Dell H200 SAS HBA (Flashed to IT mode with firmware v20)
5x146 15K cabled to the Dell H200 SAS HBA
ESXi 1 :
Dell R610 / 2xX5670 / 128GB RAM / Qlogic 2462 (4GB) / USBkey 8GB (ESXi 6 OS)
ESXi 2 :
Dell R610 / 2xX5670 / 128GB RAM / Qlogic 2462 (4GB) / USBkey 8GB (ESXi 6 OS)
For my FreeNas I have followed this topic for the fiber channel target configuration : https://forums.freenas.org/index.php?threads/freenas-9-3-fc-fibre-channel-target-mode-das-san.27653/ (btw wonderful post !)
I have made three volumes RAID0-8x1TO-SAS and RAID0-7X1TO-SAS and SSD (This is the real volumes names like that you don't get loss further). Notice I am not using the 5x146 15K cabled to the Dell H200 SAS HBA because the performance where the same, poor... so I am using them for something else.
The problems start here so I will not detail the rest of the installation, but if you need any information don't hesitate to ask.
Ok lets do some tests !
I'll start with the RAID0-8x1TB-SAS write tests (Orange Disks):
[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.081489 secs (992836705 bytes/sec)
[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 5.914924 secs (1815309542 bytes/sec)
[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1 conv=sync
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.263302 secs (849948587 bytes/sec)
[root@freenas] /mnt/RAID0-8x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10 conv=sync
10+0 records in
10+0 records out
10737418240 bytes transferred in 7.284734 secs (1473961606 bytes/sec)
Now the RAID0-7x1TB-SAS write tests (Green Disks) :
[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.077653 secs (996370717 bytes/sec)
[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 5.886430 secs (1824096812 bytes/sec)
[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=1 conv=sync
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.233587 secs (870422435 bytes/sec)
[root@freenas] /mnt/RAID0-7x1TO-SAS# dd if=/dev/zero of=testfile bs=1G count=10 conv=sync
10+0 records in
10+0 records out
10737418240 bytes transferred in 7.317794 secs (1467302641 bytes/sec)
Now on the SSD :
[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.112368 secs (965275629 bytes/sec)
[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=10
10+0 records in
10+0 records out
10737418240 bytes transferred in 5.883694 secs (1824944997 bytes/sec)
[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=1 conv=sync
1+0 records in
1+0 records out
1073741824 bytes transferred in 1.262673 secs (850372114 bytes/sec)
[root@freenas] /mnt/SSD# dd if=/dev/zero of=testfile bs=1G count=10 conv=sync
10+0 records in
10+0 records out
10737418240 bytes transferred in 7.294051 secs (1472078820 bytes/sec)
Ok so now you can understand why I am going to cry ! Look at those numbers ! Not really poor performance for this kind of hardware ?
So it's been something like 200 hours spent on this issue ! I have made a lot of tests and some of them where really helpful to understand from where the issue is coming.
Here is the most important of them :
Test 1 :
I have a friend who have almost the same setup, but he don't use a server for FreeNas, he use a gaming computer with core i7/32Gb RAM and he don't use an HBA. He use a PERC 6 E to connect his freenas to his MD1000 (which is full of 15x1TB SAS 7.2K exactly the same as mine, even serial numbers follow themself !).
Just this make me crazy ! No ECC ! Hardware RAID under ZFS !
All the prerequisite of FreeNAS are not respected and even so he have 2.5x my performance.
So I have put a PERC 6/E in my FreeNAS machine, made a RAID 0 of each disk, selected write trough, etc... And I ended up with the exact same performance ! OMG
Test 2 :
I went with my FreeNAS machine to my friend house and I have tested my H200 (We had to put some tape on to pci pin to make it work !) and my SAS 5E on his gaming computer. And OMG we had 2.5x the performance you are seeing upper ! So we started to think that the problem was the R710 which a bit old hardware, but I couldn't believe that ! (I7 + 32GB VS 2xE5620 + 64GB lol?)
Test 3 :
I installed Centos 7 on the R710, made raid with mdam, the same disposition as on FreeNAS. And WOW ! The best performance I ever saw on that machine since I am troubleshooting this. I was hitting 2.9Gb on the SAS5/E and 5.9Gb on the H200 with the SSD.
I installed Windows Server 2012 and :
I tried a copy from RAID0-8x1TB-SAS to RAID0-7x1TB-SAS -> 350 MB/s
I tried a copy from RAID0-7x1TB-SAS to RAID0-8x1TB-SAS -> 380 MB/s
I tried a copy from RAID0-8x1TB-SAS to SSD -> 650 MB/s
I tried a copy from RAID0-7x1TB-SAS to SSD -> 650 MB/s
Same as centos, FAST !
So the problem is not the H200 nor the SAS 5/E (Test 1 and 2) this left us CPU RAM and Motherboard and Test 3 proofs that they can give good performance.
This is making me pointing FreeBsd + R710 = PROBLEM !!!
Notice ! I wrote FreeBsd and NOT FreeNAS
But i can not figure out what it is, my freebsd skills equal 0+0 !
So I came here, hoping that someone have experienced this kind of issue before, and he will tell me change this 0 by a 1 and Abracadabra !!!! Lol ! I know, performance issue are very specific to each hardware which make it hard to debug.
If you have any question regarding this installation, or if you want me to make some other tests and put them here just ask :)
Last edited: