Matthieu Barbie
Cadet
- Joined
- Sep 24, 2015
- Messages
- 9
Hello all,
I've a few question to well balanced my server I build for my work. I decided to use Freenas for multiple raisons, one is the user friendly interface (if I'm not at the works and have a little problem, a well trained guy can solve the problem without losing is mind :) ).
So at first I will explain you my configuration and my goal.
My goal was to build a server that can handle many simultanous request on manies files (50 mb per files) and somes access on big files (6-12 Gb a file). The goal is to deliver to artists a quick read access to their files and a correct access for writing file. The write process can be 1/3 lower than read file, read is a priority.
The server is a 36 HDDs bay based on Supermicro case, exactly this is the configuration :
SuperMicro Storage Server R6048-E1CR36N
Rackmount 6U
Chassis CSE-847BE1C-R1K28LPB
1200W Redundant PSU (Platinium 94%)
Mainboard X10DRi-T4+
2x Xeon E5-2600v3 series
64 GB DDR4-2133 Reg ECC
LSI3108 SAS3 HW RAID
36x Hot-swap SAS3 Bays (with LSI Expander) 6TB SAS3 (12Gb/s) 7.2k 128Mb (Seagate RE 24/7)
Quad 10GBase-T LAN w/ Intel X540
Supermicro AOC-S3108L SAS3 RAID Card w/ 2GB cache Supermicro SuperCap Module (BBU) Supermicro 2-port Int-to-Ext SAS3 expansion (to ext. expanders)
2x Samsung SM863 DataCenter SSD 480GB SATA3
Intel E10G42BFSR Network Adapter Dual 10GBase-SR ( PCIE)
In attachment, the lspci from the server for more details.
I made 3 hardware raid 6 group with the LSI card (all the hardware maintenance is auto) and the result is stripped in Freenas.
I use one of SSD for ZIL and the other for L2ARC.
Here is the config optimisation I read and put in the /boot/loader.conf :
# Basics
vm.kmem_size_max="60G"
vm.kmem_size="55G"
vfs.zfs.arc_min="20G"
vfs.zfs.arc_max="50G"
vfs.zfs.prefetch_disable="1"
vfs.zfs.txg.timeout="5"
kern.maxvnodes=250000
vfs.zfs.write_limit_override=1073741824
# L2ARC
# by default at the moment
As you see, don't use option at the moment for L2ARC. At the moment in rsync from a client and run from it (standard PC hdd) to the server using NFS (6 servers running on Freenas) I got this performance in bi-drectionnal rsync :
For 2 GB iso : 1,991,245,824 100% 335.74MB/s 0:00:05 (xfr#1, to-chk=0/1)
So my question are :
- Does my loader.conf is well setup for my usage ?
- What can I do to have more performances ?
- Does I really need ZIL on a SSD or is better to remove ZIL (turning it off) and give the SSD to L2ARC ?
- What others tips / fine tunning you think I can do to have more performance ?
At this time, I just use a single standard 1 GB port to connect server to network. I'm waiting a fiber cable to connect it at 10Gb/s, exactly two fiber cables. One for the network team/floor, one for deliver on other ip the datas to the renderfarm.
Thank you in advanced,
Best Regards,
Matt
I've a few question to well balanced my server I build for my work. I decided to use Freenas for multiple raisons, one is the user friendly interface (if I'm not at the works and have a little problem, a well trained guy can solve the problem without losing is mind :) ).
So at first I will explain you my configuration and my goal.
My goal was to build a server that can handle many simultanous request on manies files (50 mb per files) and somes access on big files (6-12 Gb a file). The goal is to deliver to artists a quick read access to their files and a correct access for writing file. The write process can be 1/3 lower than read file, read is a priority.
The server is a 36 HDDs bay based on Supermicro case, exactly this is the configuration :
SuperMicro Storage Server R6048-E1CR36N
Rackmount 6U
Chassis CSE-847BE1C-R1K28LPB
1200W Redundant PSU (Platinium 94%)
Mainboard X10DRi-T4+
2x Xeon E5-2600v3 series
64 GB DDR4-2133 Reg ECC
LSI3108 SAS3 HW RAID
36x Hot-swap SAS3 Bays (with LSI Expander) 6TB SAS3 (12Gb/s) 7.2k 128Mb (Seagate RE 24/7)
Quad 10GBase-T LAN w/ Intel X540
Supermicro AOC-S3108L SAS3 RAID Card w/ 2GB cache Supermicro SuperCap Module (BBU) Supermicro 2-port Int-to-Ext SAS3 expansion (to ext. expanders)
2x Samsung SM863 DataCenter SSD 480GB SATA3
Intel E10G42BFSR Network Adapter Dual 10GBase-SR ( PCIE)
In attachment, the lspci from the server for more details.
I made 3 hardware raid 6 group with the LSI card (all the hardware maintenance is auto) and the result is stripped in Freenas.
I use one of SSD for ZIL and the other for L2ARC.
Here is the config optimisation I read and put in the /boot/loader.conf :
# Basics
vm.kmem_size_max="60G"
vm.kmem_size="55G"
vfs.zfs.arc_min="20G"
vfs.zfs.arc_max="50G"
vfs.zfs.prefetch_disable="1"
vfs.zfs.txg.timeout="5"
kern.maxvnodes=250000
vfs.zfs.write_limit_override=1073741824
# L2ARC
# by default at the moment
As you see, don't use option at the moment for L2ARC. At the moment in rsync from a client and run from it (standard PC hdd) to the server using NFS (6 servers running on Freenas) I got this performance in bi-drectionnal rsync :
For 2 GB iso : 1,991,245,824 100% 335.74MB/s 0:00:05 (xfr#1, to-chk=0/1)
So my question are :
- Does my loader.conf is well setup for my usage ?
- What can I do to have more performances ?
- Does I really need ZIL on a SSD or is better to remove ZIL (turning it off) and give the SSD to L2ARC ?
- What others tips / fine tunning you think I can do to have more performance ?
At this time, I just use a single standard 1 GB port to connect server to network. I'm waiting a fiber cable to connect it at 10Gb/s, exactly two fiber cables. One for the network team/floor, one for deliver on other ip the datas to the renderfarm.
Thank you in advanced,
Best Regards,
Matt