RickH
Explorer
- Joined
- Oct 31, 2014
- Messages
- 61
I recently completed a project for a client that involved a new FreeNAS build, and I wanted to share it here.
My client had need for a portable NAS device with 10+ TB of capacity, encryption capability, and the ability to copy large quantities of image-based data very quickly. They had been using a Cypertex appliance (basically a rebranded QNAP device with encryption built into the firmware) that they had spent almost $8k on that just wasn't meeting performance expectations. The datasets they were moving were comprised of millions of small files (scanned images) that averaged less than 60kb each. We created a 'dummy' dataset for testing that was 10 TB in size and contained approximately 50 million files (moving this many small files over a network is pretty much a worst-case usage scenario). Using their existing approach with the Ciphertex appliance took over 60 hours to move the 10 TB dataset on/off the box.
The first thing we did to improve performance was to change transfer methods from CIFS to ISCSI - using the software ISCSI initiator built into Windows to connect to a 12TB NTFS formatted target - this brought the transfer time down to 38 hours.
Our next approach involved replacing the firmware on the Ciphertex/QNAP with FreeNAS (booting from an external USB drive). The Ciphertex/QNAP appliance included a dual port 10GbE Base-T card as well as a single port 10GbE SFP+ card (along with 4 1GbE adapters on the motherboard) - the 10GbE Base-T adapter turned out to be Intel-based and was recognized by FreeNAS with no issues; however, we did have to swap the SFP+ adapter to a single port Chelsio card. We then set up the same ISCSI configuration and ran the test again - this reduced our transfer time to 25 hours (the only hardware change at this point was the SFP+ NIC). - This is a huge testament to the changes that have been implemented within FreeNAS over the past few years that address ISCSI performance!
Once we knew that FreeNAS was a truly viable option for this usage case, we set out to build the best portable FreeNAS appliance we could. The build consisted of the components below and came in at just over $4k (roughly half of what the Ciphertex appliance had run)
With this new FreeNAS appliance (nicknamed 'The Beast') assembled and tested I set up the same ISCSI configuration and re-ran the test. Transfer time was now down to 21 Hours! and my client was thrilled! Testing with larger test files and ISCSI multipathing I saw transfer speeds as high as 1,170 MiB/sec! The only thing the Ciphertex/QNAP appliance has over the new FreeNAS box are the 10GbE Base-T ports - which in my experience, just aren't used all that much. Pretty much any source system with 10GbE capabilities is going to be SFP+ based.
Build Sheet:
Case - U-NAS NSC-800
Motherboard/Proc - Gigabyte MB10-DS3
RAM - Kingston ValueRAM 128GB DDR4
Boot Drive - San Disk Cruiser Fit 32GB
Storage Drives - WD Se 5TB
HBA - IBM M1015
Carry Handle - MNPCTech Silver Billet Grooved Handle
Pictures:
Final Thoughts:
I know this build isn't for everyone, but I was pretty happy with how it turned out! We exceeded all of the performance requirements my client had at a price that was significantly less than the commercially available products. In the end I was actually kind of sad to have to turn the box over to my client, and I'm currently trying to make up an excuses as to why I need to build another one for myself... If I do get to build another, one I think the only thing I would change are the hard drives - the WD Se Drives have skyrocketed in price since I put this together, and I belive that HGST drives would provide similar performance at a much better value.
Let me know your thoughts....
My client had need for a portable NAS device with 10+ TB of capacity, encryption capability, and the ability to copy large quantities of image-based data very quickly. They had been using a Cypertex appliance (basically a rebranded QNAP device with encryption built into the firmware) that they had spent almost $8k on that just wasn't meeting performance expectations. The datasets they were moving were comprised of millions of small files (scanned images) that averaged less than 60kb each. We created a 'dummy' dataset for testing that was 10 TB in size and contained approximately 50 million files (moving this many small files over a network is pretty much a worst-case usage scenario). Using their existing approach with the Ciphertex appliance took over 60 hours to move the 10 TB dataset on/off the box.
The first thing we did to improve performance was to change transfer methods from CIFS to ISCSI - using the software ISCSI initiator built into Windows to connect to a 12TB NTFS formatted target - this brought the transfer time down to 38 hours.
Our next approach involved replacing the firmware on the Ciphertex/QNAP with FreeNAS (booting from an external USB drive). The Ciphertex/QNAP appliance included a dual port 10GbE Base-T card as well as a single port 10GbE SFP+ card (along with 4 1GbE adapters on the motherboard) - the 10GbE Base-T adapter turned out to be Intel-based and was recognized by FreeNAS with no issues; however, we did have to swap the SFP+ adapter to a single port Chelsio card. We then set up the same ISCSI configuration and ran the test again - this reduced our transfer time to 25 hours (the only hardware change at this point was the SFP+ NIC). - This is a huge testament to the changes that have been implemented within FreeNAS over the past few years that address ISCSI performance!
Once we knew that FreeNAS was a truly viable option for this usage case, we set out to build the best portable FreeNAS appliance we could. The build consisted of the components below and came in at just over $4k (roughly half of what the Ciphertex appliance had run)
Case:
My client was very concerned with the image they were portraying when they walked into their customer's office and did not want an appliance that looked 'hacked together'. To address this we ended up going with a U-NAS NSC-800 case and added a billet aluminum handle from MNPCTech to make it easier to lift out of the Pelican case used for shipping.
Motherboard/Processor:
When searching for a Mini-ITX motherboard to use in the case I was originally looking for something with a high SATA-port count as I was planning on using the one available PCI-e port for a 10GbE adapter; however, in my search I found the Gigabyte MB10-DS3 which includes 2 on-board 10GbE SFP+ ports, 2 1GbE ports, and a dedicated management/IPMI port. This comes with a non-removable Intel Xeon D-1541 processor (8 cores / 16 threads) and supports DDR4 ECC Ram. We were then able to add a SAS/SATA HBA to the x16 PCI-e slot.
RAM:
I maxed out the motherboard with 128GB of DDR4 ECC RAM using a Kingston (4x32GB) Kit
HBA:
Every FreeNAS server I run in production uses a HBA based on the LSI 2008 chip - for this build I grabbed a LSI 9240-8i (IBM M1015) off of eBay and crossflashed the P20 IT firmware. (NOTE: I typically do NOT install the HBA BIOS when I go through the crossflash procedure - this reduces the boot time of my servers by 30 seconds or more)
Hard Drives:
For the main storage drives I went with 8x Western Digital Se 5TB drives. The write performance of these drives is higher than most any other 7.2k SATA drive of any capacity out there. Since I built the appliance, the price of these drives seems to have gone up, and I don't think the small performance gain is worth it - if I were building again I would probably go with HGST drives.
Cache:
I experimented with using Intel SSD DC S3700 drives as LOG/L2ARC connected directly to the available SATA 3.0 ports on the motherboard, but for the intended usage I didn't see any performance gains whatsoever and due to difficulties fitting them in the case I ultimately did not include them on the final build. The NSC-800 case includes a couple brackets for mounting 2.5" drives internally, but they overlapped with the placement of the PCI-e HBA - I could have overcome the issues with some different sized standoffs, but didn't feel that the trouble was worth it in this case as the 128 GB of RAM and corresponding primary ARC size of over 100 GB provided plenty of performance. If I were using this device for hosting VM's or something else that wasn't so much of a sequential write I would have probably taken the extra time to fit it all in.
Boot Drive:
I typically use SATA-DOM drives for boot in my FreeNAS servers (and the Gigabyte motherboard has a power connector for a SATA-DOM), but for this build I ended up choosing to use a USB drive (San Disk Cruiser Fit 32GB). This was done primarily to increase security - the zpool is encrypted so that data cannot be accessed if something were to happen during shipping, by removing the boot drive and shipping it separately (or just carrying it) it's virtually impossible for anyone to access the data on the device.
My client was very concerned with the image they were portraying when they walked into their customer's office and did not want an appliance that looked 'hacked together'. To address this we ended up going with a U-NAS NSC-800 case and added a billet aluminum handle from MNPCTech to make it easier to lift out of the Pelican case used for shipping.
Motherboard/Processor:
When searching for a Mini-ITX motherboard to use in the case I was originally looking for something with a high SATA-port count as I was planning on using the one available PCI-e port for a 10GbE adapter; however, in my search I found the Gigabyte MB10-DS3 which includes 2 on-board 10GbE SFP+ ports, 2 1GbE ports, and a dedicated management/IPMI port. This comes with a non-removable Intel Xeon D-1541 processor (8 cores / 16 threads) and supports DDR4 ECC Ram. We were then able to add a SAS/SATA HBA to the x16 PCI-e slot.
RAM:
I maxed out the motherboard with 128GB of DDR4 ECC RAM using a Kingston (4x32GB) Kit
HBA:
Every FreeNAS server I run in production uses a HBA based on the LSI 2008 chip - for this build I grabbed a LSI 9240-8i (IBM M1015) off of eBay and crossflashed the P20 IT firmware. (NOTE: I typically do NOT install the HBA BIOS when I go through the crossflash procedure - this reduces the boot time of my servers by 30 seconds or more)
Hard Drives:
For the main storage drives I went with 8x Western Digital Se 5TB drives. The write performance of these drives is higher than most any other 7.2k SATA drive of any capacity out there. Since I built the appliance, the price of these drives seems to have gone up, and I don't think the small performance gain is worth it - if I were building again I would probably go with HGST drives.
Cache:
I experimented with using Intel SSD DC S3700 drives as LOG/L2ARC connected directly to the available SATA 3.0 ports on the motherboard, but for the intended usage I didn't see any performance gains whatsoever and due to difficulties fitting them in the case I ultimately did not include them on the final build. The NSC-800 case includes a couple brackets for mounting 2.5" drives internally, but they overlapped with the placement of the PCI-e HBA - I could have overcome the issues with some different sized standoffs, but didn't feel that the trouble was worth it in this case as the 128 GB of RAM and corresponding primary ARC size of over 100 GB provided plenty of performance. If I were using this device for hosting VM's or something else that wasn't so much of a sequential write I would have probably taken the extra time to fit it all in.
Boot Drive:
I typically use SATA-DOM drives for boot in my FreeNAS servers (and the Gigabyte motherboard has a power connector for a SATA-DOM), but for this build I ended up choosing to use a USB drive (San Disk Cruiser Fit 32GB). This was done primarily to increase security - the zpool is encrypted so that data cannot be accessed if something were to happen during shipping, by removing the boot drive and shipping it separately (or just carrying it) it's virtually impossible for anyone to access the data on the device.
With this new FreeNAS appliance (nicknamed 'The Beast') assembled and tested I set up the same ISCSI configuration and re-ran the test. Transfer time was now down to 21 Hours! and my client was thrilled! Testing with larger test files and ISCSI multipathing I saw transfer speeds as high as 1,170 MiB/sec! The only thing the Ciphertex/QNAP appliance has over the new FreeNAS box are the 10GbE Base-T ports - which in my experience, just aren't used all that much. Pretty much any source system with 10GbE capabilities is going to be SFP+ based.
Build Sheet:
Case - U-NAS NSC-800
Motherboard/Proc - Gigabyte MB10-DS3
RAM - Kingston ValueRAM 128GB DDR4
Boot Drive - San Disk Cruiser Fit 32GB
Storage Drives - WD Se 5TB
HBA - IBM M1015
Carry Handle - MNPCTech Silver Billet Grooved Handle
Pictures:
Final Thoughts:
I know this build isn't for everyone, but I was pretty happy with how it turned out! We exceeded all of the performance requirements my client had at a price that was significantly less than the commercially available products. In the end I was actually kind of sad to have to turn the box over to my client, and I'm currently trying to make up an excuses as to why I need to build another one for myself... If I do get to build another, one I think the only thing I would change are the hard drives - the WD Se Drives have skyrocketed in price since I put this together, and I belive that HGST drives would provide similar performance at a much better value.
Let me know your thoughts....
Last edited:
