Hello all,
Been lurking for about 6 months, and am ready to pull the trigger on a build. I'm wondering if I can't try for a best of both worlds build, using hdds for 24 drives and ssds for the remaining 12? Knock out the bulk of my needed storage with the spinners, and get some high performance vm storage with the ssd's. I assume there's a reason I haven't come across more build like this, but that's why I'm asking.
Background:
This will be my first experience with FreeNAS. I have a good bit of build experience, but mostly workstations and pvr builds, and have done very little but maintenance of existing builds for the last ~5 years. My last full build was a x58 w3570 gis workstation for postGRE/GIS database crunching in 2012. I also have 2 Thecus 7700s that are getting up there in age (and just about at capacity), with 7x3TB drives in each in Raid6, as well as a random assortment of maybe 2 dozen 1-4TB disks, a combination of oldscratch drives, pvrs and cad workstations, all of which I'd like to consolidate. About a third of my data is GIS related shapefiles, csvs, and file geo-databases, an additional third is aerial imagery and lidar raster data, and the final third is movies, music, and photos (of course).
What (I think) I'm looking for: I might be trying to do too much with one machine, and I can accept that if need be. Ultimately I can offload everything else and be happy with just consolidated storage. That being said, this is what I'm looking to do:
1)- (Primary) Consolidated Storage. At least 40 TB usable bulk storage. I'd like to be able to dump all of my existing drives into one primary storage machine, and re-purpose the old thecus' with new drives as additional backup for critical documents, one on-site and one-off site.
2)- (Secondary, possibly not required with #3) SAN or similar. This is foreign and new to me, but some type of iscsi or SAN to use with CAD and GIS workstations, max 5 at a time, usually 2. Latency is more important than bandwidth, especially with the nature of GIS software, but high bandwidth helps also. (Especially in ESRI, there are countless issues with running a processing task over a network. The two best ways to mitigate the issue is 1) the workstations need to see the share as a local drive, and 2) many processing tools run symmetrically with unpredictable timeout errors, so network and drive access latency is also key.)
---and/or---
3)- (Third, possibly not required with #2) VMs If practical, I'd also like to be able to run vm's as needed for GIS work. Generally this would be one linux vm with postgre/postgis database, one windows vm with esri ArcGIS, one windows/linux with ESRI ARC GIS Server, and a third vm running a versioning system (previously used git and tortoise svn, but am investigating the newer geoGIT or other alternatives.) This would be more of a longer term online resting place for existing work, and perhaps a sandbox to test configurations of future GIS processing systems architecture/workflow. Small jobs of less than a week of processing time might find themselves being run entirely within, but anything longer, and it would be exported to dedicated hardware once the configuration and workflow are figured out.
The plus of this is that network store issues listed in #2 can be avoided. The negative is that it would likely require a workstation grade graphics card to be installed to fully utilize. I'm not sure if this is even possible or practical.
4)- (This is strictly lagniappe.) Host and trans-code videos. It's not the purpose or excuse for the system, but probably will be what the processing power is used for 50% of the time. I previously have been a SageTV holdout, but may give plex another shot. I can always use this machine for media storage, and offload the trans-coding to my existing pvr machine. But that seems like a waste of processing power.
Where I'm at:
I think I have the basics filled in. Between Intel's new-ish way of grouping/naming processors, and trying to step up to a true server motherboard, I find myself facing a much steeper learning curve than I anticipated. It's made me give up on trying to do something entirely from scratch for my first build of this type. Also, I think I'm to the point of diminishing returns, research-wise, and it's time to ask some big picture questions.
I've pretty much decided on going with an x9 build, after going through many of the suggestions in the forums over the last ~6 months.
Perhaps prematurely, I've picked up one of these:
https://unixsurplus.com/collections...-2x-e5-2680-2-8ghz-192gb-2-port-10gbe-sfp-nic
36 bay SuperMicro 4u case
X9DRi-LN4F+ Mother Board
2x intel xeon e5-2680 2.7ghz
128GB Registered ECC DDR3 RAM
1x AOC Dual Port 10GbE SFP+
1x LSI 9211-8i (JBOD IT mode)
Additionally, I would add:
2x ~64GB ssd boot drives TBD
1x intel p3700 400GB or other suggestion
Additional HBAs?
For Drives, as mentioned above, I'm thinking of going in 2 pools/ phases: 1-capacity and 2-performance.
24x 3.5" 8TB hdds in the front slots for bulk storage- (possibly HGST HUH728080ALE600 3.5" 8TB SATA 6Gb/s 7.2K RPM 128M 512E ISE HDD for $229 oem)
(4 z2 vdevs @ 6ea 8TB drives ~ 54.4 TiB usable @ $101/TiB, per http://wintelguy.com/zfs-calc.pl) Open to suggestions on this.
and
12x (or more if they fit) 2.5" ssd's in the rear slots. Since I'm still investigating this, model and configuration TBD. (I know some have suggested not using the rear slots for 3.5" hdd's due to air flow issues, is this true for 2.5" ssd's also?)
I'd load the spinners first, then migrate the bulk of my individual drives using the rear drive slots, then once that's done, install the ssd's. Is this an ok protocol?
Obviously, I'm open to suggestions and critiques. I think the x9 board is where I want to be, as far as bang for the buck. (Confident enough that I already bought it. Although it can always be re-purposed or sold.)
But these are the questions I think I need to be asking, wrt hardware:
1) Am I on the right track? Missing any major blunders? Is it worth doing a combination drive type system like this? ie going for a combination of capacity and performance, or should I stick to one or the other?
2) I've given the SLOG primer a couple of reads, but I think I need to read it at least a couple more times or go look at some more examples. If I'm getting it, it's for latency and write integrity. The guide seems very good at the Why, but it doesn't go very deep into the How. How should it be sized? Is the 400GB model sufficient? If I go with the combo of hdds and ssds, would it be "implemented" with both types of drives, or should there be one SLOG per pool? I still need to get a better understanding here. The guide recommends the intel P3700. Is that still good advice?
3) The 10Gbe card included with the system seems to be an Intel x520, it's paid for, so is it worth swapping out for a Chelsio? Is the x520 bsd driver still buggy? I could always use the x520 elsewhere. Do I just need to experiment with it depending on performance?
4) Do I need to look at additional/better HBA cards? SAS3? Any reason to do a separate card per pool?
Thanks!
B
Been lurking for about 6 months, and am ready to pull the trigger on a build. I'm wondering if I can't try for a best of both worlds build, using hdds for 24 drives and ssds for the remaining 12? Knock out the bulk of my needed storage with the spinners, and get some high performance vm storage with the ssd's. I assume there's a reason I haven't come across more build like this, but that's why I'm asking.
Background:
This will be my first experience with FreeNAS. I have a good bit of build experience, but mostly workstations and pvr builds, and have done very little but maintenance of existing builds for the last ~5 years. My last full build was a x58 w3570 gis workstation for postGRE/GIS database crunching in 2012. I also have 2 Thecus 7700s that are getting up there in age (and just about at capacity), with 7x3TB drives in each in Raid6, as well as a random assortment of maybe 2 dozen 1-4TB disks, a combination of oldscratch drives, pvrs and cad workstations, all of which I'd like to consolidate. About a third of my data is GIS related shapefiles, csvs, and file geo-databases, an additional third is aerial imagery and lidar raster data, and the final third is movies, music, and photos (of course).
What (I think) I'm looking for: I might be trying to do too much with one machine, and I can accept that if need be. Ultimately I can offload everything else and be happy with just consolidated storage. That being said, this is what I'm looking to do:
1)- (Primary) Consolidated Storage. At least 40 TB usable bulk storage. I'd like to be able to dump all of my existing drives into one primary storage machine, and re-purpose the old thecus' with new drives as additional backup for critical documents, one on-site and one-off site.
2)- (Secondary, possibly not required with #3) SAN or similar. This is foreign and new to me, but some type of iscsi or SAN to use with CAD and GIS workstations, max 5 at a time, usually 2. Latency is more important than bandwidth, especially with the nature of GIS software, but high bandwidth helps also. (Especially in ESRI, there are countless issues with running a processing task over a network. The two best ways to mitigate the issue is 1) the workstations need to see the share as a local drive, and 2) many processing tools run symmetrically with unpredictable timeout errors, so network and drive access latency is also key.)
---and/or---
3)- (Third, possibly not required with #2) VMs If practical, I'd also like to be able to run vm's as needed for GIS work. Generally this would be one linux vm with postgre/postgis database, one windows vm with esri ArcGIS, one windows/linux with ESRI ARC GIS Server, and a third vm running a versioning system (previously used git and tortoise svn, but am investigating the newer geoGIT or other alternatives.) This would be more of a longer term online resting place for existing work, and perhaps a sandbox to test configurations of future GIS processing systems architecture/workflow. Small jobs of less than a week of processing time might find themselves being run entirely within, but anything longer, and it would be exported to dedicated hardware once the configuration and workflow are figured out.
The plus of this is that network store issues listed in #2 can be avoided. The negative is that it would likely require a workstation grade graphics card to be installed to fully utilize. I'm not sure if this is even possible or practical.
4)- (This is strictly lagniappe.) Host and trans-code videos. It's not the purpose or excuse for the system, but probably will be what the processing power is used for 50% of the time. I previously have been a SageTV holdout, but may give plex another shot. I can always use this machine for media storage, and offload the trans-coding to my existing pvr machine. But that seems like a waste of processing power.
Where I'm at:
I think I have the basics filled in. Between Intel's new-ish way of grouping/naming processors, and trying to step up to a true server motherboard, I find myself facing a much steeper learning curve than I anticipated. It's made me give up on trying to do something entirely from scratch for my first build of this type. Also, I think I'm to the point of diminishing returns, research-wise, and it's time to ask some big picture questions.
I've pretty much decided on going with an x9 build, after going through many of the suggestions in the forums over the last ~6 months.
Perhaps prematurely, I've picked up one of these:
https://unixsurplus.com/collections...-2x-e5-2680-2-8ghz-192gb-2-port-10gbe-sfp-nic
36 bay SuperMicro 4u case
X9DRi-LN4F+ Mother Board
2x intel xeon e5-2680 2.7ghz
128GB Registered ECC DDR3 RAM
1x AOC Dual Port 10GbE SFP+
1x LSI 9211-8i (JBOD IT mode)
Additionally, I would add:
2x ~64GB ssd boot drives TBD
1x intel p3700 400GB or other suggestion
Additional HBAs?
For Drives, as mentioned above, I'm thinking of going in 2 pools/ phases: 1-capacity and 2-performance.
24x 3.5" 8TB hdds in the front slots for bulk storage- (possibly HGST HUH728080ALE600 3.5" 8TB SATA 6Gb/s 7.2K RPM 128M 512E ISE HDD for $229 oem)
(4 z2 vdevs @ 6ea 8TB drives ~ 54.4 TiB usable @ $101/TiB, per http://wintelguy.com/zfs-calc.pl) Open to suggestions on this.
and
12x (or more if they fit) 2.5" ssd's in the rear slots. Since I'm still investigating this, model and configuration TBD. (I know some have suggested not using the rear slots for 3.5" hdd's due to air flow issues, is this true for 2.5" ssd's also?)
I'd load the spinners first, then migrate the bulk of my individual drives using the rear drive slots, then once that's done, install the ssd's. Is this an ok protocol?
Obviously, I'm open to suggestions and critiques. I think the x9 board is where I want to be, as far as bang for the buck. (Confident enough that I already bought it. Although it can always be re-purposed or sold.)
But these are the questions I think I need to be asking, wrt hardware:
1) Am I on the right track? Missing any major blunders? Is it worth doing a combination drive type system like this? ie going for a combination of capacity and performance, or should I stick to one or the other?
2) I've given the SLOG primer a couple of reads, but I think I need to read it at least a couple more times or go look at some more examples. If I'm getting it, it's for latency and write integrity. The guide seems very good at the Why, but it doesn't go very deep into the How. How should it be sized? Is the 400GB model sufficient? If I go with the combo of hdds and ssds, would it be "implemented" with both types of drives, or should there be one SLOG per pool? I still need to get a better understanding here. The guide recommends the intel P3700. Is that still good advice?
3) The 10Gbe card included with the system seems to be an Intel x520, it's paid for, so is it worth swapping out for a Chelsio? Is the x520 bsd driver still buggy? I could always use the x520 elsewhere. Do I just need to experiment with it depending on performance?
4) Do I need to look at additional/better HBA cards? SAS3? Any reason to do a separate card per pool?
Thanks!
B
