jal
Dabbler
- Joined
- Jan 12, 2015
- Messages
- 14
This will be a little anticlimactic after TrevorX's epic post. No nifty pics, and my build is very similar. I actually wanted the Q35-B case, but could only get the Q-08B (thus also no hot-swap). I used WD-Red 6T disks, and used Sandisk thumb drives for the OS. No external devices, so no SATA card.
ASRock C2750D4I
Lian-Li PC-Q08B
Crucial 8GB DDR3 ECCx2
WD Red 6TB x7 (six in the machine, one cold spare)
Corsair CX 600 PSU
The decision tree fell out from the use case and the desired form factor: I wanted a small, unobtrusive machine that would still provide 20+TB. Use case is to serve as a hub for (mostly) one geek's computing hobbies - it (as soon as I reconfigure everything) will serve a small Openstack cluster that serves media, as an archive of stuff that's been following me around since school, serves a few webapps (mostly things I've written for the household/my own amusement/some projects), hosts backups (Time Machine and remote mirroring of a few virtual hosts running at providers) and as backing store for some CI build/test work. Eventually, it may get an external case and more storage for more speed to better handle video editing, but we're not there yet, and I may just attach that to the workstation in question if/when needed.
The two big decision points were the motherboard and sizing the disks. Both are nontrivial cost differences, with the disks obviously offering the biggest delta. I liked the ASRock mobo, so the choice there was between the 2750 and the 2550. Reading around seemed to indicate that the 25500 actually is a little underpowered, so it wasn't that hard to spend a few more dollars. Considering this box should live for a long time and the old cliche about a disk's stable-state being full, it I chose to just max those out, even if I was wincing a bit at the cost. RAM is another one. I decided to see how it behaves with 16G before adding two more sticks. Once I get everything reconfigured, I'm going to run some builds and watch performance - given that the bigger ones are many thousands of individual small files and some fairly heavy write activity, we'll see how that goes.
I wish the Lian Li cases were easier to come by... But I found one that would work, after I decided I can live without hot swap. An aesthetic annoyance is that the case has front ports for USB3 and audio which are going unused. (I'd prefer they not be there.) But this is no big deal. A bigger problem is cable management inside - see below.
I originally planned to build a duplicate for remote mirroring, and may yet do that. But cost obviously comes in to play, and it is probably overkill right now. For the time being, I have two 4x4TB WD Greens in external enclosures I'm going to rotate off-site for backup (these were previously used for a lot of what this box is taking over, and are big enough to hold my current data). A smallish set of critical stuff is also mirrored to an offsite backup provider. I chose the Greens because they were absurdly cheap at the time. When I get close to needing more backup space, or those start dying, I'll reconsider where I'm at.
The one problem I ran in to is that the USB header I bought for hosting two SanDisk Cruzer thumb drives (to run mirrored) for the OS appears to be broken. The motherboard only sees one of the adapters. For the time being, I'm running with the one stuck in one of the rear USB ports, and I need to RMA the header and try that again.
Oh, one other small problem cropped up during installation. I didn't expect the GUI to enumerate network links at startup (really, I didn't think about it), but it makes sense in retrospect, and most people probably do things in a less haphazard fashion.
Cabling is also a bit of a mess in this case - the power and SATA cables don't have enough room, given the location of the motherboard and drive bays. If I were doing this again, I would probably wait and look harder for the Q35.
Config-wise, this is one pool with RAIDZ2. I'm using GELI (yes, the keys are in a safe place), because I like to not have to think about disk disposal, the CPU has hardware support, reading around seemed to indicate that it isn't a significant bottleneck in my HW configuration, and high write performance isn't generally that important here. I've done basically no tweaking so far, taking a wait-and-see approach. Datasets, at least for now, are set up for home directories, Time Machine, logs, database, and media.
I need to spend some more time thinking about how I want to handle the VMs. Several of them (mostly the build machines) are disposable; they can come up via PXE from a boot image and a Saltstack formula just as easily as booting, the only cost being time. This offers a lot of options: for the time being, I'll probably just use the crappy old internal drives on the physical machines for the VM root volumes; when they die, I will probably move them over to use this box via iSCSI.
The non-disposable VMs require more thought. The plan now for the database VM (Postgres) is to mount its dataset over NFS (I'm aware of the warnings on that and know how to configure it to work). Mounting it as a block device instead is tempting, but I don't want to make the DB my first experiment there because too many other things go down if I break it. This is my hobby, not my job, so it might take me a while to come back to it. Most of the non-disposable ones are halfway set up with Saltstack formulas, but I've gotten sloppy, and (in addition to incomplete formulas) need to fully separate config from data to make those disposable. Doing that isn't a quick job, so I'm stuck balancing doing it right and not wanting this to take me a year to migrate.
Logging... This is the part where the old hands will start telling me I'm making a mistake. I want better reporting than what is baked in. This box will send logs to my loghost, which shoves stats and logs into Postgres for some purposes and Elasticsearch for others (something I already do with other machines). It is nice that FreeNAS uses collectd, but also a problem, in that I want to tweak what it watches. I'm going to spend some time reading the code and learning how the stock setup is configured before doing anything, but will probably customize those and write some scripts to restore the stock-state in order to handle upgrades. (Yes, I know that is not recommended behavior and I understand the risks. Professionally, I work in devops and do this sort of thing for a living, and by temperament void warranties as a matter of course. Closed course, don't try this at home, etc.) I need to learn more about the plugin architecture, and if it seems appropriate/feasible I'll probably write that stuff as a plugin.
Security: This machine does not have direct internet access, that's blocked at the gateway (an OpenBSD box). NTP is proxied, and I need to dig around to figure out what it wants to look at when checking for updates in order to proxy that as well. It probably wants net access to other resources, and that will shake out as I go. As above, logs will be processed on a VM and stored in DBs; additionally, part of the log parsing looks for interesting security-related stuff and ships that to an offsite loghost. I already have a CA I use for other things, and need to generate keys for this machine and set that up.
All-in-all, this has been a great experience. FreeNAS rocks. I'm happy to be back in FreeBSD-land - professionally, everything I do is on Linux; I've been away for a long time, and have a lot to come back up to speed with. The forums here are an excellent resource, and what little contact I've had with you all has been really positive. Three thumbs up!
One suggestion I would throw out there for others: when building, don't just shove all the drives in at once. Do it one at a time and boot in between, so that you can physically label the drives with their IDs. The first pass through I forgot to do this, like I always do, and remembered only when checking something that looked odd at first. Your future self will be very happy with your past self the first time you need to swap a drive.
---
Oh, and a little about me. I'm a career techie living in the Bay Area, US. At various times, I've been a sysadmin, DBA, programmer, consultant, and startup management. I'm currently a systems engineer/devops monkey at a startup. For pleasure, I enjoy long walks and sunsets... wait, wrong forum.
(Edit: clean up text, add a bit about me.)
ASRock C2750D4I
Lian-Li PC-Q08B
Crucial 8GB DDR3 ECCx2
WD Red 6TB x7 (six in the machine, one cold spare)
Corsair CX 600 PSU
The decision tree fell out from the use case and the desired form factor: I wanted a small, unobtrusive machine that would still provide 20+TB. Use case is to serve as a hub for (mostly) one geek's computing hobbies - it (as soon as I reconfigure everything) will serve a small Openstack cluster that serves media, as an archive of stuff that's been following me around since school, serves a few webapps (mostly things I've written for the household/my own amusement/some projects), hosts backups (Time Machine and remote mirroring of a few virtual hosts running at providers) and as backing store for some CI build/test work. Eventually, it may get an external case and more storage for more speed to better handle video editing, but we're not there yet, and I may just attach that to the workstation in question if/when needed.
The two big decision points were the motherboard and sizing the disks. Both are nontrivial cost differences, with the disks obviously offering the biggest delta. I liked the ASRock mobo, so the choice there was between the 2750 and the 2550. Reading around seemed to indicate that the 25500 actually is a little underpowered, so it wasn't that hard to spend a few more dollars. Considering this box should live for a long time and the old cliche about a disk's stable-state being full, it I chose to just max those out, even if I was wincing a bit at the cost. RAM is another one. I decided to see how it behaves with 16G before adding two more sticks. Once I get everything reconfigured, I'm going to run some builds and watch performance - given that the bigger ones are many thousands of individual small files and some fairly heavy write activity, we'll see how that goes.
I wish the Lian Li cases were easier to come by... But I found one that would work, after I decided I can live without hot swap. An aesthetic annoyance is that the case has front ports for USB3 and audio which are going unused. (I'd prefer they not be there.) But this is no big deal. A bigger problem is cable management inside - see below.
I originally planned to build a duplicate for remote mirroring, and may yet do that. But cost obviously comes in to play, and it is probably overkill right now. For the time being, I have two 4x4TB WD Greens in external enclosures I'm going to rotate off-site for backup (these were previously used for a lot of what this box is taking over, and are big enough to hold my current data). A smallish set of critical stuff is also mirrored to an offsite backup provider. I chose the Greens because they were absurdly cheap at the time. When I get close to needing more backup space, or those start dying, I'll reconsider where I'm at.
The one problem I ran in to is that the USB header I bought for hosting two SanDisk Cruzer thumb drives (to run mirrored) for the OS appears to be broken. The motherboard only sees one of the adapters. For the time being, I'm running with the one stuck in one of the rear USB ports, and I need to RMA the header and try that again.
Oh, one other small problem cropped up during installation. I didn't expect the GUI to enumerate network links at startup (really, I didn't think about it), but it makes sense in retrospect, and most people probably do things in a less haphazard fashion.
Cabling is also a bit of a mess in this case - the power and SATA cables don't have enough room, given the location of the motherboard and drive bays. If I were doing this again, I would probably wait and look harder for the Q35.
Config-wise, this is one pool with RAIDZ2. I'm using GELI (yes, the keys are in a safe place), because I like to not have to think about disk disposal, the CPU has hardware support, reading around seemed to indicate that it isn't a significant bottleneck in my HW configuration, and high write performance isn't generally that important here. I've done basically no tweaking so far, taking a wait-and-see approach. Datasets, at least for now, are set up for home directories, Time Machine, logs, database, and media.
I need to spend some more time thinking about how I want to handle the VMs. Several of them (mostly the build machines) are disposable; they can come up via PXE from a boot image and a Saltstack formula just as easily as booting, the only cost being time. This offers a lot of options: for the time being, I'll probably just use the crappy old internal drives on the physical machines for the VM root volumes; when they die, I will probably move them over to use this box via iSCSI.
The non-disposable VMs require more thought. The plan now for the database VM (Postgres) is to mount its dataset over NFS (I'm aware of the warnings on that and know how to configure it to work). Mounting it as a block device instead is tempting, but I don't want to make the DB my first experiment there because too many other things go down if I break it. This is my hobby, not my job, so it might take me a while to come back to it. Most of the non-disposable ones are halfway set up with Saltstack formulas, but I've gotten sloppy, and (in addition to incomplete formulas) need to fully separate config from data to make those disposable. Doing that isn't a quick job, so I'm stuck balancing doing it right and not wanting this to take me a year to migrate.
Logging... This is the part where the old hands will start telling me I'm making a mistake. I want better reporting than what is baked in. This box will send logs to my loghost, which shoves stats and logs into Postgres for some purposes and Elasticsearch for others (something I already do with other machines). It is nice that FreeNAS uses collectd, but also a problem, in that I want to tweak what it watches. I'm going to spend some time reading the code and learning how the stock setup is configured before doing anything, but will probably customize those and write some scripts to restore the stock-state in order to handle upgrades. (Yes, I know that is not recommended behavior and I understand the risks. Professionally, I work in devops and do this sort of thing for a living, and by temperament void warranties as a matter of course. Closed course, don't try this at home, etc.) I need to learn more about the plugin architecture, and if it seems appropriate/feasible I'll probably write that stuff as a plugin.
Security: This machine does not have direct internet access, that's blocked at the gateway (an OpenBSD box). NTP is proxied, and I need to dig around to figure out what it wants to look at when checking for updates in order to proxy that as well. It probably wants net access to other resources, and that will shake out as I go. As above, logs will be processed on a VM and stored in DBs; additionally, part of the log parsing looks for interesting security-related stuff and ships that to an offsite loghost. I already have a CA I use for other things, and need to generate keys for this machine and set that up.
All-in-all, this has been a great experience. FreeNAS rocks. I'm happy to be back in FreeBSD-land - professionally, everything I do is on Linux; I've been away for a long time, and have a lot to come back up to speed with. The forums here are an excellent resource, and what little contact I've had with you all has been really positive. Three thumbs up!
One suggestion I would throw out there for others: when building, don't just shove all the drives in at once. Do it one at a time and boot in between, so that you can physically label the drives with their IDs. The first pass through I forgot to do this, like I always do, and remembered only when checking something that looked odd at first. Your future self will be very happy with your past self the first time you need to swap a drive.
---
Oh, and a little about me. I'm a career techie living in the Bay Area, US. At various times, I've been a sysadmin, DBA, programmer, consultant, and startup management. I'm currently a systems engineer/devops monkey at a startup. For pleasure, I enjoy long walks and sunsets... wait, wrong forum.
(Edit: clean up text, add a bit about me.)
Last edited:
