Jeremy Lea
Cadet
- Joined
- Jan 7, 2015
- Messages
- 5
Hi all,
I was looking for comments on a new build. I am replacing a VMware Essentials Plus cluster, and was going to use their VSA software, but that was killed between when I had to put a quote in and when I got the money (and replaced with vSAN, which blows the budget). I then considered a Synology device, but the cost compared to FreeNAS is not good... I am very comfortable with FreeBSD and ZFS (reg@), but I tend to go with more mainstream options in case someone else has to take over admin duties from me. However, in this case I can't see the benefits.
I work with a small research group, where I am the defacto IT person. I can only replace hardware every six years or so, and it typically takes 18 months between when I have to put together a budget and when I actually get money. However, I can get money for upgrades (more RAM, new HDD, etc.). As a result I like to buy newest generation hardware and make sure it can be incrementally upgraded if needed. I have a nice budget, but I also have lots of things to do with it, so I don't want to waste money on things that won't be used.
We currently have a 3 machine VMware cluster that is a bit cobbled together. I have around 25 VMs, some of which are not active, and they are using ~1TB of disk space for the OS disks, about ~2TB of critical data files, ~1TB of database storage (in SQL Server and PostgreSQL) and ~4TB of backups of various types. I also have ~60TB of not so critical data in a Dell MD3000+2xMD1000, which I'm not going to touch. The storage is split between local storage in the hosts and the MD3000, with the VMs mostly on a shared LUN in the MD3000 to enable vMotion. I currently have 64GB of RAM in the cluster, which is my constraint for more VMs. I don't generally have HA requirements (we don't make money on IT), but a bunch of people notice if things go down for extended periods. Mostly the storage is quiet, but it does need to be fairly snappy when it is needed (and needs to still be considered snappy in six years time). Mostly, I just need to make sure that critical data never disappears ;-)
I plan on replacing two of the ESXi nodes in the cluster with diskless Haswell based machines (in a four node 2U machine with two nodes for VMware and two nodes for other compute tasks - still deciding on ASUS RS720Q-E8-RS12 or Intel H2312XXKR2/HNS2600KP). The one machine handling the MD3000 will stay as the third ESXi host, mostly because it will be to difficult to unravel the mess of 2TB raw disk mappings that I had to do to make that 60TB of data available on the network... I don't want a repeat of that.
So anyway, on to the FreeNAS build. I'm looking at a mostly Intel box:
Chassis: Intel R2224WTTYS+AXX1100PCRPS PSU
Controller: Intel RMS3JC080 (LSI SAS 3008 based) and Intel RES3TV360 SAS Expander
CPU: 2x Intel E5-2620v3 (maybe E5-2609v3?)
RAM: 2x Crucial 32GB DDR4 kits (4x8GB) (CT4K8G4VFS4213) (maybe 4x?)
OS: 2x 80GB SSD (probably Intel 530)
VM Storage: 8x480GB SSD (mirrored, stripped) for the VMs and the databases (Intel 530)
Data Storage: 16xWD10EFRX (1TB 2.5in Red drives), in two RAIDZ2 pools. I might switch some of these for 2TB (ST2000LM003) drives, and make two RAIDZ3 pools (9x1TB, 6x2TB).
Network: 2xNetgear S3300-28X (GS728TX), stacked, with the storage multi-pathed via two VLANs.
I don't think I'll need any SAS disks, and with the main IO load on the SSDs I don't think I'll be needing special ZIL/L2ARC devices. I'll probably dedup the VM storage, because it has a lot of copies of Windows. Currently the build price of the box is about $8000. The hardware is mostly available now, but it looks like there might be some lag because the chassis has just been released. Without disks, this is about $4,800, which is ~$800 more than a Synology RS3614RPxs which is what I was considering - but it is a much nicer box.
I will stick to backing up within the VMware environment, although I might consider using ZFS send to make my offsite backups. Also, I will use iSCSI for everything. I might present some of the storage (e.g. for SQL Server) directly to Windows rather than go via VMware.
Some questions:
1. Should I be considering Enterprise class SSDs (at ~twice the cost)?
2. Could I use the cheaper CPU? It's only ~$300 extra for the 2620s, but I feel like the CPU will be mostly idle anyway...
3. Should I put in more than 64GB of RAM?
4. Does anyone know of issues with this hardware? Generally I find FreeBSD works well on brand new server hardware, even if it doesn't support all the newest features.
5. I kind of like having everything virtual. Should I consider running ESXi on this machine with FreeNAS on top of that? I considered for a while buying two boxes like this and make on physical server and a passive virtual server on the other with hastd running to keep a backup. This would be more like the VSA architecture I originally planned.
6. Does anyone have any ideas on iSCSI<->Windows performance vs iSCSI<->ESXi<->Windows?
7. Should I just buy a TrueNAS box?
Any other thoughts are welcome.
-Jeremy
I was looking for comments on a new build. I am replacing a VMware Essentials Plus cluster, and was going to use their VSA software, but that was killed between when I had to put a quote in and when I got the money (and replaced with vSAN, which blows the budget). I then considered a Synology device, but the cost compared to FreeNAS is not good... I am very comfortable with FreeBSD and ZFS (reg@), but I tend to go with more mainstream options in case someone else has to take over admin duties from me. However, in this case I can't see the benefits.
I work with a small research group, where I am the defacto IT person. I can only replace hardware every six years or so, and it typically takes 18 months between when I have to put together a budget and when I actually get money. However, I can get money for upgrades (more RAM, new HDD, etc.). As a result I like to buy newest generation hardware and make sure it can be incrementally upgraded if needed. I have a nice budget, but I also have lots of things to do with it, so I don't want to waste money on things that won't be used.
We currently have a 3 machine VMware cluster that is a bit cobbled together. I have around 25 VMs, some of which are not active, and they are using ~1TB of disk space for the OS disks, about ~2TB of critical data files, ~1TB of database storage (in SQL Server and PostgreSQL) and ~4TB of backups of various types. I also have ~60TB of not so critical data in a Dell MD3000+2xMD1000, which I'm not going to touch. The storage is split between local storage in the hosts and the MD3000, with the VMs mostly on a shared LUN in the MD3000 to enable vMotion. I currently have 64GB of RAM in the cluster, which is my constraint for more VMs. I don't generally have HA requirements (we don't make money on IT), but a bunch of people notice if things go down for extended periods. Mostly the storage is quiet, but it does need to be fairly snappy when it is needed (and needs to still be considered snappy in six years time). Mostly, I just need to make sure that critical data never disappears ;-)
I plan on replacing two of the ESXi nodes in the cluster with diskless Haswell based machines (in a four node 2U machine with two nodes for VMware and two nodes for other compute tasks - still deciding on ASUS RS720Q-E8-RS12 or Intel H2312XXKR2/HNS2600KP). The one machine handling the MD3000 will stay as the third ESXi host, mostly because it will be to difficult to unravel the mess of 2TB raw disk mappings that I had to do to make that 60TB of data available on the network... I don't want a repeat of that.
So anyway, on to the FreeNAS build. I'm looking at a mostly Intel box:
Chassis: Intel R2224WTTYS+AXX1100PCRPS PSU
Controller: Intel RMS3JC080 (LSI SAS 3008 based) and Intel RES3TV360 SAS Expander
CPU: 2x Intel E5-2620v3 (maybe E5-2609v3?)
RAM: 2x Crucial 32GB DDR4 kits (4x8GB) (CT4K8G4VFS4213) (maybe 4x?)
OS: 2x 80GB SSD (probably Intel 530)
VM Storage: 8x480GB SSD (mirrored, stripped) for the VMs and the databases (Intel 530)
Data Storage: 16xWD10EFRX (1TB 2.5in Red drives), in two RAIDZ2 pools. I might switch some of these for 2TB (ST2000LM003) drives, and make two RAIDZ3 pools (9x1TB, 6x2TB).
Network: 2xNetgear S3300-28X (GS728TX), stacked, with the storage multi-pathed via two VLANs.
I don't think I'll need any SAS disks, and with the main IO load on the SSDs I don't think I'll be needing special ZIL/L2ARC devices. I'll probably dedup the VM storage, because it has a lot of copies of Windows. Currently the build price of the box is about $8000. The hardware is mostly available now, but it looks like there might be some lag because the chassis has just been released. Without disks, this is about $4,800, which is ~$800 more than a Synology RS3614RPxs which is what I was considering - but it is a much nicer box.
I will stick to backing up within the VMware environment, although I might consider using ZFS send to make my offsite backups. Also, I will use iSCSI for everything. I might present some of the storage (e.g. for SQL Server) directly to Windows rather than go via VMware.
Some questions:
1. Should I be considering Enterprise class SSDs (at ~twice the cost)?
2. Could I use the cheaper CPU? It's only ~$300 extra for the 2620s, but I feel like the CPU will be mostly idle anyway...
3. Should I put in more than 64GB of RAM?
4. Does anyone know of issues with this hardware? Generally I find FreeBSD works well on brand new server hardware, even if it doesn't support all the newest features.
5. I kind of like having everything virtual. Should I consider running ESXi on this machine with FreeNAS on top of that? I considered for a while buying two boxes like this and make on physical server and a passive virtual server on the other with hastd running to keep a backup. This would be more like the VSA architecture I originally planned.
6. Does anyone have any ideas on iSCSI<->Windows performance vs iSCSI<->ESXi<->Windows?
7. Should I just buy a TrueNAS box?
Any other thoughts are welcome.
-Jeremy