I till start with the most important tl;dr; question, then go into the details of the server.
It will have a SAS-3 HBA and SAS-3 backplane and use 7200 RPM SAS-drives. Do I need SAS-3 drives for getting the HBA<->backplane part to operate in SAS-3 mode or will the SAS-2 speed only affect the backplane<->harddrive part?
Reason for asking is that Super Micro Computer (SMC) seems to be dumping out the older HGST 7K4000 drives for 30% less then 7K6000 and that makes quite an impact on the price of the whole system.
Edit: My finding about this can be read in my replies #12 and #15. Question is answered.
Background:
Obviously this is system for a company (my employer) but I hope I can get good advice here on the forum. I've been reading the sticked threads and some others so I hope I avoid some newbie mistakes in the choice of hardware.
We use 100-150 virtual machines (mostly Windows Server) spread across 4 VmWare ESXi servers which are really bottlenecked by the storage solution we have today. Most immediate problem is lack of space (and that is not expandable any longer). It's a year or so left on the expected life span for the storage solution. But many of the virtual machines don't have such a heavy load and are not mission critical, quite a few are just inactive space hogs (but the virtual machine owners want to keep them). And there are powered off machines. Tiering our storage for virtual machines with one expensive solution with a lot of redundancy (multipaths, master/slave controllers etc.) and uptime (on-site support) to another with more space but only redundancy for the data itself will be how we work from now on. This system will be sort of a retirement home for low-use virtual machines! It should have a lot of space but if say the motherboard brakes we won't have a problem with waiting for a replacement.
Okay, over to the fun part, the choice components for the build itself. This will be used with the latest FreeNAS version (9.3 Stable).
We're going with a SMC machine and I'm leaning towards letting SMC built it also.
Thanks for any advice and best regards.
Christer
It will have a SAS-3 HBA and SAS-3 backplane and use 7200 RPM SAS-drives. Do I need SAS-3 drives for getting the HBA<->backplane part to operate in SAS-3 mode or will the SAS-2 speed only affect the backplane<->harddrive part?
Reason for asking is that Super Micro Computer (SMC) seems to be dumping out the older HGST 7K4000 drives for 30% less then 7K6000 and that makes quite an impact on the price of the whole system.
Edit: My finding about this can be read in my replies #12 and #15. Question is answered.
Background:
Obviously this is system for a company (my employer) but I hope I can get good advice here on the forum. I've been reading the sticked threads and some others so I hope I avoid some newbie mistakes in the choice of hardware.
We use 100-150 virtual machines (mostly Windows Server) spread across 4 VmWare ESXi servers which are really bottlenecked by the storage solution we have today. Most immediate problem is lack of space (and that is not expandable any longer). It's a year or so left on the expected life span for the storage solution. But many of the virtual machines don't have such a heavy load and are not mission critical, quite a few are just inactive space hogs (but the virtual machine owners want to keep them). And there are powered off machines. Tiering our storage for virtual machines with one expensive solution with a lot of redundancy (multipaths, master/slave controllers etc.) and uptime (on-site support) to another with more space but only redundancy for the data itself will be how we work from now on. This system will be sort of a retirement home for low-use virtual machines! It should have a lot of space but if say the motherboard brakes we won't have a problem with waiting for a replacement.
Okay, over to the fun part, the choice components for the build itself. This will be used with the latest FreeNAS version (9.3 Stable).
We're going with a SMC machine and I'm leaning towards letting SMC built it also.
- Barebone. SuperStorage Server 5048R-E1CR36L
- Which includes the motherboard X10SRH-CLN4F. The LSI3008 SAS-3 controller is in RAID mode (IR) but according to an "application engineer" at SMC (via my sales representative) replied it is possible to flash it to IT mode (simple HBA) and pointed to the ftp://ftp.supermicro.nl/driver/sas/lsi/3008/Firmware/ FTP for firmwares to the LSI3008 so I believe them.
Via the mpr driver it should be supported, right? - The chassis is the 847BE1C-R1K28LPB with space for 36 3.5" hotswap harddrives. I plan to have the MCP-220-82609-0N 2x2.5" rear hotswap kit installed, just asking SMC to really confirm it's used for SATA-ports. Also some interior MCP-220-84701-0N 2x2.5" drive bays. I'm also requesting 2 additional 3.5" hotswap caddies so I can have spare drives mounted in caddies ready to go. Plus the MCP-210-84601-0B front bezel because I like to have bezels on servers :)
- Which includes the motherboard X10SRH-CLN4F. The LSI3008 SAS-3 controller is in RAID mode (IR) but according to an "application engineer" at SMC (via my sales representative) replied it is possible to flash it to IT mode (simple HBA) and pointed to the ftp://ftp.supermicro.nl/driver/sas/lsi/3008/Firmware/ FTP for firmwares to the LSI3008 so I believe them.
- CPU. The motherboard is a single-socket LGA 2011 board which probably holds down the price a bit plus the processors are a lot cheaper compared for the same speed. I'm thinking of getting the Intel Xeon E5-1650 V3. 6 core, 15 MB smart cache and 3.5 GHz. Closest contender in dual cpu range is the E5-2643 V3 which only offers 5 MB more cache and nearly trice the price.
- RAM. The motherboard have 8 DIMM slots and the price for 16 GB modules is quite okay so I'm thinking of 4x16 = 64 GiB RAM (DDR4 ECC REG). Leaves room for expansion without replacing modules. Looking at the Samsung memory that is on the list of tested memories for the board.
- Zpool harddrives. Thinking of buying 20 drives with 18 in the chassis. Looking at 4 TB drives which will give us plenty of space. 3 vdevs each containing RAIDZ2 of 6 drives. Usable drives and capacity: 12*4 = 48 TB minus 10-20% free space. Write performance is helped by more vdevs, right? I'm leaning towards the HGST Ultrastar 7K6000 4TB SAS if I should have SAS-3 or its predecessor 7K4000 if SAS-2 is okay (the first question I asked). Anyone with thoughts of these HGST drives? Another choice would be the (PDF) Seagate Enterprise Capacity 3.5 HDD SAS 512E 4TB ST4000NM0034
- SLOG. As this is going to be a datastore for virtual machine harddrives (vmdk) and I really want sync writes (sync=always) with a lot of random writes I've gathered is the time you should have a SLOG. I've understood that you actually only need a few GB but I think HGST ZeusRAM SSD is too expensive. I'm think I will choose a Intel S3710 SSD with 200 GB and crank up the over provisioning size (hope it's possible with hdparm in linux for this drive) so I only end up with say 20 GB usable space. In it's standard config it is rated for 10 DW 5 years and I don't expect we'll be stressing it to that. Have I've gotten it right that the ZIL do sequential writes? The 200 GB drive is rated for 300 MB/s where the 400 GB drive is rated for 470 MB/s. They have the same rated IOPS of 43 000 so if I misunderstood the ZIL. I don't think I will have the SLOG on mirrored drives. Don't shoot me yet! As I understand it newer ZFS versions will work fine if the SLOG dies, it will just direct the ZIL to the zpool and nothing was lost since the ZIL exists in RAM. The SSD is capacitor backed SSD and the data in the SLOG is only important for like 15 seconds? So it should be that the SSD dies AND the computer dies (resets or other thing) within that little time frame which would corrupt the zpool. The system will be powered through a UPS with extra battery so we're looking at ~30 minutes of battery time.
- L2ARC. Bigger and not so much writing. Thinking of the Intel S3610 SSD with 800 GB. 3 DW for 5 years. Perhaps we will have little use for it with 64 GiB RAM?
- Boot media. The Supermicro SATA DOM (Disk on Module) 32 GB. There quite inexpensive so I'm thinking of buying 4. Perhaps plugging them all in (2 via special SATA ports, 2 via normal and extra power cable) or have some spare outside the machine. I know FreeNAS boots to RAM but I would like to avoid having to reinstall and load config files. And mirroring only two would make me nervous if one brakes and waiting for a new for probably few weeks.
- Storage NIC. Intel X520-DA2. The two SAN switches have mostly SPF+ ports and I have good experience with the X520 before. Should be supported by the ixgbe driver.
Thanks for any advice and best regards.
Christer
Last edited: