Turning old system into a Flash based FreeNas.

[velmeran]

My homelab has been growing every year along with the various things I like to tinker with, and I now find myself pushing the IOP's too heavily on my DL380G7 and UCS 240 M3 systems local storage (8x spinning disk in both). This is mostly due to a few key VM's that would benefit if they were moved to flash storage but there's no expansion room available in the servers themselves.

I have an old gaming system I setup as a third host which has lots of CPU power, but only 16GB of ram. I've mostly used it for one or two high performance VM's like game servers or elasticsearch DB. But its also in a large full-tower with lots of room for internal drives and 4x 5.25 bays. I'm thinking of getting some of the IcyDock bays for the 5.25 bays and 8x SSD's along with a SAS/SATA controller and would like some opinions / sanity check.

System Currently:
i7-3930K Preocessor
Asus X79 motherboard (not sure on the exact model, but its a Full ATX one).
Mellanox X2 10Gb NIC
16GB NON-ECC RAM (4x 4GB DDR3-2400)
500GB 850 EVO SSD (will likely be replaced by an old Patriot Pyro 120GB SSD for local FreeNAS OS).

What I'd like to Add:
LSI 9211-8i SAS Controller
IcyDock 2x5.25 tough armor mount for 8x 5-15mm ssd drives.
Mini Sas cables
8x Samsung 500GB 860 EVO's (I'm not sure what the best Drives per-raid group setup is for this, suggestions welcome).
Intel DC P3600/3700 (Hope to get one cheapish off Ebay for Zlog/Power loss protection).

Total is ~ $1150 which is cheaper than trying to build an all new system for this.

I know the ram's not ECC, and the drives aren't enterprise etc, but this is not a production server and if things went south I have my lab VM's backed up nightly to other freenas systems. I do have 10Gb in my lab and look to make better use of it with this as I have some datasets in the 30-40GB range I want to do some ingest/exporting of but in my current setup it would be hours if not days with my strangled Host IOPs. The CPU is also super overpowered so might see about having it run some jails or something instead of having them as VM's on the main hosts.

Thank you for any suggestions and help you can provide.

 

[HoneyBadger]

I know the ram's not ECC, and the drives aren't enterprise etc, but this is not a production server and if things went south I have my lab VM's backed up nightly to other freenas systems.

As long as you're willing to stand by that disclaimer if something goes amiss. ;)

There's been another user reporting issues with the 860 EVO behind a SAS controller, and the issue might be related to the TRIM implementation. They're getting a lot of failed disks, or at least disks appearing to fail:
https://www.ixsystems.com/community/threads/ssd-failures-from-zfs-pool.73692/
If you're willing to go the eBay route for the SLOG, might I suggest you look for off-lease or refurbished datacenter SSDs like the Samsung PM863, Intel DC S3500/S3600?

16GB of RAM is a little bit on the light side as well, but I understand you might not have the DIMM slots to expand without replacing existing.

 

[velmeran]

As long as you're willing to stand by that disclaimer if something goes amiss. ;)

There's been another user reporting issues with the 860 EVO behind a SAS controller, and the issue might be related to the TRIM implementation. They're getting a lot of failed disks, or at least disks appearing to fail:
https://www.ixsystems.com/community/threads/ssd-failures-from-zfs-pool.73692/
If you're willing to go the eBay route for the SLOG, might I suggest you look for off-lease or refurbished datacenter SSDs like the Samsung PM863, Intel DC S3500/S3600?

16GB of RAM is a little bit on the light side as well, but I understand you might not have the DIMM slots to expand without replacing existing.

Yep, more than willing to deal with the fall out if things go sideways, and yea, 16Gb isn't great but its not really worth it to buy more right now unless I find it really is bottle necking what I want to do. Interesting about that thread, I think I will try and see if I can get the Intel or Samsung disk for a reasonable price.

 

[velmeran]

Humm, while looking for different drives, has anyone used the Samsung 883 DCT drives? the 480GB models are $120 on amazon new and look to have power failure protection and designed for somewhat better enterprise life vs the 860's.

 

[HoneyBadger]

The 883DCT is the successor to the PM863, and it will definitely be better suited for steady-state writes than the consumer EVO drives.

If you've got a compatible M.2 or U.2 slot for the SLOG device, I'd suggest the shiny new Optane P4801X - a user just posted some results in the SLOG benchmarking thread in my signature and it's showing a lot of promise; otherwise, the PCIe-based Intel NVMe cards are also fine. Both the enterprise P-series cards and consumer-oriented Optane 900p have been used to great success by several users here.