Thats the thing, 2.5" only.
But im all ears for suggestions on what spinning disk would suit this setup.
2.5" spinning disks for ZFS top out at 2TB with the Spinpoint M9T (2TB, 2014 era) and the WD Red 1TB drive (2013 era)
The Spinpoints are designed for laptops and gaming consoles and are not quite NAS-appropriate, but are usable.
Basically, all 2.5" hard drives beyond that size went SMR, which is (approximately) incompatible with ZFS.
How big of an impact does a RMS300 (PCIe) slog have on a system like this?
i bet alot if i go with spinning disk?
If you are thinking that SLOG is a write cache, ... it isn't.
https://www.truenas.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/
The SLOG can make sync writes for your VM environment faster, but never faster than the underlying pool with sync writes disabled.
I can get SAS3 SSDs at 400GB for roughly $90 each (refurbished of course).
That seems like it could be a bad choice. Samsung was recently blowing out 860 EVO 1TB's for $99, new, and the 870 EVO's are going for about $120.
There seems to be a lot of undue angst about using consumer grade SSD's in servers. However, I've been doing it for at least a decade, with a relatively large inventory, and most of the original SSD's we purchased are still hanging around. Part of this is simply figuring out what your actual write workload is. The 60GB and 120GB SSD's we bought a decade ago were estimated to last a few years until SSD prices dropped, that happened, they got replaced with 480-500GB SSD's for similar prices, but they didn't actually wear out. I ran across four 120's I gifted to a good cause just last week and they are still truckin'.
That's not to say enterprise SSD doesn't have value, sometimes it does, but also often it doesn't. Back in 2015 on Black Friday, the Intel 535 (Sandforce) 480GB SSD's were selling for, I wanna say about $160 each, and we bought a *bunch*. These have about 73TBW endurance, and I put them in roles where I knew we could burn through that in just two or three years, but I saw SSD price trends decreasing, and they had Intel's 5 year warranty, and the price for so-called "datacenter" SSD was several times that. I put SSD's in RAID1 with a spare, so outfitting a machine with 5 x 480TB SSD's was actually still cheaper than a single nonredundant "enterprise" SSD like a S3500 1.2TB ($1100) with its 275TBW.
We had some of the 535's burn up as expected, but a lot of them lasted longer. Over the last two years of their warranty, I think we turned in half a dozen to Intel, who sent us brand new 545s SSD's in exchange (bonus!). I just had another 535 lose it earlier this month, darn, I won't be getting an RMA replacement for it. :-(
So, the thing is, ZFS works better when you have more free space. If your endurance requirements can fit within what consumer SSD's can provide, it may not be such a good idea to spend money on enterprise SSD, especially "refurb" ones.
