Short simple version: I'm setting up my FreeNAS build to handle general file serving and as a ESXi VM store. I got very lucky with the ZIL, which got upgraded to a top-end Intel P3700 datacenter SSD via RMA, and now I don't know how to make best use of that device. My hardware:
Even with overprovisioning and the potential demand from sequential sync snapshot writes by ESXi across the 10G link, I still can't see this server doing anything with a 400G ultrafast datacenter SSD that it wouldn't do just as well with 40GB or at most 1/4 of what it has. I don't know if the ESXi writes will saturate the SSD (and if they did, whether I'd notice if they were interspersed with a second task if other data was on the SSD). But even if they did, ESXi's highest speed writing is only "now and then", when it snapshots, so it might turn out to be tolerable.
In short, I'm itching to make better use of the 400GB Intel, as long as it doesn't much impact the intermittent use as ESXi ZIL (which I can test and find out for myself). I don't really want to have an entire 400GB SSD with 40GB effective use and 300GB plus whatever other NVRAM Intel overprovide effectively wasted (just gives it extra lifetime, but I'll never need that scale of lifetime, looking at Intel's stats on PB writes). But what other sensible uses can I make for it, on a file server? Can I use it for L2ARC as well as for ZIL? I suspect perhaps it might well be fast enough to handle both tasks, and if so how can it be set up that way? Or to act as ZIL for a second pool too - only I don't have a second pool needing ZIL at the moment?
* itch itch itch!! *
What additional uses could I try for such a high speed SSD, and how would I set it up for them?
- Baseboard/CPU/RAM - Supermicro X10SRi, E5-1620 v3 (3.5 GHz), 96 GB 2400 ECC ram. I can add another 32-64GB if needed but prefer to see how this performs first.
- PSU - EVG Supernova G2 850W (exceptional score for stability at jonnyguru.com)
- Main NICs - twin 10G chelsio low latency +Finasar transceivers (T420-LL-CR) directly connected to the VM server and my workstation. The baseboard also has a 1Gb Intel NIC for other connections. Network is 9k jumbo frame throughout, although this is less crucial these days.
- Boot drive is mirrored 2x Intel 320 40GB SSDs
- Main data pool - currently (2x6tb) + (2x6tb) + (2x4tb), i.e. 6 disks in mirrored pairs, for a total of 14.5tb usable space, and about 8-9tb of data to load initially; all disks are enterprise 7200's. I prefer mirrors to RAIDZ (that's decided), and will expand 2-way to 3-way when I can afford it. If the pool gets above about 70% full, I'll add extra pairs of mirrored disks "early" rather than let it fill up and forcing ZFS to get write-heavy on a single new vdev.
- ZIL - Intel P3700 400GB NVMe, reviews seem to show it's not only fast but maintains a low write latency better than almost any other drive on the market under continual loads, which is the main thing. I got lucky with this, I got it via RMA. It was the one I wanted but couldn't afford given other components in the build. It's not mirrored but that's a risk I have considered and accepted.
- L2ARC (tentative) - Since VM files can be tens of GB in size and I expect to frequently rollback VMs, the server could have ARC pressure issues or slowdown. I've got a Samsung 960 EVO 250GB NVMe available to trial as L2ARC once the rest is stable. It should be able to saturate a 10G link with ease, if the data is held. I'm wary of letting L2ARC get so big it's a net loss on RAM performance, as web pages often warn, but I think with proper tuning and limits set on the different RAM/L2ARC uses, it can help.
- UPS/backup - UPS to buy soon, not done yet but power is very stable historically, here. Old server will be used as a backup once data migrated.
- Anything else I've missed?
Even with overprovisioning and the potential demand from sequential sync snapshot writes by ESXi across the 10G link, I still can't see this server doing anything with a 400G ultrafast datacenter SSD that it wouldn't do just as well with 40GB or at most 1/4 of what it has. I don't know if the ESXi writes will saturate the SSD (and if they did, whether I'd notice if they were interspersed with a second task if other data was on the SSD). But even if they did, ESXi's highest speed writing is only "now and then", when it snapshots, so it might turn out to be tolerable.
In short, I'm itching to make better use of the 400GB Intel, as long as it doesn't much impact the intermittent use as ESXi ZIL (which I can test and find out for myself). I don't really want to have an entire 400GB SSD with 40GB effective use and 300GB plus whatever other NVRAM Intel overprovide effectively wasted (just gives it extra lifetime, but I'll never need that scale of lifetime, looking at Intel's stats on PB writes). But what other sensible uses can I make for it, on a file server? Can I use it for L2ARC as well as for ZIL? I suspect perhaps it might well be fast enough to handle both tasks, and if so how can it be set up that way? Or to act as ZIL for a second pool too - only I don't have a second pool needing ZIL at the moment?
* itch itch itch!! *
What additional uses could I try for such a high speed SSD, and how would I set it up for them?
Last edited:
:D
