SSD or HDD + SLOG

[clicks]

Hi,

I'm currently evaluating using FreeNAS as a centralized storage solution for virtualization (KVM/Proxmox).
Until now we used some FreeNAS boxes for plain backup purposes without any special performance needs but the new system should deliver some decent speed for about 40 server VMs.
As prices for SSD drop, even for enterprise grade SSDs like for example the PM853T, and their reliability reached good values it might be useless to invest in spinning discs anymore.
What do you think is more feasible to do to get a powerfull storage? Using multiple spinning discs in a mirror and a fast SLOG/L2ARC or a RaidZ (2 vdevs) of SSDs? I need about 5 TB of fast data storage?
Opions? Experiences?

(I already spoke to iXsystems about a TrueNAS system, but as they don't have a german subsidiary it isn't too easy)

 

[marbus90]

iXsystems Europe is coming in approx. 6-12 months however. Everything points to Germany as HQ. ;)

Note that the TrueNAS only supports SAS SSDs, which adds even more to the bill. About raidz I'd love to see some benches myself - until I did them myself I'll trust iXsystems and say "do striped mirrors". Sooo... if you're going to have 5TB of iSCSI VM storage, you'll need around 20TB raw according to the ZFS Primer, which states that with iSCSI only 50% of the volume should be used due to fragmentation issues. Probably avoidable trough going all-flash, even then I'd be more comfortable with ~12TB raw for striped mirrors and some headroom. Looking at the Micron M500DC you'll find that the 480GB variant is more optimized towards write-intensive application while the 800GB loses around 10k write IOPS and is hence more recommended for read-heavy application. I'd also still mix different vendors as SSD controller issues did bite me quite often.

Also I've seen that proxmox 3.4 supports ZFS RAID on local storage. Maybe that's worth a shot as well.

For the FreeNAS Hardware Design part there are imho 2 options:
- 2U 24x2.5" Server + 2U 24x2.5" JBOD
- 1 or 2U Server with 2x JBODs connected to it -> easier replacement of the headnode in case of failure

In any case that's Xeon E5 (or maybe Xeon D, but these lack the serious I/O connectivity of the E5's) range with at least 64GB RAM. The recommended Chelsio 10Gb SFP+ cards are available at sona.de IIRC. You might get around with Intel X520 cards, however for iXsystems there is no way around Chelsios in the TrueNAS.

 

[clicks]

Good to hear about a Europen branch of iXsystems - although it will be too late for my current project. Although I know that SAS is the preferred interface for discs I won't use them for the storage as they are far too expensive, especially the SAS SSDs. And I don't think the better error detection (and multipathing) are worth this price.

Thanks for your response and the hardware recommendations. Mixing SSD vendors is a good idea to distribute possible failure rates.

If I interpret your posting right you would prefer a flash solution over conventional HDDs as well. But 12TB raw storage for "just" 5TB usable space is a quite heavy overhead. Actually I was leaning more towards a Raidz with 2 vdevs of 4 960GB drives plus a hot-spare. I know it lowers the performance in contrast to a mirrored setup but I have to balance costs and effects. Doing a mirrored setup would only be affordable with HDDs.

 

[marbus90]

The TrueNAS is a dualcontroller system with high availability. To connect each disks to both controllers you'll need those SAS SSDs.

With 12TB raw and striped mirrors you'd get 6TB usable = 5.4TiB, with platter systems that shouldn't be filled above 90% in any case due to the performance/space optimisation... so, depending if you meant TB or TiB that's a close cut already (assuming striped mirrors). It wouldn't be that feasible to mix vendors in a raidz enviroment since you wouldn't gain much redundancy from that.
I'd rather do an 8disk raidz2... dual parity is quite important.
If you're not using lz4 compression currently, you may be saving quite a lot trough that. Deduplication might be worth a shot as well for a 5TB pool -> needs at least 128GB RAM. Maybe consider a HDD pool as well to move older snapshots to.

 

[cyberjock]

dedup doesn't work well for VMs, even when using the same base image. Things quickly deviate at the blocklevel and dedup becomes nothing but a performance killer for a very small gain.

I have yet to see anyone do VMs with RAIDZ1s. Everyone I've ever seen try to go with RAIDZ1s eventually bailed for mirrors. The performance penalty with the Z1 writes is pretty high, so unless your workload demands virtually zero disk usage all the time, you'll be forced to go mirrors or you'll start seeing the datastore drop out. And when that happens all your VMs are "powered off". You won't be happy.

So yeah, I can understand the need to balance cost versus benefits, but this is a battle you're virtually guaranteed to lose. There is no way to win. ZFS protects your data at any cost. To offset that kind of expectation you need to throw more hardware at it.

 

[HoneyBadger]

dedup doesn't work well for VMs, even when using the same base image. Things quickly deviate at the blocklevel and dedup becomes nothing but a performance killer for a very small gain.

Dedup works for one kind of VM - floating, non-persistent desktops in Horizon View.

Everything else is a bad idea for the reasons stated.

 

[clicks]

After having read mostly negative things about the native deduplication in ZFS in the last years I didn't consider it at all. Compression is fine an will definetly be used.

I completely agree with all of you that a mirrored setup is definetely preferable over Raidz from a straight performance point of view. I will have to do some tests to decide which setup is favorable. So the question still boils down to flash vs. cached spinners. Is there any argument for HDDs? Actually I can't see none, but perhaps I'm missing something.

 

[HoneyBadger]

Is there any argument for HDDs?

Capacity per dollar. I can't get 2TB of flash for $60.

 

[marbus90]

Well, you can achieve the same mass of storage far cheaper. Drop some 10-15krpm HDDs in there, one cheap (SAS maybe if you plan for TrueNAS migration) SSD for L2ARC and one ZeusRAM or other high-endurance SSD for a SLOG. Maybe you can drop a single Intel P3700 in there for SLOG and L2ARC combined. IIRC the IOPS of that card with a mix of 70/30 r/w were 150000 for the 400GB version. At that mass of IOPS with low latency it might be acceptable.