Hybrid flash storage combines solid-state drives (SSDs) and hard disk drives (HDDs) in one storage design, so fast flash serves your active data and lower-cost disk holds the rest. If all-flash pricing is the sticking point, hybrid is the design to price it against. Flash prices have risen by more than 300%, and our V160 announcement puts the raw cost gap between flash and HDD at 20X.
Hybrid designs use caching, explicit data placement, or tiering, which are different mechanisms that vary by product, and the right mix depends on your data rather than a rule of thumb. Four questions drive the decision. How much of your data is active at any one time? How do your reads and writes behave? What happens to latency when a read misses flash? How much capacity do you need to grow into?
The sections below turn those answers into a configuration, showing how TrueNAS, an OpenZFS-based enterprise storage platform for file, block, and S3-compatible object storage, uses flash and disk, and covers when all-flash is truly the better call. Decision makers can focus on the comparison table, the customer data point, and the all-flash section. The sizing and component sections are written for your storage architects.
What Hybrid Flash Storage Means
Vendors use “hybrid” for multiple different mechanisms, and a quick way to cut through the marketing is to ask which one you’re being quoted.
- Caching. Caching keeps a second copy of data that lives permanently on disk in another tier of storage, like flash or non-volatile memory. A read that misses the cache is served from disk.
- Explicit placement. Specific kinds of data are written to flash and live only there. Plan this one carefully. If that flash fills up or fails, you have a capacity or availability problem, not just a performance one.
- Scheduled tiering. You set a policy, and data moves between flash and disk on a schedule you control.
- Automated tiering. The system watches how data is accessed and moves it between flash and disk over time. How quickly it reacts depends on the product.
Hybrid cloud storage is typically sold as a separate product, because of its ties to remote storage, either in a public or private cloud system; sometimes it’s sold as an additional tier on top of a hybrid flash configuration, but it’s outside the scope of this article. Hybrid flash storage is closer to the “hybrid drives” popular in laptop computers, which combine a spinning disk with a flash buffer.
How do Hybrid Flash storage systems compare for organizations that need both speed and capacity?
Hybrid flash storage systems compare against all-flash systems on latency consistency, sustained throughput and IOPS, usable capacity after redundancy, and total cost over the planned lifecycle. In general, hybrid designs cost significantly less per terabyte of total capacity, while all-flash storage gives more consistent latency because every read comes from flash.
The right choice depends on your workload pattern, its active working-set size, and whether disk-read latency fits your workload’s target.
Whichever candidates you’re weighing in a hybrid vs all-flash comparison, run each one under the same conditions, with the same dataset, workload, protocol, and client count. Then fill in your own numbers.
| Dimension | What to measure | Hybrid-specific question |
|---|---|---|
| Latency consistency | Median and tail latency across a full workload cycle | How far does latency rise when reads miss flash and hit disk? |
| Throughput and IOPS | Sustained results at your block size and read/write mix, not burst | Do results hold after the working set changes? |
| Hybrid Responsiveness | Latency and throughput of an existing workload when a new workload is added. | Can hybrid policies be tuned manually, or can it only be set array-wide? |
| Capacity | Raw versus usable after redundancy, plus growth over the lifecycle | How much flash does the active data need versus the total? Can you change the ratio later? |
| Power and rack space | Per usable terabyte at your target performance | How do drive bays and enclosures change with the media mix? |
| Recovery and rebuild | Rebuild time and performance during it | Can the hybrid tiers offer identical redundancy? Is there a performance impact if a drive is lost? |
| Total cost | Media, support, power, and refresh over the planned lifecycle | What does adding or changing flash later cost? |
How to size a configuration
Start with measurements, because a flash-to-disk ratio that fits one workload will miss another. The four questions above expand into six measurements.
- Read/write mix and access pattern. Record random versus sequential, block sizes, protocols used, and client count.
- Active working set over time. Measure across daily, weekly, and period-end cycles. The working set at quarter-end may look nothing like a typical Tuesday.
- Latency and throughput targets under sustained load. Define the target at the application level, and test under load rather than at idle.
- Cache behavior, including misses. Find out what a read costs when the data isn’t in any of the available hybrid tiers. Record both average and peak values when recording latency and bandwidth.
- Special vdev redundancy. Provide the same level of redundancy for your special vdev as you do your pool – with the upcoming TrueNAS 27, you can deploy RAIDZ based special vdevs for efficient fault tolerance across large flash tiers.
- Capacity and lifecycle. Compare raw versus usable capacity, expected growth, support, power, and refresh costs.
Workload-fit table
Think of this table as a measurement plan you can take back to your team. Each row lists what to collect before you decide.
| Workload | What to measure | Evidence needed |
|---|---|---|
| VMs and VDI | Boot and login storms, random read/write mix, synchronous write rate | Latency during peak events, ARC hit ratio, synchronous write volume |
| Transactional databases | Commit latency, synchronous write throughput, IOPS at your block size | Commit latency under peak load, database write throughput |
| Analytics | Scan size compared with cache size, sequential throughput | Throughput on a cold run, and whether scans exceed cache |
| File shares | Share of metadata operations, small-file counts, concurrent users | Directory-listing and open latency at peak, metadata footprint |
| Backup targets | Ingest throughput, restore speed, retention growth | Sustained writes across the backup window, restore time for a representative set |
| Video and surveillance | Sustained writes per stream, stream count, retention | Writes at full stream count, playback during recording |
| AI datasets | Whether training passes re-read the same data, dataset size versus cache, small-file counts | Cache hit behavior across repeated passes (see Choosing Storage for AI Workloads. File, Block, or Object?) |
Inputs to collect before you compare
Collect these inputs for your workload. Each one should come from a measurement.
| Input | Where to get it |
|---|---|
| Active working set across a full cycle | Daily, weekly, and period-end measurements |
| Read/write mix and block sizes | Application or protocol statistics |
| Share of synchronous writes | Workload profiling |
| Latency and throughput targets under load | Application owners |
| Cache hit and miss ratios | arcstat (live) and arc_summary (historical) on a test system |
| Hybrid versus all-flash results on the same test | Matched-condition runs on your own data |
Run the same test on each candidate configuration, then compare the results using the table above.
A customer data point. By switching to a TrueNAS hybrid architecture and native protocols, a leading Manhattan neural data lab ingested 850 TB/day at 11 GB/s, saving over 90% compared to competing all-flash quotes.
How TrueNAS and OpenZFS use flash and disks
TrueNAS is built on OpenZFS, which gives you separate components that each do one job. You get to choose which job flash does for your workload, which means you can spend on flash only where it pays off.
- ARC. The Adaptive Replacement Cache (ARC) is a cache in system memory (RAM) that holds the pool data accessed most frequently and most recently, so it serves reads first.
- L2ARC. Level 2 ARC (L2ARC) extends ARC onto dedicated SSDs, so reads come from flash instead of your pool’s slower data disks. L2ARC is read-only and non-critical, so if a cache drive fails here, TrueNAS reads from the pool instead. One thing to watch is that each cached block needs a small header in ARC, so on a memory-constrained system, adding too much L2ARC can make ARC less effective. How much it helps depends on your workload and how much memory you have, and our L2ARC guide shows you how to check.
- SLOG. A “Separate Log” device for synchronous writes. Sometimes called “ZFS write cache” – even though it isn’t – it benefits workloads that demand synchronous writes, like transactional databases, NFS exports, and virtualization workloads. Workloads that don’t demand synchronous writes bypass this entirely.
- Special vdev (Fusion Pool). A special vdev is an allocation class that stores metadata, the indirect blocks of user data, and deduplication tables, and it can be set up to accept small file blocks too. TrueNAS calls pools built with one Fusion Pools. It isn’t a cache, because the blocks routed there exist only there. While special vdevs currently require a mirror layout, which uses half of the raw SSD capacity for redundancy, TrueNAS 27 adds RAIDZ support. Since losing a special vdev loses the pool, give it at least the redundancy of your data vdevs. If your pool has RAIDZ data vdevs, you can’t remove a special vdev once you’ve added it, so it’s worth sizing with care. On the upside, a special vdev can cut the average number of spinning-disk I/Os needed to find and access a file by up to half. The Fusion Pools documentation covers the setup.
TrueNAS 27 also brings first-generation dataset tiering to Enterprise systems. You set a policy, and data moves between a flash and disk tier within a single pool, live, with checksums and open file access maintained. Dataset tiering combines with the other data acceleration methods of OpenZFS, so ARC and L2ARC keep delivering cached reads, while SLOG still protects and accelerates synchronous writes. The TrueNAS Tech Talk (T3) podcast walked through the idea in Episode 56 and Episode 62.
ARC and L2ARC fill based on access patterns, the special vdev takes the block types and sizes you configure, dataset tiering follows the policy you set, and SLOG covers synchronous writes only. Because each solves a different problem, you can buy flash for the problem you actually have.
When All-Flash is the Better Fit
Your measurements may point away from hybrid storage as the solution. You should consider all-flash in these cases:
- Your active working set is close to, or larger than, what you can practically cache, so misses are regular occurrences
- The application can’t tolerate the latency variance of reads served from disk
- Total capacity is small enough that the media cost difference doesn’t change your budget
If your results point to all-flash for one workload and hybrid for another, ask any vendor whether both run from the same management interface and data services. TrueNAS runs the same operating system, management interface, and data services across all-flash, hybrid, and high-capacity configurations. A smaller all-flash pool and a larger disk-based hybrid can even coexist on the same system, providing the consistent performance for a small demanding workload while leveraging the hybrid benefits elsewhere. Reach out to the TrueNAS team to see which of our line of versatile appliances fits your needs.
If you’re still comparing platforms, our guide to what to look for when choosing an enterprise storage appliance is a good place to start. And if you’d like the market backstory on why hybrid is back in the conversation, see why hybrid storage is hot again or hear the team talk it through on our T3 podcast.
FAQ
What is the difference between hybrid flash and all-flash storage?
Hybrid flash storage mixes SSDs and HDDs, while all-flash storage uses only SSDs. In an SSD vs HDD comparison, flash gives you faster response and disks give you more capacity for the money, which is why hybrid designs use both. Our V160 announcement puts the cost difference between flash and HDD raw storage at 20X. All-flash keeps every read on flash, so latency is more uniform.
How do hybrid flash storage systems compare for organizations that need both speed and capacity?
Hybrid flash storage systems compare against all-flash systems on latency consistency, sustained throughput and IOPS, usable capacity after redundancy, and total cost over the planned lifecycle. Caching-based designs tend to help repeated reads, explicit placement helps whatever you put on flash, and tiering helps data with a predictable lifecycle. The most reliable comparison is one you run yourself, on your own data, under the same conditions.
How is caching different from tiering?
Caching keeps a copy of data in flash while the permanent copy stays on disk. Tiering moves data between flash and disk, so each block lives in one place at a time. In OpenZFS, ARC and L2ARC are “cache” and the special vdev is a “tier.” TrueNAS 27 adds scheduled dataset tiering on Enterprise systems.
How much flash does a workload need?
It depends on your active working set, not your total capacity. Measure the working set across daily, weekly, and period-end cycles, then test candidate configurations against your own data.
What happens when the active working set does not fit in cache?
Reads that miss cache are served from disk, so their latency rises toward disk speed. Check your hit and miss ratios, then decide whether more memory, L2ARC, a special vdev for metadata, or more flash overall is the right fix. Keep in mind that L2ARC uses some ARC memory.
Which TrueNAS Enterprise appliances support hybrid configurations?
All TrueNAS appliances can be configured as hybrid or all-flash, but some are better suited for specific workloads than others. Check out the platform listings on the TrueNAS Enterprise page for an overview of each.
When is all-flash a better fit?
All-flash tends to fit when the active working set approaches or exceeds practical cache size, when the application can’t tolerate disk-read latency variance, or when total capacity is small enough that media cost stops mattering.
Nobody should have to take a storage vendor’s word for it, ours included. Using the free TrueNAS Community Edition, build the layout your measurements point to, and test it on your own workload.
What is an all-flash array?
An all-flash array is a storage system that uses only flash media for persistent storage, in place of the spinning hard disk drives found in traditional arrays. That means SSDs only, including NVMe drives. Because everything runs on flash, you generally get lower latency and higher IOPS than with mechanical drives, along with a smaller footprint and lower power use. The trade-off is cost, since flash is significantly more expensive per GB than disk, so it’s worth weighing against a hybrid design for data that isn’t latency-sensitive.
What is storage tiering?
Storage tiering is a way of organizing data into layers, or tiers, based on its importance and how often it’s accessed. Critical, frequently used data sits on high-performance storage such as SSDs, and less critical data sits on more economical media such as hard drives, which keeps your premium capacity for the work that needs it. Data moves between tiers either dynamically or manually over time. Tiering also differs from caching. Tiering permanently moves data between storage layers, while caching creates temporary copies of active data.