SSD partitions for L2ARC, SLOG, Metadata and Boot

[naskit]

I have:

• 8 x Seagate IronWolf NAS 4TB HDDs
• 2 x SAMSUNG PM893 960GB Enterprize SSDs
• 2 x SAMSUNG PM893 480GB Enterprize SSDs

I know partitioning drives is not normally recommended for TrueNAS, but to get the most out of this drive list, I am considering the following pool and partitioning schemes:

1. RAIDZ3 VDEV incorporating all 8 4TB Seagate HDDs for main data pool
2. The following SSD partitioning scheme:

I would love to get any thoughts on this from those who are more experienced in this.
The SSD partitioning is what I am really wanting advice on.
I know the criticality of maintaining the metadata and SLOG pools for integrity of the main data-pool, hence the 4-way mirrors.

 

[joeschmuck]

I would love to get any thoughts on this from those who are more experienced in this.

This is easy. Do not do it. While it can be done and I have done this to see if I could partition the drive to use some of the wasted space, it's not worth the headache. We tell people to purchase the boot drive based on price now, or I do, and not capacity. If you buy a 256GB SSD for $30 or could buy a 32GB SSD for $50, you buy the $30 drive. So long as you have 16GB (what I shoot for) then all is good. In your case you have two 1TB drives and I'd use them for some other project, or think of a boot drive as being really overprovisioned

If you do this, if you have a problem and you ask for help, most people are going to tell you that it's unsupported. If you are capable of doing this, you should be capable to troubleshoot as well.

There are so many threads on this, you have been a member since 2021, you should know to do a Google search on the topic and read those threads. There are some good explanations why not do this.

Lastly, it's not supported in TrueNAS. One day it may be supported as I would like to see it along with others, but not right now.

Why have a 4 way boot mirror? Stick with a single boot drive and backup your configuration data for easy restoration should you need to.

 

[naskit]

Thanks for your reply.

Just so I understand, as long as I have a config backup, it doesn't matter if I lose the boot drive as I just replace it then restore the config and all pools and configs will become available again (assuming everything else still works and is connected the same way)?

(This assumes the config is not on the NAS itself, it has to be elsewhere...).

I tried so hard to find smaller SSDs, but they are just impossible to find now for a reasonable price (I was trying to get SLC or MLC, not TLC or QLC - my research indicates SLC is the most robust). I could only find TLC at 480GB and 960GB sizes that were at acceptable prices. You can still buy SLC SSDs but they are now sold as being for 'industrial' applications and they are prohibitively expensive.

I needed a server grade mobo for ECC but it also needed to be an ITX to fit inside the SilverStone DS380, and the only one I could find with enough SATA ports was an ASRock Rack with 13 SATA ports. So I actually only have one more SATA available to me - 8xHDD plus 4xSSD = 12.

As for SLOG, I recognize that that is a potential SPOF and cause of data loss if I lose power during a flush to disk (SLOG-->HDD). I will be using a UPS, but I also wanted to protect against bit rot in the SLOG, hence the mirrors.

So, how about this instead (yes, boot is way overkill, but I have the drives now, so this is maybe the best compromise in terms of use):

Thoughts?

 

[ChrisRJ]

To more or less repeat what @joeschmuck wrote: This is the kind of setup where the rule "if you have to ask, you should absolutely not do it" applies.

As for SLOG, loss of external power is just one of several failure conditions that can lead to loss of data. So a UPS is not suitable to mitigate that risk. Besides, a UPS has a relatively high risk of failure as well, compared to other components. Please read up on SLOG (link in my signature under "Recommended readings").

For 8 HDDs I would consider a RAIDZ2 instead of RAID3.

You have not provided the use-case, which is by far the most important information for us, to give you proper recommendations.

 

[Arwen]

I would add this about SLOGs:

• Only certain uses benefit from a SLOG, like NFS or iSCSI, not SMB.
• Next, SLOGs should have PLP, Power Loss Protection, meaning on power loss, any existing data is not damaged. Very few SATA SSDs or NVMes have PLP.
• SLOGs don't actually have to be that big. In some cases, a 16GByte device is more than adequate.
• Last, SLOGs, (when needed), are a write mostly device, so they should be on high write endurance type SSDs or NVMes. Mis-calculating this when sharing with L2ARC / Cache, Boot device or Special Metadata can cause them to fail earlier than expected.

Basically, unless you KNOW you need a SLOG, you don't need a SLOG.

I added a SLOG to my old FreeNAS MIni because I initially used NFS. But, I ended up using RSync protocol for all my backups and simple SSH / SCP for other small things. Thus, a persistent metadata only L2ARC / Cache would have been a better choice.

 

[Constantin]

I have:

• 8 x Seagate IronWolf NAS 4TB HDDs
• 2 x SAMSUNG PM893 960GB Enterprize SSDs
• 2 x SAMSUNG PM893 480GB Enterprize SSDs

I would love to get any thoughts on this from those who are more experienced in this.

You’re asking for trouble. I get it, those boot and L2ARC SSDs may mostly operate as WORM drives, etc. But the underlying drives you want to use for all those applications will have to include something that can handle all use cases.

For SLOGs, that would be a PLP device like commercial-grade Intel Optane series. Some users here have successfully partitioned these $$$ platforms to do multiple duties, but they knew what they were doing, were ok with data loss, and had adequate backups. If that doesn’t describe you, expect to lose that pool and its contents.

Lastly, IIRC you will have to do all this work from the CLI. Use UUIDs to hopefully ensure that the TrueNAS application doesn’t later get confused about what partition was supposed to go with whatever dataset you propose. None of this is impossible but it requires careful work and documentation should you have to fix it in six months once the first SSD reaches its maximum design life and taps out.

 

[Etorix]

I tried so hard to find smaller SSDs, but they are just impossible to find now for a reasonable price (I was trying to get SLC or MLC, not TLC or QLC - my research indicates SLC is the most robust). I could only find TLC at 480GB and 960GB sizes that were at acceptable prices. You can still buy SLC SSDs but they are now sold as being for 'industrial' applications and they are prohibitively expensive.

For boot, just get the cheapest SSD you can find. QLC is fine. (I have hoarded a small stotck of 16 GB Optane M10 to use as boot drives. 9.99 E apiece. Technical opposite of QLC. Price is the qualifying feature.)

A 960 GB L2ARC requires at least 96 GB RAM. You have not explained the use case for a L2ARC.
Nor have you justified the use case for a SLOG. iSCSI? VM? Database? SMB sharing from the NAS will NOT use a SLOG.

 

[joeschmuck]

Just so I understand, as long as I have a config backup, it doesn't matter if I lose the boot drive as I just replace it then restore the config and all pools and configs will become available again (assuming everything else still works and is connected the same way)?

If you save the configuration and secret key file (an option during the config backup in the GUI) and as you said, the hardware is operational, then yes. Additionally you could relocate your drives in your pool(s) to other hardware and restore your backup, you will have access to your data. This is one of the benefits of TrueNAS.

Just a bit more information... If you have remote access (IPMI) to your server and do not have physical access, this is where mirrored boot drives come into play. A remote administrator can log into the IPMI and reconfigure the system to boot from the second mirrored drive. A mirrored boot drive does not mean automatic failover so you need to tell the motherboard the new boot order. The exception would be if the failed drive was not recognized at all by the hardware, but you are apt to have a drive with corrupt data or fails to work but is still recognized by the hardware then one that looks to be removed.

You have quite a bit of feedback and hopefully you understand that our group is trying to keep you from doing the wrong thing.

 

[naskit]

Thanks all. I will try to explain the thinking behind my initial (and subsequent) proposal more clearly:

1. I will never willingly buy a second hand disk for reasons that I assume are obvious to everyone here - I consider you all to be more expert than me with respect specifically to storage and disks
2. Why 480GB? 16, 32, 64, even 120GB SSDs in the 2.5" format are (AFAIK) no longer available as new
3. USB is not recommended for use as a TrueNAS boot drive, only for initial install and casual file transfer
4. I have 64GB ECC RAM (mobo max)
5. Why 960GB? Why not buy myself more NAND for wear leveling and SSD longevity for the L2ARC? There is nothing wrong with overprovisioning the L2ARC, plus, I get the distinct impression that apart from a longer boot time (loading files back into the L2ARC from rust) L2ARC seems to only be of benefit (according to Lucas and Jude) - having one can only improve performance. I don't plan on shutting this down regularly at all.
6. Why do I want a metadata pool? Because, as I understand it, metadata can dramatically speed up file browsing/searching and the user experience
7. But why did I originally propose those partitions?
   1. because I have only 13 SATA ports and no NVMe
   2. because I couldn't find smaller new SSDs (points 2 and 4 above) and I may as well make the most of them
   3. because having a 4-way mirror for metadata and SLOG - both of which will cause data loss if they are lost - is more robust than dedicating one SSD to metadata and one to SLOG, which would both mean a SPOF. Creating partitions and mirroring across the drives means that I can lose 3 of the 4 SSDs before my pool or files in the SLOG are actually lost. Yes, it is more complex, but at the same time, it is far more resistent to data loss.

After your initial comments, I suggested a much simpler layout.

• Only certain uses benefit from a SLOG, like NFS or iSCSI, not SMB.

I do plan to at least backup some databases to the NAS. I am guessing backing up database files to the NAS is very different from hosting the database on the NAS.

• Next, SLOGs should have PLP, Power Loss Protection, meaning on power loss, any existing data is not damaged. Very few SATA SSDs or NVMes have PLP.

Power Loss Protection - these SSDs do have it - I specifically choose Enterprise SSDs for the benefits they offer over retail.

I spent a very long time researching, comparing and choosing my SSDs.

• SLOGs don't actually have to be that big. In some cases, a 16GByte device is more than adequate.

As per my points 2 and 4 above.

• Last, SLOGs, (when needed), are a write mostly device, so they should be on high write endurance type SSDs or NVMes. Mis-calculating this when sharing with L2ARC / Cache, Boot device or Special Metadata can cause them to fail earlier than expected.

As mentioned, these are Enterprise SSDs, but I take your point about 'sharing them between SLOG and metadata and L2ARC'. I am thinking now that I may benefit more by having a metadata mirror instead of a SLOG if it comes to a choice.

Use cases intended:
- SMB file shares (most obvious)
- NFS file shares (as I move more to Linux-based systems)
- database storage/backup
- computer snapshot/backup
- vm backup

If you have remote access (IPMI) to your server and do not have physical access, this is where mirrored boot drives come into play. A remote administrator can log into the IPMI and reconfigure the system to boot from the second mirrored drive. A mirrored boot drive does not mean automatic failover so you need to tell the motherboard the new boot order. The exception would be if the failed drive was not recognized at all by the hardware, but you are apt to have a drive with corrupt data or fails to work but is still recognized by the hardware then one that looks to be removed.

You have quite a bit of feedback and hopefully you understand that our group is trying to keep you from doing the wrong thing.

Yes, the mobo has BMC/IPMI. I figured mirroring the boot, SLOG and metadata partitions should in theory increase the robustness and I could plan for disk replacement if/when one failed. A 4-way mirror gives me 3 'spares' for backup. I didn't know boot would not automatically failover to another disk in the mirror - thanks for the tip.

You have quite a bit of feedback and hopefully you understand that our group is trying to keep you from doing the wrong thing.

Yes, and thank you all very much for it. I will ponder all your valuable input. And I know that whatever I choose to do, I must accept the decision.
I was also wanting to increase the number of 'copies' for each of those partitions to give ZFS the best chance possible at finding a good copy of each file. Especially for the metadata and SLOG - though I need to check in my FeeBSD Mastery ZFS books whether I can apply that setting to those specific partition/storage types.

 

[naskit]

You’re asking for trouble. I get it, those boot and L2ARC SSDs may mostly operate as WORM drives, etc. But the underlying drives you want to use for all those applications will have to include something that can handle all use cases.

For SLOGs, that would be a PLP device like commercial-grade Intel Optane series. Some users here have successfully partitioned these $$$ platforms to do multiple duties, but they knew what they were doing, were ok with data loss, and had adequate backups. If that doesn’t describe you, expect to lose that pool and its contents.

Lastly, IIRC you will have to do all this work from the CLI. Use UUIDs to hopefully ensure that the TrueNAS application doesn’t later get confused about what partition was supposed to go with whatever dataset you propose. None of this is impossible but it requires careful work and documentation should you have to fix it in six months once the first SSD reaches its maximum design life and taps out.

Thank you for your thoughts. Yes, they are PLP as per my previous post. These have a 5 year warranty and being Enterprise are designed with spare NAND to take over when cells die (haha, get it? When NAND cells die? Pun unintended...)

Been in networking and firewalls for over 2 decades so CLI and UUIDs are a no brainer for me. I label everything and even have pix of all drive labels with serial and model, because I know how important it is to know exactly which PHY drive is where and to which SATA mobo port each is connected.

Am am weighing up everything you guys have said here with what I am reading in the Jude/Lucas ZFS books too.

 

[Jamberry]

Hey @naskit! I have been down that path.
Same thinking, same reasons as you (13Sata), and so on.
The little bit frustrating result is that there seems to be no technical reason why partitions should not work. It is just a support thing.
This is more a missing feature request than a technological barrier
You could even open a feature request, unfortunately, due to the enterprise nature of TrueNAS I would not expect iXsystems or OpenZFS to change this.

 

[morganL]

Thank you for your thoughts. Yes, they are PLP as per my previous post. These have a 5 year warranty and being Enterprise are designed with spare NAND to take over when cells die (haha, get it? When NAND cells die? Pun unintended...)

Been in networking and firewalls for over 2 decades so CLI and UUIDs are a no brainer for me. I label everything and even have pix of all drive labels with serial and model, because I know how important it is to know exactly which PHY drive is where and to which SATA mobo port each is connected.

Am am weighing up everything you guys have said here with what I am reading in the Jude/Lucas ZFS books too.

Thanks for the ideas and write up. I think the logic is right and it is something we (iX) have as a potential future roadmap item. Would love to have a real system as validation.

Since we haven't done it yet, you would be a brave pioneer. None of this has been tested and you may uncover some bugs. I'd recommend you make sure you backup any valuable data to another system or cloud (e.g ix_Storj).

The quality of the drives is important.. Enterprise grade. You need to make sure the endurance is there and that there are enough spare cells. You may want to "under provision" partition capacity... some people call it "overprovision".

I'd generally recommend 2-way mirrors and potentially a spare. Its less work for the system and better tested code path. Its not clear that the SLOG has to be mirrored... since there is a copy of SLOG data in DRAM.

Boot, SLOG, L2ARC as partitions make sense. I'm a little more skeptical that metadata vdevs should be added as partitions (especially as a pioneer) since that is hardest to recover. A larger persistent L2ARC reduces some of the need for metadata vdev. Obviously, you'll have to manually partition the drives.

Plan for success, but prepare for failure... and then keep a journal that you can publish and we can learn from. Good luck!

 

[Arwen]

@naskit - Your comment;

Why 960GB? Why not buy myself more NAND for wear leveling and SSD longevity for the L2ARC? There is nothing wrong with overprovisioning the L2ARC, plus, I get the distinct impression that apart from a longer boot time (loading files back into the L2ARC from rust) L2ARC seems to only be of benefit (according to Lucas and Jude) - having one can only improve performance. I don't plan on shutting this down regularly at all.

Over provisioning is not necessarily accomplshed by having a larger device. ZFS would attempt to use all of such a device, like 960GB for L2ARC. Thus, not really making it more reliable.

As far as I know, the only real over provisioning is to use the SATA Host Protected Area. This is from the manual page of hdparm:

-N Get/set max visible number of sectors, also known as the Host Protected Area setting. Without a
parameter, -N displays the current setting, which is reported as two values: the first gives the
current max sectors setting, and the second shows the native (real) hardware limit for the disk.
The difference between these two values indicates how many sectors of the disk are currently hidden
from the operating system, in the form of a Host Protected Area (HPA).

Next, the maximum size of a L2ARC device is generally 5 times the size of RAM. Some say up to 10 times is okay. With 64GB of RAM, that suggests a maximum of 640GB L2ARC, (if using 10 times for maximum). Going way too large simply means you are using RAM for L2ARC pointers and not ARC, (aka first level read cache).

 

[Davvo]

I would never use a drive with multiple partitions, the performance penalty makes it a no go for me. This is relevant for everything, but especially with syncwrites since they are already slow... why should we make them even slower?

Regarding L2ARC, it's referenced in the ARC so too much of it with too few RAM can harm your performance instead; making it permanent is your choice since it's not standard behaviour (you just have to set a tunable).

Regarding metadata VDEVs, you want to have the same level of redundancy as the rest of your pool in order not to create a weak point in the system (usually a 3-way mirror since most users go with a RAIDZ2 layout). If you want to improve your read performance, L2ARC alone will likely suffice; if you want to improve your small file writes, you have to carefully plan the space and the file's size you have to work with.
Please read the documentation about fusion pools.

For the boot drive you can just use an internal USB with either a quality USB stick or a NVMe to USB adapter; either way, if you don't place the system's dataset in the boot drive it won't see much activity. Doing so leaves you with one more SATA port available and you can easily service it when needed; for the config backup you can easily get it manually or with scripts (like the multi_report from joe, which does simplify your system's health monitoring by a LOT): I get a weekly backup, never had to use it so far.

With 64GB of RAM I would go with a 500 GB SSD for L2ARC, possibly metadata-oriented or reserved (you can tune how ARC and L2ARC handle different types of data), and if required by your use-case a couple of SLOG SSDs in mirror (the bigger the size the more you can do underprovision / overprovision work with them)... which would leave you with 8 ports reserved for HDDs and 3 for SSDs, with one left for your boot drive in case you decide not to go with an internal USB; if you also require a metadata VDEV thus making a fusion pool, I fear you have to either: use an HBA in IT mode, use a PCIe m.2 card and use m.2 nvme for SLOG (better) or metadata VDEV (less ideal, you want the SLOG to be as low-latency as possible), use a different Motherboard, or do partitioning black magic. Do note that the PCIe m.2 card requires bifurcation, but almost all server grade motherboards have such a feature (just check wheter it is x4x4x4x4 or a different mode and move accordingly).

TL;DR: Imho it's not worth the hassle given the alternatives, but it has worth in the "let's see what happens" category if you are willing to cope with performance penalties.

Edited for grammar corrections.​

 

[morganL]

@naskit - Your comment;

Over provisioning is not necessarily accomplshed by having a larger device. ZFS would attempt to use all of such a device, like 960GB for L2ARC. Thus, not really making it more reliable.

You csn format/partition the drive so that L2ARC does not have access to the full drive capacity. For example, add an idle partition.

 

[morganL]

I would never use a drive with multiple partitions, the performance penalty makes it a no go for me. This is relevant for everything, but especially with syncwrites since they are already slow... why should we make them even slower?

​

You certainly have to design the system to meet a performance goal. Sometime those goals are quite modest and then a single or multiple SSDs might work. As SSDs get faster and larger, there are more use cases. NVMe gen5 SSDs are very fast.. 10GB/s... and their minimum size is getting quite large e.g 2-4TB.

 

[Constantin]

i agree that making the best use of various SSD resources is attractive, especially in SOHO systems with a low quantity of drives. In my system there are two SATADOMS (boot), one L2ARC (non-metadata), and one SLOG (optane) plus three SVDEV drives. That’s a lot of ports and wouldn’t it be nice if four drives could do the work of 7?

Given how much improvement potential there is re: sVDEV implementation and/or maintenance, I expect it will be a long while before we see the GUI develop the kind flexibility that the OP envisions re: SSD multi-duty. It doesn’t seem to be a development priority ATM given how long sVDEV has been around vs. what iXsystems has done in the GUI for sVDEV.

Don’t get me wrong, I appreciate being able to run a fusion pool because it does make a significant difference re: backup speeds, browsing, and so on. But getting it set up takes the CLI, ditto periodic reviews to see if the sVDEV is still empty enough to hold all the small files and metadata. It’s pretty shocking to me that something as important as sVDEV drive fill is not accessible from the GUI.

This sVDEV data is easily accessible by the CLI, just no GUI dashboard, or line item in the GUI menu, etc. iXsystems does it for pool fill… so the omission speaks loudly re priorities.

Among other issues, bundling SSD duties would require a significant rethink re: GUI reporting, ie the ability to gauge how well the SSD pool is handling all the duties a user is throwing at it (sort of how we can monitor ARC, etc performance).

This would be a non-trivial lift just for the GUI, never mind allowing the user to partition drives for multiple duties. Combining the boot pool with L2ARC could be a natural for systems with a lot of WORM data, for example.

 

[Etorix]

1/ I will never willingly buy a second hand disk for reasons that I assume are obvious to everyone here - I consider you all to be more expert than me with respect specifically to storage and disks

Which "obvious reason"? I indeed prefer to have new HDDs, but have no qualms about second-hand entrerprise-grade SSDs, second-hand Optane (SLOG/L2ARC) and second-hand anything for boot drive.

5/ Why 960GB? Why not buy myself more NAND for wear leveling and SSD longevity for the L2ARC? There is nothing wrong with overprovisioning the L2ARC, plus, I get the distinct impression that apart from a longer boot time (loading files back into the L2ARC from rust) L2ARC seems to only be of benefit (according to Lucas and Jude) - having one can only improve performance. I don't plan on shutting this down regularly at all.

Nothing wrong with overprovisioning for longevity indeed. But there's no "loading into L2ARC" at boot: It starts empty, and gets filled as time goes—or, if persistent, it's already filled.
L2ARC management uses RAM, so L2ARC can be detrimental: Too much L2ARC can degrade performance by evicting ARC.

6/ Why do I want a metadata pool? Because, as I understand it, metadata can dramatically speed up file browsing/searching and the user experience

As discussed many times in the forum, special vdevs are potentially risky because they are pool-critical.
If mostly read benefits (fast browsing) are sought, then a persistent L2ARC for metadata can safely provide the same result with a single drive. This does not speed up writes (which go to the pool), but no redundancy is required.

7/ But why did I originally propose those partitions?

1. because I have only 13 SATA ports and no NVMe
2. because I couldn't find smaller new SSDs (points 2 and 4 above) and I may as well make the most of them
3. because having a 4-way mirror for metadata and SLOG - both of which will cause data loss if they are lost - is more robust than dedicating one SSD to metadata and one to SLOG, which would both mean a SPOF. Creating partitions and mirroring across the drives means that I can lose 3 of the 4 SSDs before my pool or files in the SLOG are actually lost. Yes, it is more complex, but at the same time, it is far more resistent to data loss.

Different uses have different requirements.

• SLOG (if useful at all) requires PLP, low write latency and high endurance; read performance is irrelevant (it's never read back under normal circumstances). Redundancy is not really necessary for home-lab use: If the SLOG fails while in use (without taking down the whole server with it), ZFS will revert to the slower ZIL (lose performance, not data). If the server crashes, the SLOG will be read and no data shall be lost (the risk of an URE over a few GB worth of data is truly negligible). Data (up to two transaction groups) will be lost ONLY if the server has an unclean shotdown and then the SLOG fails upon reboot; for business-critical uses, one may want to guard against this fringe case, but for a home lab?
• L2ARC requires low read latency and endurance; PLP not needed, redundancy not needed.
• Special vdev requires redundancy, redundancy and redundancy.
• Boot has no special requirement at all.

Optane can serve as both SLOG and L2ARC, and actually has the mixed read/write performance to handle both duties simultaneously.
Mixing special vdev with anything else is dangerous: With your first layout, high writes from SLOG and/or L2ARC duties could wear out the drives, whose failure would then take down the pool-critical spcial vdev. As mirrorred drives would all wear down at the same rate, the complete SLOG+L2ARC+special vdev may well prove a giant single point of failure… and SSDs tend to fail abruptly without any advance warning.
For the sake of convenience (drive replacement, migrating a data pool wholesale to another server, etc.), I would not mix boot with anything else. Just throw at it a $10 drive of the kind that is NOT used by the pool (i.e. NVMe boot for a HDD pool / SATA boot for a NVMe pool). (If I were using SCALE and its apps, I suppose that boot+app pool could be acceptable: No redundancy on the app pool but replicate to a safer storage pool, or save the settings; if the single drive fails, reinstall everything.)

Use cases intended:
- SMB file shares (most obvious)
- NFS file shares (as I move more to Linux-based systems)
- database storage/backup
- computer snapshot/backup
- vm backup

As opposed to live database/VM, backups want async. NFS would default to sync writes, but for regular file sharing it may be acceptable to either disable sync writes or to just live with "regular" sync writes. I don't think you have a strong use case for a SLOG.

 

[Volts]

Change the data drive layout from Z3 to Stripe+Mirror. Much better actual performance w/ 8 drives. Not much difference in capacity when considering overhead and Z3 slop.

Don't use slog unless you have critical sync writes. Backups don't need sync. And sync=disabled is faster still.

Don't use a meta vdev with a complex partitioning scheme because it's critical for the data. If you lose the meta vdev you lose the data. If you want to recover the data in another machine, it's WAY easier to just take the data disks.

Don't try to use a big data L2ARC, probably, unless you have huge amounts of RAM. It's not worth sacrificing primary ARC.

The goofy partition and L2ARC hack that I do endorse is a metadata L2ARC, secondarycache=metadata, vfs.zfs.l2arc.rebuild_enabled=1. It sure helps keep things snappy like a meta vdev, and especially after a reboot. But it can be disconnected without impact.

 

[Davvo]

Change the data drive layout from Z3 to Stripe+Mirror. Much better actual performance w/ 8 drives. Not much difference in capacity when considering overhead and Z3 slop.

It depends on what he's after, striped mirrors don't give you the same resiliency.