With 18 HDD / SDD, which is the best configuration?

[WhiteTiger]

With 18 HDD/NVMe , which is the best configuration?
I have a server with 15x300GB HDDs, 1x250GB M.2 connected to a USB 2.0 stick and 2x480GB NVMe available on PCI express.
I can only boot from HDD or USB.
I would like advice for the best configuration.
I was thinking of using
2xHDDs for booting, in mirroring, partitioning them in order to make the best use of them.
1x250GB (over USB 2.0) for cache
2x480GB for higher performance accesses
I wanted to use the HDDs with RAID-Z3 for better security because the drives are not new.

The server has 2 Xeon 6 cores and 64GB RAM.

Thanks in advance for any advice.

 

[Etorix]

It all depends on your use case… Just storage? SMB, NFS, iSCI? Any jail/VM?

USB is not suitable for caching. This can only serve as boot drive.
Mirrored boot drives are not necessary if you keep a copy of the configuration and can cope with some downtime while replacing a failing boot drive.
64 GB RAM is not enough ARC to set up 2*480 GB as L2ARC. (Rule of thumb: L2ARC ≤ 5 * RAM)

 

[WhiteTiger]

It all depends on your use case… Just storage? SMB, NFS, iSCI? Any jail/VM?

Thanks for the reply.
I will use it for SMB for up to 4 users.
Maybe even VM, but only for testing.

USB is not suitable for caching. This can only serve as boot drive.

If I use the 250GB M2 disk to boot how do you recommend partitioning it?

64 GB RAM is not enough ARC to set up 2*480 GB as L2ARC. (Rule of thumb: L2ARC ≤ 5 * RAM)

I only have 64GB RAM, at the moment I am not expected to take any more.
How much memory did you think is needed?
Or use only 1 480GB NVMe, but then the other how do I use it?

What about the other 15 HDDs?
All 15 HDDs for RAID-Z3, including 3 parity disks?
Or 2x7 HDDs for RAID-Z3 and use the last 300GB disk for boot (to be partitioned). In this case either I use the 250GB one on the USB for other things or I unplug it and use it on the PC.

 

[Etorix]

Then SMB sharing from raidz3 is perfectly appropriate. VM, or any form of block storage, demand (a stripe of) mirrors over any form of raidz. If you get serious with VMs, they would absolutely thrive on a mirror of your 480 GB NVMe drives.
TrueNAS wants a whole drive for boot. Partitioning is unsupported.
At this point it is not apparent whether you would actually benefit from a L2ARC cache on top of 64 GB RAM.
15 drives in one vdev is a bit too wide; two vdevs is better, even if 7-wide is not very space efficient with Z3.

So I'd suggest:
2 * (7-wide raidZ3), 15th HDD as spare
boot on M.2 (or on the 15th disk HDD and reuse M.2 drive elsewhere)
2*480 GB NVMe as mirror for VM, or reused elsewhere

 

[WhiteTiger]

So I'd suggest:
2 * (7-wide raidZ3), 15th HDD as spare
boot on M.2 (or on the 15th disk HDD and reuse M.2 drive elsewhere)
2*480 GB NVMe as mirror for VM, or reused elsewhere

Many thanks