Recommendation (and why) RAID-Z2 configuration 12 disk + 4 disk PCIe + 2 disk SSD

[jjonsson]

I have gotten my hands on an Oracle ZFS 7120 Appliance. All disks are wiped, no original OS on it. So I have installed TrueNAS on it, and it seems to work.

Now I wonder how I get the most performace/free space out of it, together with the safety of my data. I want to use RAID-Z2.

So how do I configure the disks/volumes ?

The server is equipped with:

12 x SAS 7200 RPM 2 TB disk on separate controller
4 x 24 Gb disk on Sun 541-3731 XTA-FAS-S3IE96GB 96GB PCIe Flash Accelerator F20 SAS HBA
2 x SATA SSD 120 Gb disk on MB SATA controller

The server has 48 GB RAM.

I can read that it is recommended to install TrueNAS on a SSD type disk, not USB (The server has internal USB key slot)

So, how do I get most performance and storage out of it ?

Should I use the 2 x SSD for storage (ZIL, L2ARC whatever it is called) or install TrueNAS on the as OS boot disk ?
Or should I use 1 or 2 x 24 Gb drives on the PCIe Flash card for TrueNAS OS boot drive ?

Or maybe something else ?
The primary use will be as VMWare ESXi shared storage. I will install a dual FC 8 Gbit/s HBA adapter in the server.

Any help is appreciated

 

[sretalla]

So, how do I get most performance and storage out of it ?

First you need to explain or at least understand what your workload needs/means in terms of "performance"... IOPS or Throughput. See below.

Should I use the 2 x SSD for storage (ZIL, L2ARC whatever it is called) or install TrueNAS on the as OS boot disk ?

If you don't even know which you want, you may not even want either.

I vote for Install the OS here.

Or should I use 1 or 2 x 24 Gb drives on the PCIe Flash card for TrueNAS OS boot drive ?

Maybe an OK option too, but those may serve you better in other roles depending on what you want the system to do. Maybe as a metadata special VDEV. (relatively new concept, search the forums for discussion on it)

The primary use will be as VMWare ESXi shared storage. I will install a dual FC 8 Gbit/s HBA adapter in the server.

OK, so it's IOPS that you need... you'll be wanting mirrors and possibly (depending on how fast and reliable/power-safe your SSDs are) a SLOG.

Clearly mirrors do not meet what I'm hearing as (one of?) your main desire... maximize available storage, but you will not be happy with performance if you don't give the IOPS needed for block storage... read this:

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

I'll leave it to you to do some searching in the forums about SLOG, but this article (later pages) explains some important parts of what it does... https://arstechnica.com/information...01-understanding-zfs-storage-and-performance/

I don't confirm that your SSDs are suitable for use as SLOG, but you wil probably want to think about one due to ESXi and it's hunger for Sync writes.

I have not seen any discussion on using that type of HBA, so research that carefully and see if you see any examples of people reporting success with them... if not, I would advise finding a better, more standard option likel Chelsio or Mellanox 10G ethernet.

 

[jjonsson]

I have not seen any discussion on using that type of HBA, so research that carefully and see if you see any examples of people reporting success with them... if not, I would advise finding a better, more standard option likel Chelsio or Mellanox 10G ethernet.

This servers main purpose has been running as ZFS storage. It's an Oracle ZFS Appliance. So the purpose of the HBA (I'm assuming you mean the Sun 541-3731 XTA-FAS-S3IE96GB 96GB PCIe Flash Accelerator F20 SAS HBA) is something ZFS related.

 

[jgreco]

Be aware that Sun's hardware was designed for Sun systems, and often isn't well-supported or supported at all under FreeBSD and Linux. Expect that you might need to replace HBA and ethernets. FC isn't supported.

 

[sretalla]

I will install a dual FC 8 Gbit/s HBA adapter in the server.

I was talking about this "HBA". (as I had indicated by quoting that part of your statement above where I mentioned it).