New pool configuration

[smcclos]

I am setting up a new pool and wondering what the best way to set it up. I have 6 SATA 10 TB Drives, and wondering if it would be better to have 1 VDEV Z1 with all the disks, or 2 VDEV, Z1 of 3 disks each.

I plan to build both and real world test, but was curious if others have gone down that this path before me.

I know that I would have 50TB vs 40 TB, I am just curious about performance between these two configurations.

 

[Etorix]

With drives of this size, the advisable configuration is 6-wide raidz2.

 

[winnielinnie]

1 vdev

6-way mirror

Take no chances



Just kidding. RAIDZ2 with all disks makes sense.

But if you want performance (instead of sheer capacity and two-disk failure redundancy), you can do 2-way mirrors, three vdevs total. (This will yield only 30 TiB usable capacity, and each vdev can only tolerate a single-drive failure, but you will reap better performance.)

 

[smcclos]

Thanks for the suggestions, the drives will be arriving in a couple of days, and will try that.

 

[winnielinnie]

and will try that

...the 6-way mirror?

 

[smcclos]

The 6 drive Z2 configuration.

 

[ChrisRJ]

What is your use-case? RAIDZ2 is great for "archival" purposes, where there is no demand for IOPS. But if you want something fast, then mirror might be more appropriate.

 

[smcclos]

Not 100% sure yet. My current NAS, NAS1 is a physical machine, but something with the S5572 motherboard will not work with TrueNAS 13.0, it has to stay at 12.0.

So I decided to install a LSI SAS9206-16e into my ESXi host and create a new NAS as a Guest. So first thought is to use this as my primary NAS, and location for my VMware datastores, so throughput is a concern, but since it is a homelab as long as it is "responsive" I don't care about IOPS.

Know 'Responsive" is a qualitative word, but as long as my benchmark testing between the established NAS, and my new build, I really don't care.

 

[HoneyBadger]

Much like some people are happy with a 320kbps MP3, and others insist that they can hear a difference between a lossless FLAC file and the original uncompressed WAV, "responsive" is up to you to quantify. Since you're planning a real-world test, you should be able to tell fairly quickly which will suit your use case.

I will say that if you want 100% safety for your active VM disks, you'll want to use synchronous writes on your datasets or zvols by setting sync=always - see the resource below on why it's required.

Resource - Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster?

This post is not specific to ESXi, however, ESXi users typically experience more trouble with this topic due to the way VMware works. When an application on a client machine wishes to write something to storage, it issues a write request. On a...

www.truenas.com

Getting solid performance out of sync writes will require the use of a separate log device - usually an NVMe SSD with a high write endurance and a low write latency. (Optane works well for this, there are other options too, but a general consumer SSD often isn't up to the task.)

Alternatively, because this is a homelab, if you're comfortable with the risk and have frequent backups, you can simply run sync=standard and accept the risk of potentially having to revert to an older snapshot.

 

[NickF]

My current NAS, NAS1 is a physical machine, but something with the S5572 motherboard will not work with TrueNAS 13.0, it has to stay at 12.0. .

Why? DDR3 generation hardware should work fine with 13.

 

[HoneyBadger]

should work fine

A lot of things fall under this umbrella, but due to various issues like buggy BIOSes, incomplete or inaccurate UEFI firmware, or the phase of the moon, older hardware occasionally fails the test of time.

 

[NickF]

A lot of things fall under this umbrella, but due to various issues like buggy BIOSes, incomplete or inaccurate UEFI firmware, or the phase of the moon, older hardware occasionally fails the test of time.

Sure, but I've run Supermicro X8, X9 and X10 boards on 13, plus a bunch of old HP Gen7 era and Dell 11th gen stuff. I've not ever owned or managed a Tyan board, but I would be surprised if it doesn't work when similar boards from other folks do. Tyan isn't typically associated with really crappy products.

 

[smcclos]

I will say that if you want 100% safety for your active VM disks, you'll want to use synchronous writes on your datasets or zvols by setting sync=always - see the resource below on why it's required.

Getting solid performance out of sync writes will require the use of a separate log device - usually an NVMe SSD with a high write endurance and a low write latency. (Optane works well for this, there are other options too, but a general consumer SSD often isn't up to the task.)

Will look into the sync settings

I use SSDSC2KB480G7 as SLOG and ZIL for each pool, so I think I am covered there. They are SATA drives, but high endurance.

 

[smcclos]

Why? DDR3 generation hardware should work fine with 13.

I have tried, it didn't work with this motherboard, Here is my post detailing that saga: Bizarre upgrade from 12.1 to 13.0

 

[HoneyBadger]

Will look into the sync settings

I use SSDSC2KB480G7 as SLOG and ZIL for each pool, so I think I am covered there. They are SATA drives, but high endurance.

Intel S4500 - the closest that's already been benchmarked by a forum member is probably the S4610 which does respectably well enough for a SATA SSD.

SLOG benchmarking and finding the best SLOG

You don't need to forget that 0815 NAND with PLP might be enough for 99% of the home systems postet here. Spot on. While overkill is fun to benchmark, it's important to build appropriate solutions. We'd all love to be able to set up all-flash systems with NVDIMM SLOG devices and Optane special...

www.truenas.com

 

[smcclos]

Results for SSDSC29B480G7:

Code:

root@vb-truenas01[~]# smartctl -a /dev/da1
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Intel S4510/S4610/S4500/S4600 Series SSDs
Device Model:     IN]TEL SSDSC2KB480G7
Serial Number:    PHYS820304LN480BGN
LU WWN Device Id: 5 5cd2e4 14f82b48d
Firmware Version: SCV10150
User Capacity:    480,103,981,056 bytes [480 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jun 26 10:12:39 2023 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Code:

root@vb-truenas01[~]# diskinfo -wS /dev/da1
/dev/da1
        512             # sectorsize
        480103981056    # mediasize in bytes (447G)
        937703088       # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        58369           # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        ATA INTEL SSDSC2KB48    # Disk descr.
        PHYS820304LN480BGN      # Disk ident.
        mps0            # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:     83.8 usec/IO =      5.8 Mbytes/s
           1 kbytes:     89.4 usec/IO =     10.9 Mbytes/s
           2 kbytes:     85.8 usec/IO =     22.8 Mbytes/s
           4 kbytes:     88.9 usec/IO =     44.0 Mbytes/s
           8 kbytes:    102.9 usec/IO =     75.9 Mbytes/s
          16 kbytes:    129.7 usec/IO =    120.4 Mbytes/s
          32 kbytes:    188.1 usec/IO =    166.1 Mbytes/s
          64 kbytes:    280.6 usec/IO =    222.7 Mbytes/s
         128 kbytes:    444.6 usec/IO =    281.2 Mbytes/s
         256 kbytes:    848.3 usec/IO =    294.7 Mbytes/s
         512 kbytes:   1528.9 usec/IO =    327.0 Mbytes/s
        1024 kbytes:   3029.1 usec/IO =    330.1 Mbytes/s
        2048 kbytes:   6364.1 usec/IO =    314.3 Mbytes/s
        4096 kbytes:  13359.4 usec/IO =    299.4 Mbytes/s
        8192 kbytes:  26534.8 usec/IO =    301.5 Mbytes/s

Looks like my bottleneck is the SATA connection. I would have thought it should be higher.

Checked my production bare metal config, and getting the same performance. Different cables, different HBA, different system

Code:

Synchronous random writes:
         0.5 kbytes:     91.7 usec/IO =      5.3 Mbytes/s
           1 kbytes:     93.1 usec/IO =     10.5 Mbytes/s
           2 kbytes:     94.8 usec/IO =     20.6 Mbytes/s
           4 kbytes:     89.0 usec/IO =     43.9 Mbytes/s
           8 kbytes:    103.3 usec/IO =     75.7 Mbytes/s
          16 kbytes:    123.9 usec/IO =    126.1 Mbytes/s
          32 kbytes:    164.5 usec/IO =    189.9 Mbytes/s
          64 kbytes:    244.9 usec/IO =    255.2 Mbytes/s
         128 kbytes:    390.2 usec/IO =    320.4 Mbytes/s
         256 kbytes:    840.7 usec/IO =    297.4 Mbytes/s
         512 kbytes:   1557.1 usec/IO =    321.1 Mbytes/s
        1024 kbytes:   3133.3 usec/IO =    319.1 Mbytes/s
        2048 kbytes:   6365.6 usec/IO =    314.2 Mbytes/s
        4096 kbytes:  12574.6 usec/IO =    318.1 Mbytes/s
        8192 kbytes:  26786.6 usec/IO =    298.7 Mbytes/s