8 x 2TB of hard drives, what configuration is recommended? And some performance problems.

[zllovesuki]

Greetings! After many nights of interweb Googling and searching, I'm still indecisive about the configuration.

Currently I have:
Xeon E3-1230 V3
4 x 8GB Crucial 1600MHz ECC CAS 11
ASRock E3C224
Corsair CX430 80+ bronze (thinking of getting a gold, but too much wattage to be efficient)
6 x 2TB WD Red (2 more will be here tomorrow)
Corsair C70 case with 5.25 hard drives bay

Switch:
TP-Link TL-SG2424 (all machines LACP)

Clutch:
APC Smart-UPS 1000VA

Code:

[root@Hypervisor ~]# zpool status
  pool: Storage
state: ONLINE
  scan: none requested
config:

    NAME                                           STATE     READ WRITE CKSUM
    Storage                                        ONLINE       0     0     0
      mirror-0                                     ONLINE       0     0     0
        scsi-SATA_WDC_WD20EFRX-68_WD-WMC4M2912068  ONLINE       0     0     0
        scsi-SATA_WDC_WD20EFRX-68_WD-WCC4M3028400  ONLINE       0     0     0
      mirror-1                                     ONLINE       0     0     0
        scsi-SATA_WDC_WD20EFRX-68_WD-WCC4M3089372  ONLINE       0     0     0
        scsi-SATA_WDC_WD20EFRX-68_WD-WCC4M3096188  ONLINE       0     0     0
      mirror-2                                     ONLINE       0     0     0
        scsi-SATA_WDC_WD20EFRX-68_WD-WCC4M0778730  ONLINE       0     0     0
        scsi-SATA_WDC_WD20EFRX-68_WD-WMC4M1695775  ONLINE       0     0     0
    logs
      scsi-SATA_SanDisk_SDSSDHP140874400287-part1  ONLINE       0     0     0

errors: No known data errors

Previously I was running ZFS on Linux and serving Netatalk (Time Machine and file access to the whole house) + NFS (XenServer on another E3), I realized that NFS was painfully slow. Yep, sync writes. I added a SanDisk Ultra Plus for ZIL, much better. Yet I will be adding a PCIe SSD for ZIL (should be here Friday).

Then strange things happened. (Spinners') IO utilizations were all about 80% when they are doing Time Machine, and when I tried to stream via Nginx, it stutters (apparently the hard drives couldn't keep up). Network problem is eliminated, CPU was around 50% when 3 of us are doing time machine and streaming, memory was around 24GB.

Now throw that out of the mind, I will be adding two more hard drives to the zpool. Before that, I am waiting for 3 x 3TB Seagate coming for offsite backup. I will back the files to another NAS then rebuild the spool.

Now the real question is, what configuration to go with 8 x 2TB? I will be using FreeNAS (ZoL is dumb: SPL and VFS don't like each other--double caching.)

Workload: Moderate amount of random reads and large amount of sequential reads. Random (sync) writes are out of the mind as I will be adding a ZIL. https://calomel.org/zfs_raid_speed_capacity.html shows that RAIDZ2 is a better choice for a balance of sequential performance and capacity in the case of ~8 hard drives. However I read it off somewhere that I will only have one disk of random read performance if I do a single RAIDZ2/Z. I am thinking of RAID 10 (4 vdevs, each has two disks mirror), RAID 60 (2 vdevs, each has four disks RAIDZ2), or just plain old RAIDZ2. Capacity is not the concern, however, I cannot seem to find a real number concerning about random performance between a stripe of mirrors and RAIDZ/Z2. Data integrity is not a burning concern as I will have 2 x offsite cold backup.

Or, just SAME (Stripe And Mirror Everything)

Thanks guys.

 

[anodos]

A RAIDZ vdev will have the IOPS of approximately a single disk. If your work is IOPS intensive then performance of a single vdev may be inadequate. If you don't need the capacity or redundancy of RAIDZ2 then go with mirrors.

 

[zllovesuki]

A RAIDZ vdev will have the IOPS of approximately a single disk. If your work is IOPS intensive then performance of a single vdev may be inadequate. If you don't need the capacity or redundancy of RAIDZ2 then go with mirrors.

blah, I will just do 4 two-way mirror and offsite backup then.