looking for advise on how to configure my storage

Joined
Jan 22, 2021
Messages
5
Hi all, Hopefully I am asking this in the right place.

This will be my first ZFS configuration and am moving away from an Xpenology VM in Esxi. Thinking Truenas core?

My question is how best to configure my hardware that I have to work with. one system as primary and one as a backup, and this is for my work, backup, plex library, transcoding, many dockers/jails/VM's, the works.

1x Dell R630 w/ 2x Xeon E5-2670 v3 and 48gb ECC RAM.
1x Dell R630 w/ 2x Xeon E5-2670 v3 and 128gb ECC RAM.
1x Netapp DS4246 IOM6 with x12 Seagate 16tb sata 4kn 6Gbps EXOS Drives.
1x Netapp DS4246 IOM6 with x20 Seagate 6tb SAS 12Gbps 4kn USED enterprise drives.
5x 1tb sata 3 SSD
1x Nvidia Quadro p1000

so realize the 6tb drives are superior, but they are used from a friend who decommissions enterprise gear and I can pick up the drives on the cheap. The 16tb exos drives have been purchased in 2020, so they are not old and each have factory 5 year warranties on the drives.

what my current plan is:
Primary: Truenas Core
1x Dell R630 48gb ECC RAM.
1x Netapp DS4246 with x12 Seagate 16tb in 2-6 disk vdevs in a raid z2
1x 1tb SSD
will I need SSD caching, or a log drive, if so How big is necessary?

Backup: Unraid
1x Dell R630 w/ 2x Xeon E5-2670 v3 and 128gb ECC RAM.
1x Netapp DS4246 x20 Seagate 6tb SAS USED enterprise drives. (XFS)
1x Nvidia Quadro p1000 (Plex)
4x 1tb Sata SSDs raid 10 (BTRFS) (for dockers, VM's, Plex)

Is this a solid plan and are there any recommendations or gotchas?

Thanks!
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
many dockers/jails/VM's

x12 Seagate 16tb in 2-6 disk vdevs in a raid z2
These things don't match. Consider having as many VDEVs as possible in Mirrors for block storage. RAIDZ2 is a good match for reliable media storage (plex, etc).

1x 1tb SSD
Consider using that for jails, replicate jail appdata to the RAIDZ2 for safety.

1x Dell R630 48gb ECC RAM.
TrueNAS will probably benefit a lot more from additional RAM... for block storage 64GB is usually the starting point if you want good performance.

will I need SSD caching, or a log drive, if so How big is necessary?
If you want to do block storage for a lot of VMs/NFS/iSCSI, it's probably going to be wise to have it:

Backup: Unraid
Not sure if this is really optimal as you won't be able to use ZFS replication to your backup host, hence killing your best automated backup option.
 
Joined
Jan 22, 2021
Messages
5
ok Great, appreciate the response and the links!

These things don't match. Consider having as many VDEVs as possible in Mirrors for block storage. RAIDZ2 is a good match for reliable media storage (plex, etc).

the majority of this storage is Media/video/pictures ~80TB Id say. Around 6~8TB is block VM's or ISCSI. I am looking for a nice mix of speed, redundancy, and get the most out of the drives I have, mirrors just seem like space hog when I would go from 8 drives usable to 6. but if there's something I don't know please let me know.


Not sure if this is really optimal as you won't be able to use ZFS replication to your backup host, hence killing your best automated backup option

When it came to unraid, I really just love the community around it and the vast docker support. not only as a backup target, it is going to act as a lab as well.

Would rsync not be a good way to accomplish backups?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Would rsync not be a good way to accomplish backups?
Perhaps effective, but not as efficient as a snapshot-based block-level replication. You may find it slow with rsync depending on the profile of your data... lots of photos for example may mean the rsync checking is really slow.
If you do decide to use it, look at these threads:

Around 6~8TB is block VM's or ISCSI.
How important this function is and how much performance you want is entirely up to you.

If you don't use Mirrored VDEVs, you're severely limiting the IOPS of your setup and adding SLOG won't compensate in any meaningful way for it if the pool isn't capable behind that. Read @jgreco 's post that I linked for you and decide for yourself how you want to do it.
 
Joined
Jan 22, 2021
Messages
5
Perhaps effective, but not as efficient as a snapshot-based block-level replication. You may find it slow with rsync depending on the profile of your data... lots of photos for example may mean the rsync checking is really slow.
If you do decide to use it, look at these threads:


How important this function is and how much performance you want is entirely up to you.

If you don't use Mirrored VDEVs, you're severely limiting the IOPS of your setup and adding SLOG won't compensate in any meaningful way for it if the pool isn't capable behind that. Read @jgreco 's post that I linked for you and decide for yourself how you want to do it.

I really appreciate the help! like I said this will be my first ZFS build so the more knowledge i can get up front the better. I will read though those posts and hopefully come up with a solid option.

would mirroring the 6tb drives to say 10 mirror vdevs and make that primary just be a better option? then use the 16's as just deeper, slower storage? the IOPS increase would be huge and I should be able to get more of those as just cold spares
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
would mirroring the 6tb drives to say 10 mirror vdevs and make that primary just be a better option? then use the 16's as just deeper, slower storage? the IOPS increase would be huge and I should be able to get more of those as just cold spares
It sounds like you're heading down the right path with that thinking... do some more reading and confirm it to your own satisfaction.
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,904
Just wanted to throw in that the approach of having a "simple" (i.e. non-enterprise) SSD, or pair thereof, for running the VMs and doing backups to RAIDZ2 storage, is exactly what I am doing here. I have hourly delta backups and daily full backups, and for my level of paranoia that is sufficient.
 
Joined
Jan 22, 2021
Messages
5
Just wanted to throw in that the approach of having a "simple" (i.e. non-enterprise) SSD, or pair thereof, for running the VMs and doing backups to RAIDZ2 storage, is exactly what I am doing here. I have hourly delta backups and daily full backups, and for my level of paranoia that is sufficient.
Thanks Chris, that sounds like a good plan.

so I think Primary is 10 6tb mirror vdevs. The 2nd system maybe unraid as hypervisor for just some certain functions, then passthrough the HBA to a truenas VM as the backup target, and like sretalla mentioned, I can use the replication feature. I don't see needing caching for backup.

This way plex can run as a docker with the nvidia card on unraid (from what I read, truenas does not have Nvidia driver support for this function) and then it can just connect to the shares of the virtualized backup truenas so its not needing to drain resources from the primary system.
 
Top