Build a Truenas server with 7x8TB raidz2

[teapot9]

Hello,

While I have known ZFS for some time, this is my first time actually using it.

My current NAS has 15TB used space and I need more storage.
I recently got a supermicro CSE-846 server, probably overkill for my needs, with an Intel XEON E5620, and 24GB RAM included.

I plan to get 6x8TB or 7x8TB HDD in a raidz2 vdev for my main stogage.

Stored data:

• Mostly media files: pictures, videos (a lot), music.
• Some git repositories and source code, but most of the code is either on github or gitlab.
• A dozen VMs, some are experimental, and others are hosting personal services (media sharing/streaming, file hosting, backup services, ...).

I am thinking of getting 3x1TB or 3x2TB SSD in raidz1 to store the VMs.

For backups, I currently have an offsite backup server where I also plan to use Truenas with daily replication.
I would like to transfert my current 5x6TB HDDs in there and add some more to get a 8x6TB raidz2.
Currently, the backup server has 1x12TB HDD with an incomplete backup.
I am not sure what I will do with this drive. Maybe keep it as cold spare or backup, or for a possible future vdev with 12TB drives, or sell it on ebay.

So here are some questions:

1. Is there any remarks regarding my plan? Or need more informations?

2. Are my raidz vdevs reasonably sized (number of drive, size of drives)?
Since I have backups, I don't feel too afraid of fatal pool failure. But restoring from backup would be a pain I don't want to live every n months.

3. Considering resilvering time of 8TB drives, is loosing too many drives something I should worry about (with 6 or 7 drives in raidz2)?

4. What is the advantage of having a cold/hot spare? Would having a spare be equivalent to running a raidz3?

5. All I need to backup from the system itself is the configuration and the encryption keys, am I right?

6. Is ZFS record size something I should worry about? Or is the 128K default fine anywhere?
Since I sort my data by type, I was wondering if I would get any advantages from using different record size for videos than other data.

Thank you

 

[NugentS]

Keep the 12TB, stick it in a USB case and use it as an emergency cold backup by copying files to it and then unplugging and placing in a safe / secure point. It won't hold all the files - but should be the ones you really don't want to loose. Or keep it as the first drive in a future vdev.
1. Seems sensible - just follow the recommended hardware for TrueNAS
2. Seems good - you are allowing for more than one failure and you have a (theoretical) backup in the design
3. You have a backup - Z2 if fine, if you are feeling paranoid use Z3
4. Its always worth having a cold spare around. I don't like hot spares as they spin all the time accumulating wear for no purpose
5. Yup - there are scripts that will automate such tasks: https://github.com/Spearfoot/FreeNAS-scripts
6. Generally not. If however you split your data up by dataset - so videos on one dataset - you could use a larger record size. I doubt you will notice a significant difference though

 

[teapot9]

Thank you for your answers, that was really helpful!

Some more questions:

7. Is it ok to have some of the drives bought used? Or should I prefer new ones?

8. Should drives be tested before use? If so, how should they be tested?
I was thinking of running a long smart selftest, or running a write/verify with badblocks (which will probably be quite long).
I did find this resource regarding testing.

 

[NugentS]

OK
7. That's entirely up to you and you appetite for risk. Many do use 2nd hand discs. But if you do then test them well AND have a backup AND be prepared for early / more frequent failures (RAIDZ2 or Z3)
8. I use badblocks to test both new and secondhand disks