Raid Z3 or Raid Z2+Hot spare?

[kirkdickinson]

Reading about spares and was interested in where that would be a good option. I currently have two 4 year old FreeNAS boxes. Office server with 6-4TB in Z2 and Home Server with 4-4TB in Z1.

I am about to build a New Office server which will have twice the capacity (6-8TB Z2) and decided to redo my older servers also. I ordered 10 8TB drives. The New server will have 6-8TB in Z2 and the Home server will have 4-8TB in Z1. (home server already upgraded)

So now I have the old office server which has lots of extra room for drives. I thought about redoing that and redoin the pool from 6-4TB to 10-4TB. I also have an extra 4TB drive I could throw in there. Since these drives are now 4 years old, I was wondering if it would be better to set it up with 11-4TB in Z2 with a spare or run 11-4TB in Z3?

All these drives in question are Western Digital Reds. The 8TB are Red Plus.

My old Windows server with an Areca RAID card had 750GB WD Reds and I got 6 reliable years before the first drive failed. I figured that these WD Reds probably have some life left to use in a backup server.

 

[Dan Tudora]

hello
with a 4 years old HDD be gentle, and do not have many expectation of life/reliable
make a server "backup of the backup" server for the "old man" and be gentle and do not "put" this online 24/7 time
just start this thing for a couple hours on day for "backup of the backup" scenario
and YES Z3 is yours solution
zfs send | zfs receive will help you to have 3th backup of your data
cheers

 

[kirkdickinson]

hello
with a 4 years old HDD be gentle, and do not have many expectation of life/reliable
make a server "backup of the backup" server for the "old man" and be gentle and do not "put" this online 24/7 time
just start this thing for a couple hours on day for "backup of the backup" scenario
and YES Z3 is yours solution
zfs send | zfs receive will help you to have 3th backup of your data
cheers

Thanks. So Z3 is better than Z2 with a hot spare? Is that because of the possibility of multiple disk failures with older drives making a rebuild impossible? Would that hold true for a system with new drives? I figured that the 4TB Reds are fairly cheap right now, I might just order a couple extras to have around and monitor this system and swap new drives as needed. I have some Reds that have lasted 8 years so I know that they are pretty tough.

Interesting. I never thought about having it running only a few hours a day. How would you do that? Would the server be up all the time, but the pool on standby? Is there a scheduler in FreeNAS to do such a thing?

 

[Dan Tudora]

Is that because of the possibility of multiple disk failures with older drives making a rebuild impossible?

YES because on the old HDD failure rate is expected (old, old is old), AND yes spare are spin like all of drive, and yes the silver time/temp of drive incress

Interesting. I never thought about having it running only a few hours a day. How would you do that? Would the server be up all the time, but the pool on standby? Is there a scheduler in FreeNAS to do such a thing?

is a many script/command to shutdown/waikup a server/NAS at a time scheduler at yours need
I use some of that
WAKEONLAN is your answer for wakeup server/NAS on scheduler manner
at mine this is on the router/gateway (pfsense), but is many router with that function
or if you have other machine/computer run 24/7 can set this thing to do that
for shutdown of NAS is a script somwhere to do that
for windows wakeonlan I use https://www.depicus.com/wake-on-lan/wake-on-lan-gui
and have a command line tool
success

 

[Dan Tudora]

hello
look like that at mine
put that in a a file with .sh extension and what name you want (shutdown.sh ) and make executable as root in /root folder
use winscp program (from windows of course) for create/edit that thing
or other program for other operating system
on FreeNAS/TrueNAS make a cron task to run this shutdown.sh when you want to shutdown the NAS


#!/bin/sh

/sbin/shutdown -p now
(time.sleep(60)

 

[ChrisRJ]

It should be noted that running an HD 24x7 puts much less stress on it than daily startup and shutdown. As an anecdote: Seagate's data center drives (Exos line) are certified by Synology only when run 24x7, i.e. without putting them to sleep during time of inactivity.

 

[Herr_Merlin]

hello
with a 4 years old HDD be gentle, and do not have many expectation of life/reliable
make a server "backup of the backup" server for the "old man" and be gentle and do not "put" this online 24/7 time
just start this thing for a couple hours on day for "backup of the backup" scenario
and YES Z3 is yours solution
zfs send | zfs receive will help you to have 3th backup of your data
cheers

This advice is rather retarded.
If the disks run 24/7 before keep them running 24/7. Cycling will kill them faster.
keep the position equal e.g. horizontal or vertical for the mounting don't rotate in the other system.
Go for Z3 with a spare disk.

at work we have production servers running 10+ year old FC and SAS disk without issues and heavy workloads 24/7. The power cycle of those disk is usually below 100 times.. and running hours above 80.000h

take such disk a server is decommissioned and put in a Workstation and power cycle it twice daily.. well it's a few months and it will die!

 

[Chris Moore]

My recommendation is RAID-z2 with COLD spares.

 

[Chris Moore]

I am about to build a New Office server which will have twice the capacity (6-8TB Z2)

I would suggest adding a second vdev, so you have 12 drives in two 6 drive vdevs. Still at RAID-z2. Making more vdevs increases the data rate to the drives because each vdev behaves as a single drive from a performance standpoint. You might not even need a new server, just add a second vdev to the existing one. However, there are advantages to a new pool because when you load the data it will be spread across all the disks. If you only add a second vdev to an existing pool, the data is not rearranged and only new data will be placed on the new disks. There are ways around that if you are interested.

 

[Chris Moore]

I was wondering if it would be better to set it up with 11-4TB in Z2 with a spare or run 11-4TB in Z3?

You want to keep the number of drives per vdev low, if performance matters to you at all. Smaller vdevs are better, but have more of them. That is the reason some folks that want best performance choose mirror vdevs and use thirty of them. Thirty vdevs of two drives is much faster than four vdevs of 15 drives. I have done the testing, others have also, but the more vdevs you have, the more IOPS you can have. This is important when you have many users requesting data simultaneously.

 

[Chris Moore]

be gentle and do not "put" this online 24/7 time
just start this thing for a couple hours on day

That is not gentle. Initial startup is when most drives die. Starting and stopping a drive repeatedly is the fastest way to kill it. I have enterprise storage drives that have run 24/7 for as much as 10 years with no failures at all. The idea that you are being gentle on a drive by turning it off is false.

 

[Chris Moore]

So Z3 is better than Z2 with a hot spare?

No. RAID-z3 has more overhead because you are writing to an additional drive. It is slower. The only way you could think of it as "better" is that it allows for a maximum of three failures at the same time where RAID-z2 only allows for 2 failures. If you are monitoring your systems, it is incredibly unlikely that you will need the additional "safety" of three drive failures and a hot spare is a pure waste of a drive. I have had two drives fail in the same day in the same vdev, and replaced them both at the same time. It isn't even scary when you have a backup. Each vdev is a separate failure domain, so if you have 12 drives in two RAID-z2 vdevs, you already have 4 drives in the pool that could fail before you loose data.

 

[kirkdickinson]

I would suggest adding a second vdev, so you have 12 drives in two 6 drive vdevs. Still at RAID-z2. Making more vdevs increases the data rate to the drives because each vdev behaves as a single drive from a performance standpoint. You might not even need a new server, just add a second vdev to the existing one. However, there are advantages to a new pool because when you load the data it will be spread across all the disks. If you only add a second vdev to an existing pool, the data is not rearranged and only new data will be placed on the new disks. There are ways around that if you are interested.

Well... revisiting this thread after a couple weeks. My new server with 6 8TB drives in z2 has been humming along nicely for a couple weeks. I still have "Old Reliable" for my active documents and database stuff and am only using the new server to mirror everything and for my daily backups from workstations. (I back up email files, user data, photos, and a once a week image of each workstation boot drive).

I am thinking that I will run the new server for a month or so just to make sure I trust it and let the snapshots build up. Will promote that to main server then and wipe "old reliable" and go from 6-12 4TB drives. Rebuild "Old Reliable" from scratch as a TrueNAS system. I think I will just use Z2 because that will only be a backup server at that point. I need reliability more than speed. There really isn't that much access. Just a lot of files.

 

[Dan Tudora]

That is not gentle. Initial startup is when most drives die. Starting and stopping a drive repeatedly is the fastest way to kill it. I have enterprise storage drives that have run 24/7 for as much as 10 years with no failures at all. The idea that you are being gentle on a drive by turning it off is false.

hello
I understand what you say
BUT I say "backup of the backup"
main "backup of the backup" server is poweron/work 6 hour at week when make sync at important files from other server, short smart test and scrub at 28 days
REPEAT 6 hours/week
I don't know what is the best for me (HDD) but work in this manner for 2 or 3 years now

 

[Chris Moore]

I don't know what is the best for me (HDD) but work in this manner for 2 or 3 years now

If you are trying to save power, or reduce the heat output from the equipment, then it makes sense to shut down when not needed. The most frequent failure point for hard disk drives in general is at start and stop. You may get another decade out of the drives, or one could fail tomorrow. There is no way to know for certain.

 

[Dan Tudora]

There is no way to know for certain.

hello
IS a way
wait for failure and after that know
is a RAIDZ1 with 4 WD Green/2 TB (heeee remember green WD what must be mode in firmware with some WD utilities software ??)
NOT die no one (for now, stick in wood)