Stripe to a Z1 or Z2 Pool

[holmes.j]

I built a FreeNas rig 4 years ago and I don't think I really knew what I was doing back then (nor am I any smarter today). I have 2x 8TB drives that are striped. Pretty sure I knew that was a bad idea back then, but then I thought I was going to upgrade things soon so it didn't really matter.

4 years later and I haven't touched the thing. I noticed I was down to 450GB free so I've bought 2 more 8TB drives. I figured I could find a way to convert this stripe to a Z1 or Z2 pool but that doesn't seem possible. Is it?

So what's my next best option? I've got 4x 8TB drives. I'm not opposed to buying another one or two if that's what I have to do to get some sort of failure protection.

I've updated to TrueNAS-12.0-U5.1 so I'm on the latest and greatest.

 

[holmes.j]

So trying to make sense of what I've read:

I can't add to my exisitng vdev stripe. I can add to an existing zpool, but I can't remove vdevs from pools.

Will this work?

1. Buy another disk and create a 3 drive Z1 vdev in a new pool
2. Somehow copy freenas and all data to the new pool. How do I even do that?
3. Kill the old pool.
4. Buy another disk and create another 3 drive z1 vdev and add to the new pool from step 1.

 

[NugentS]

1. Buy another disk and create a 3 drive Z1 vdev in a new pool - Yes
2. Somehow copy freenas and all data to the new pool. How do I even do that?
   1. Snapshot send and recv from the old pool to the new pool - see below
3. Kill the old pool. - Yes
4. Buy another disk and create another 3 drive z1 vdev and add to the new pool from step 1. - Yes

Although I think I would prefer a 6 drive Z2 pool as you can lose any two drives rather than 1 drive from each vdev but its difficult to see how you would get there unless you can get the data off the disks first onto some backup media and thats approx 14TB of data

As for the send recv - take a look at the script I use to "churn" the datasets when I was populating the metadata special vdev.

Code:

zfs snap BigPool/SMB/Common@migrate
zfs send -R BigPool/SMB/Common@migrate | zfs recv -F BigPool/SMB/Common_New
zfs snap BigPool/SMB/Common@migrate2
zfs send -i @migrate BigPool/SMB/Common@migrate2 | zfs recv -F BigPool/SMB/Common_New
zfs destroy -rf BigPool/SMB/Common
zfs rename -f BigPool/SMB/Common_New BigPool/SMB/Common

This script does the following
Snapshots BigPool/SMB/Common then
Copies the snapshot to /BigPool/SMB/Common_New - this will be slow as its all the data
Does another snapshot of BigPool/SMB/Common then
Copies the snapshot to /BigPool/SMB/Common_New - this will be fast as its only a delta from the previous snapshot as a just in case
Destroys /BigPool/SMB/Common
Renames /BigPool/SMB/Common_New to /BigPool/SMB/Common

This does the job on the same pool, but you can always point the copy (strictly zfs recv) to the new pool instead

 

[holmes.j]

I just ordered a 16gb external hard drive. So I'll copy over there.

My rig only has 5 internal bays, so guess I'm stuck with 5 drives. 5 drives in z2? This is a home media server so performance isn't really that important.

 

[Patrick M. Hausen]

5 drives in RAIDZ2 will work well for a media server.

 

[NugentS]

That will give you an extra 8TB of useful space.
[Resiliancy really sucks from a cost for capacity PoV]

You may need to look into a new case OR an external SAS enclosure and an HBA with an external port or just bigger disks in the long run.

 

[Etorix]

4- or 5-wide Z2 is the safest option.

Otherwise, you could attach one new 8 TB drive to each of the old drives, to turn them from single vdev into 2-wide mirrors. That would, at least, be a setting with some redundancy but not as safe as a raidz2 and less space efficient than a 5-wide raidz2.

 

[holmes.j]

I'm thinking to make this with the effort I'll do a 6 drive z2 array. Should hold me over for a few more years. When I need more space I can add to the pool.

While I'm doing this - does anyone ever separate their data into a separate pool from the system/jail pool? Wondering if I should setup a smaller 4 disk z2 pool for the system/jails

Thanks @NugentS for the script. Looks straightforward.

Now to buy an external enclosures, any recommendations for a 6 disk array? Also, Is there a list of sata cards with external ports that work well with truenas? I assume it's the same hookups. Time to do some reading

Thanks y'all!

 

[jgreco]

Now to buy an external enclosures, any recommendations for a 6 disk array? Also, Is there a list of sata cards with external ports that work well with truenas? I assume it's the same hookups. Time to do some reading

STRONGLY disrecommended, these generally use SATA port multipliers which range from merely tragic to catastrophically bad.

Do head on over to the SAS Primer to learn how to do this correctly.

 

[NugentS]

As @jgreco says - do not use an external SATA / SATA Expansion ports. TrueNAS is commercial grade and likes serious hardware.

You can get external SAS arrays - but they are generally commercial gear so look in the second hand market

However a new case is probably your cheaper option.

 

[Etorix]

@holmes.j The cheapeast and easiest would be to go for a 5-wide Z2. Should you need more space, you would replace drives by larger ones—and get the expansion upon replacing the last one.
If you're thinking into investing to expand your system, take this opportunity to describe it and explain your use and what these jails are.

 

[holmes.j]

@Etorix and @NugentS thanks for the help.

My system:
NZXT H440 Case
Intel Xeon 2.5 E5-2670 V2 10 Core Ivy Bridge
Hynix 64GB 4x16 ECC
ASRock EPC602D8A
4x 8TB WD Red Nas Disks
2x SanDisk 250GB USB 3.0 for boot drive

I have a handful of jails:
Plex Media Server, Transmission, Jacket, Sonarr and Radar. It's purely a media server. I don't backup anything on it, all photos and whatnot are on the cloud.

I'm not afraid to spend a little money to get this done right and do it now. But not trying to throw $5k at it.

I just bought 2 of the 8GB drives from Newegg, no idea what their return policy is but guess I could upgrade them. Not sure I want to drop $2500 on 5x12TB drives though.

Maybe I do the 5x8TB for a bit of breathing room now and then figure out either a bigger case or an external sas setup.

What would y'all do?

 

[holmes.j]

STRONGLY disrecommended, these generally use SATA port multipliers which range from merely tragic to catastrophically bad.

Do head on over to the SAS Primer to learn how to do this correctly.

Thanks for the primer. That's a great read. I'm a software engineer, this side of the world has changed a LOT since I last looked.

 

[jgreco]

this side of the world has changed a LOT since I last looked.

Well the SAS standard dates from 2004 so I don't know about that. But it's fun to bring people up into large scale solutions. Do ask questions!

 

[holmes.j]

Well the SAS standard dates from 2004 so I don't know about that. But it's fun to bring people up into large scale solutions. Do ask questions!

Touche. I guess my needs have changed a lot.

 

[rvassar]

Do burn in the 16Gb external drive you bought before committing to it as a backup. Most HDD's die in infancy, old age, or mid-life abuse. You need to trust it, even as a temporary medium while you transition your pool.

However... You can mismatch size in a ZFS vdev. The smaller device defines the size of the vdev. I would buy a second 16Tb drive. Mirror one of your 8Tb drives with the 16Tb unit and wait for it to silver in, then add the second 16Gb unit as a replacement mirror for the other 8Tb unit in the same vdev and wait for it to silver in. Then mirror the remaining 8Tb drive with the old now unused 8Tb unit and expand your pool to a "RAID10" redundant 24Tb. Then replace the 8Gb drives at your leisure and as your budget allows due to age... I think you can do this, but it might require using the command line, not the GUI.

 

[Etorix]

Board, CPU and RAM by the textbook. Only the case is letting you down.
Better not use the 2 SATA ports from Marvell, but with 6 SATA ports from the chipset and 4 more from a mini-SAS connector you have some room before even bringing in a HBA.

Two boot drives on SATA-to-USB adapters? Take the adapters off and attach the drives to SATA ports!
You might use this (unsupported) trick to partition the drives and move your jails to a small mirror pool on the boot drives.

HOWTO: setup a pair of larger SSDs for boot pool and data

Moderator Warning: This HOWTO will damage the ability of your TrueNAS appliance to properly maintain itself, will break the ability to replace failed drives, and may cause problems with upgrades and updates. TrueNAS is not designed to do this sort of partitioning, and the developers are not...

www.truenas.com

 

[holmes.j]

Hi all,

I'm thinking a bigger case might actually be the simpler avenue for now. All this talk about SAS vs SATA has got me a little confused. Missed this part in the primer: The primary takeaway: You can connect SATA drives to an SAS port and it is expected to work. You cannot connect SAS drives to a SATA port. That absolutely won't work.

@Etorix just curious, why not use the 2 SATA ports from Marvell? If I can get away with it then I can just max out 8x8TB disks in a Z2 array (with a bigger case) and probably call it a day. Otherwise I guess I'm going HBA shopping. Any recommendations there?

Should I bother with the separate pools for system/jails vs. media? Or is that overkill? Sounds kinda fun if I get bigger case. Would these need to be NAS drives or just regular hard drives? I guess if I split out the main/system pools I'd have to go HBA shopping anyways.

And the boot drives are just USB sticks, not sure if that makes a difference. I thought I saw that was recommended back in the day.

 

[NugentS]

In general TrueNAS doesn't like marvel chipsets. However the SE9172 driver has apparently been tweaked to work better

Top Picks for FreeNAS HBAs (Host Bus Adapters)

We have our top picks for getting fast and reliable FreeNAS HBAs (host bus adapters) for SAS and SATA, using proven options for FreeNAS and ZFS

www.servethehome.com

USB sticks are not recommended (I do however use them on one of my NAS's). If you do use them do the following:
1. Make sure you have two different ones, mirrored
2. Make sure you get a regular copy of the config file in a safe place (not on the NAS). I get mine emailed to me every day
3. Have a spare
4. Move the system dataset off the bootpool onto a regular pool as that seriously reduces the wear and tear on the USB sticks
5. Use USB 2.0 - probably lasts longer and speed is NOT an issue

If you do all that then they will probably (maybe) last for a considerable period of time before one dies.
[Remember Computers sense weakness]

I wouldn't separate jails and media onto different pools without a specific need. They will be in different datasets and each dataset can be configured for the contents (if you want to)

NAS drives are better (with the possible exception of WD Red). Make absolutely sure you are buying CMR and not SMR drives. SMR and ZFS do not mix. Do not buy WD Green or similar - they aren't really designed for this.

Also be warned (based on one of your previous comments) that you cannot just add to a pool. All you can do is add a new vdev, with its own resiliancy etc. In theory, next year maybe you might be able to extend a RAIDZ pool by adding another drive - you just can't yet

 

[jgreco]

@Etorix just curious, why not use the 2 SATA ports from Marvell?

Because some of Marvell's parts suck ***, while others don't. However, "SATA port" add-ons are right on up there alongside Realtek ethernet chips as things that have been ripped off by back alley Asian counterfeiters, for sale to unwitting PC enthusiasts looking for a bargain, and this means that there is some additional risk to your pool -- especially since disk communications problems can lead to pool corruption.