Efficient way to move and recreate vdevs? Best use of extra SSD/nvme drives?

[Dizzy49]

Long story short, I'm moving off of Windows and Drivepool.
I have 5x 18TB Seagate Exos drives that I created a 5w RAIDZ2 vdev/pool and I am copying as much as I can from my windows machine to free up those drives to create additional vdevs.

In the Windows machine I have 6x 18TB Seagate Exos, and 3x 16TB Seagate Exos drives that I would like to reuse.
As I am in the last 20-30hrs of the transfer I realized that I want to get another 20TB drive. Since i was using Drivepool I didn't have parity drives and I need to account for that "lost" space. I don't believe I can simply add another drive to the vdev and I will need to recreate it.

I was thinking I'd do a 6w RAIDZ2 with the 18TB, and a 3w RAIDZ1 with the 16TB. Create a pool with those two vdevs and copy everything from the vdevs with the 20TB drives. Add the new 20TB, create new vdev and add it to the pool with the others.

1) Is that the best way to move the data?
2) Is that a good plan for the vdevs? This is all video storage. New things get added, some get replaced on occasion, but mostly reading. Temp transcode directory is on a SSD.


For the rest of the machine, I have a 120GB SSD with TrueNAS installed, then three 5w RAIDZ1 with 1TB Samsung SSDs. That is my app storage, VMs, and day-to-day accessed files and more important files (ie all the family pics and documents). Everything is backed up weekly on some spinning drives, and most is backed up in the cloud monthly (pics, docs, etc).

This is all housed in a Dell R740 with the following specs
2x Xeon Gold 6150
384GB Ram (12x32gb)
2x 1GB NIC
2x 10GB NIC
HBA330 for SSDs
LSI 9207-8e to connect to 2x EMC KTN-STL drive shelves

3) I also have 2x Samsung 970 EVO 250GB NVMes and a Sabrent Rocket Q4 1TB NVM drive, what would be a good use for them? Or should I just throw them enclosures and use them as portable drives?

 

[NugentS]

I think (from reading what you are saying) that will work. A couple of caveats.

1. TN will complain mildly about unbalanced vdevs - you can ignore this
2. I think you will end up with 6w Z2+3w Z1+1w Stripe - this is bad.
   1. The 6WZ2 is fine
   2. The 3WZ1 is not so hot. Z1 is not reccomeded for drives of that size
   3. The 1W Stripe - is bad. Your whole pool will fail if this drive fails

However your description is confusing so I may have got the wrong end of the stick.

For now do nothing with the Samsung 970 & Sabrent Rocket. I would consider one of them as an L2ARC (metadata only) to cache metadata and make listing files and folders quicker. But this can be added later - it will however take quite a while to becomne useful.

 

[Dizzy49]

I think (from reading what you are saying) that will work. A couple of caveats.

1. TN will complain mildly about unbalanced vdevs - you can ignore this
2. I think you will end up with 6w Z2+3w Z1+1w Stripe - this is bad.
   1. The 6WZ2 is fine
   2. The 3WZ1 is not so hot. Z1 is not reccomeded for drives of that size
   3. The 1W Stripe - is bad. Your whole pool will fail if this drive fails

However your description is confusing so I may have got the wrong end of the stick.

For now do nothing with the Samsung 970 & Sabrent Rocket. I would consider one of them as an L2ARC (metadata only) to cache metadata and make listing files and folders quicker. But this can be added later - it will however take quite a while to becomne useful.

Not sure where the 1w stripe is coming from.
6x 18TB = 6w Z2
3x 16TB = 3w Z1

I was debating on even using the 16 TBs but I think I need to for the amount of data I have. I think I'll make is a separate pool for now so I can move the data off the 20 TBs so I can recreate that vdev with the new 20TB drive. So I'll have.
6x 20TB = 6w Z2 \ == Pool: Video 6x 18TB = 6w Z2 / 3x 16TB = 3w Z1 == Pool: tmpVideo
Then once everything is on the Video pool I'll move the 16TBs to backup duties.

As for the NVMes, I was hoping to do something to help list files and folders quicker on the Video pool since there is A LOT of files and folders :D When you say it will take a while to become useful, how do you mean? Like I need lots of data before it's useful? Or the system needs to map things initially or something?

Thanks!

 

[NugentS]

There are two options for metadata - a metadata vdev which is pool critical and L2ARC (Metadata only) which is not.

The metadata vdev contains the metadata of any new metadata written to the pool - its not a cache. Ideally its setup before writing to the pool as that way it gets populated as you write data. If you already have data on the pool then this needs to be re-written to move the metadata to the vdev. If you lose this vdev you lose the pool

L2ARC (Metadata only) is a cache - so as you access files, the metadata is cached to the vdev and thus it becomes useful over time. This is / should be persistent over reboots. This is not pool critical - you lose the vdev and the pool just shrugs and goes back to the on HDD metadata. L2ARC (metadata only) is just a specifically limited L2ARC and does not require that much space.

You have 384GB of RAM - which is a lot. Assuming that this is a home server then do not be tempted by a full L2ARC until you have settled the pool into use and have used it for several weeks then check you Arc Hit Ratio. If thats > 90% then the full L2ARC will not help (metadata only will however)

 

[ChrisRJ]

I was thinking I'd do a 6w RAIDZ2 with the 18TB, and a 3w RAIDZ1 with the 16TB. Create a pool with those two vdevs [..]

That is a bad idea. You would reduce the overall redundancy level of your pool to RAIDZ1, because that is the lowest common denominator within your vdevs. In general a RAIDZ1 with 16 TB drives is strongly discouraged, but even more so in what you envision.

I would either get additional 16 TB drives to make another RAIDZ2 vdev out of it. Or go for a 3-way mirror, which would be equivalent to RAIDZ2 in terms of redundancy.

 

[Dizzy49]

The metadata vdev contains the metadata of any new metadata written to the pool - its not a cache. Ideally its setup before writing to the pool as that way it gets populated as you write data. If you already have data on the pool then this needs to be re-written to move the metadata to the vdev. If you lose this vdev you lose the pool

I do need to rewrite all the data, so I could use both the 250GB nvme in a mirror

You have 384GB of RAM - which is a lot. Assuming that this is a home server then do not be tempted by a full L2ARC until you have settled the pool into use and have used it for several weeks then check you Arc Hit Ratio. If thats > 90% then the full L2ARC will not help (metadata only will however)

Yeah, I wasn't sure about that, I remember reading that the L2 wasn't much help until you run out of ram, and with 384gb that might take a while :)
Yes, this is a home server for the most part. I am a consultant and run some things off the server for work, but nothing major.

 

[sfatula]

Another option is to mix drive sizes in a single vdev. So, for example, 6 18TB and 3 16TB in the same vdev. The 18TB will be treated as 16TB, until some day if you replace the 16TB with 18TB. Once they are all replaced, you gtet to use all the space again.