Expansion or Rebuild

[Sneffets]

Hello, I want to start by saying, I have spent ample amount of time doing research on this topic and I have not really found too many people with the same "options" that I have...so I am looking for some good solid opinions or if I am missing something...thank you in advance.

I have two FreeNAS servers, both running 11.1-U6, both are mult-core Xeon Processors, the main "production" one has 96GB of RAM and 6 - 4TB SAS drives (RAIDZ2), SSD Caching, etc...the second one isn't as powerful, 16GB RAM, 4 - 8TB SATA drives (RAIDZ1), SSD cache as well...and I zfs sync the snapshots via a 10GB Mellanox (point-to-point)...so I do have the current dataset on a snapshot(s) on the backup FreeNAS.

The production FreeNAS is running out of space, I am at about 93% capacity and I want to add two more 4TB SAS drives, bringing it to a total of 8 drives...per my research I have discovered (unless I missed something) that I cannot simply "add" two more drives to the existing RAIDZ2, but I could create another VDEV and use that to expand the capacity, BUT since I am running RAIDZ2 adding these drives via a MIRROR (since I only have two more drives) isn't really going to get me the additional space I was looking for....SOOOOOO given that I do have a snapshot copy of the almost 14TB of data...I think my BEST option would be to Destroy the existing Pool, re-create a new one...probably with RAIDZ1 (initially I did not have a backup NAS)...then copy the data (snapshots) back to the Production FreeNAS (this take about 21 hours using nc enabled zfs send/receive)????

IF this is indeed my best option...am I going to have any issues "converting" or importing the snapshot(s) into the live data? I noticed people discussing making a copy of the data and/or importing the snapshot...as I will only have about 7TB remaining once the snapshot(s) are moved over...I would not have enough space to keep the data + snapshots...just trying to verify this process/procedure so I don't lose all of my data, or run into an instance where I have the data, but not enough space to "make" it live. I am pretty sure from what I have read, this should not be an issue, but again...I wasn't really able to find someone doing a snapshot restore from a backup NAS (not at least one that seemed solid enough to gamble my data on).

Any words of wisdom would be greatly appreciated...thanks again for sharing the knowledge!

 

[kdragon75]

re-create a new one...probably with RAIDZ1

With 8 large drives, RAIDz1 will be extremely risky. If any two fail at the same time, you screwed. Rebuild times could be days. Thats days you would be without ANY redundancy! Stick with RAIDz2!

 

[Sneffets]

With 8 large drives, RAIDz1 will be extremely risky. If any two fail at the same time, you screwed. Rebuild times could be days. Thats days you would be without ANY redundancy! Stick with RAIDz2!

Good point and good information...I think I will stay with the RAIDZ2 :)

Am I correct in assuming the best option would be to rebuild and sync the data back from my backup NAS?

 

[kdragon75]

Good point and good information...I think I will stay with the RAIDZ2 :)

Am I correct in assuming the best option would be to rebuild and sync the data back from my backup NAS?

Yeah, if your pool was only 50% I would say adding vdevs is fine (just not efficient in your case) but at 93% most of the new data would go to the new disks and anything written to the old vdev would likely be highly fragmented and your performance would suck.

 

[Sneffets]

Yeah, if your pool was only 50% I would say adding vdevs is fine (just not efficient in your case) but at 93% most of the new data would go to the new disks and anything written to the old vdev would likely be highly fragmented and your performance would suck.

Since I have to rebuild, I am going to add one (or two more @$150/drive) 4TB SAS Constellation drives so I should have plenty of space going forward...I had everything running in an HP Z800 chassis mod'ed to handle two LSI 8 channel Mega RAID (just popped out the old 4 SATA pre-wired backplane, so nothing major)...so I was out of room, but I found a used SuperMicro board ($100) that will support the LGA1366 Xeon processor and 96GB of 1333 ECC RAM, then I ordered a new E-ATX case (room for 10 - 3.5HDD), 850W Modular ATX-12V PS, and now I can get out of that proprietary tank of a case...LOL

 

[Sneffets]

Ok, I finally got everything backup and running on the Newly remodeled old server...10 x 4TB Constellation SAS drives RAIDZ2...my backup server that I was sending the snapshots to has the data...but I am not sure about the exact "retrieval" process...I assume I just reverse when I seeded the Backup server with the Snapshot, but I noticed that zfs list -t snapshot does not show any snapshots, since the snapshots were from my main server...??? I can browse the actual data that is mounted in the Dataset = BackupData/Replication/Vault1/(various folders)...so I destroyed and re-created the same Dataset structure that I had - DataVault/Vault now do I just copy the actual data reversing the statements below...and can I copy just a folder at a time? Having some issues were it says "cannot open 'the local file that I tried to copy' dataset does not exist???


destination: nc -w 60 -l 3333 | zfs receive -F DataVault/Vault1
source: zfs send BackupData/Replication/Vault1/Folder1 | nc -w 20 172.xx.x.x 3333

I also tried it with a specific file within a folder, and the same message:
source: zfs send BackupData/Replication/Vault1/Folder1/TestFile1.txt | nc -w 20 172.xx.x.x 3333

I am pretty sure I had the key authentication working before...meaning when I ssh'd or zfs send from the Primary server to the backup...I did not have to enter any username/password, but I can't seem to get it working again, despite reading through several articles on the web...I know I have to copy the ssh/rsa public key from the source server to the destination server...but I am not doing something correct...

 

[danb35]

First, zfs send operates on snapshots, not directories or files. If you don't have any snapshots, you can create them with zfs snapshot. Second, snapshots are taken of datasets, not directories or files.