Pool Layout for SSDs and HDDs - Separated zpools

[Shigure]

I‘m new to ZFS too and I will experiment with 4x 4TB HDDs and 2x 1TB SSD to determine the best vDev design.
Special vDev for Metadata + small files with SSDs and either RAIDZ2 or striped mirror for HDDs.
Very curious about speed differences

In theory 4x RAIDZ2 gives you 2x read speed, and 2x SSD mirror also gives you 2x read speed and they both have no write speed benefit(if my understaning is correct). But SSDs are faster than HDD for sure XD.

For a relatively high-capacity small form factor system, a 3-way mirror is also an option. Together with e.g. a 1 TB SSD that would make a nice all-in-one system for SMBs, once TrueNAS Scale is mature enough.

At least here in Europe, with surging prices for electricity, the number of drives is increasingly becoming a factor to consider.

3-way mirror is an option and I did consider it, while with the Node 304 has 6 drive bays so technically I can build a stripe zpool with 2x 3-way mirror vdevs, the performance boost cannot justify the cost and capacity lost anymore. And yes more drives will consume more power, generate more heat and maybe more noise. Hope everything goes back to normal soon in Europe, US finally got back to some reasonable gas price(at least in my area).

50% is not THAT bad, tough I would concur that 6-8 wide is the sweet spot for raidz2.

The Node 304 has 6 bays.
With two bays adapted to hold 4*2.5" that leaves a 4-wide raidz2, which would make for small and secure setup.
To use all bays for 3.5", you may shift the app pool to a single NVMe drive (with backup to the HDD pool), or, if the board can bifurcate x8x4x4, use this adapter to run the HBA and a mirror of M.2 NVMe drives for the app pool. This would cost more to set up, and I have no idea where the financial sweet spot is: Four 16 TB or six slightly smaller drives.

I think 6 drive RAIDZ2 will be the sweet spot as well, with relatively balanced cost and capacity efficiency.
My MB does support PCIe bifurcate to x8x4x4 though I didn't really consider it because the card itself is $100, and I need another riser cable with added additional cost. Maybe some day I will get one of this so it can run the HBA and NIC then add an additional M.2(also frees the backside M.2, oh I suddenly have so many empty M.2 slots lol).
If we aim for the same capacity then it will be 4x 16TB or 6x 8TB, take WD Red Pro official price for calculation, the prices are 4x $300 vs 6x $220 which means without deal the cost of more drives will be slightly higher, but in theroy go with 6 drives you do get read speed gain? But in real world I'm not sure how much difference I will see.

 

[joeschmuck]

Just to make sure you are aware, a properly designed vdev of HDD's will blow the SSD's out of the water. Of course a properly designed SSD would kick some butt too. So do not think that HDD's are slow in this application, they are not. The major downside to a SSD is it's limited number of erase cycles, where the HDD does not have this limitation. This is key to know if you are developing a NAS that will be writing a lot of data, and if you do want SSD's then there are some designed for this high write capability.

A little more about my story, I started with four 2TB HDD's in many different configurations and after goofing around with them for several months, I ended up rebuilding my pool into a RAIDZ2 using six 2TB HDD's. Four years later when my drives warranty had expired by 1 year, I changed my setup around to four 6TB HDD's. I gained a little more capacity even though I didn't actually need it, and lowered my power consumption and heat generation. The bad side was if I need to replace a drive, it now takes up to 3 times longer to resilver and this is a vulnerability. I also slightly lost some throughput speed for data but if I didn't test it, I would never had known in my day to day use. My use case does not require ultra high speed access. I did still maintain full 1 Gbit throughput over Ethernet for both reading and writing large files.

So you could change things up if you feel like it at a later date but the downside is you need to backup the data or transfer it somehow. I backed up my data twice, once to an internal HDD, the second to a USB external hard drive. The first was of course much faster and what I used to restore my data to the pool. Again, I didn't have a lot of data. If you have 20TB, that is a lot of data so these things should be considered.

I highly recommend that you play around with TrueNAS and the various vdev configurations before placing much data on your pool.

We are throwing a lot at you but you are asking questions and I think you want the best bang for the buck, like most of use desire. And the best design. But the best design is subjective and depends on how you use it.

Take your time and make a good decision, after all you will need to live with it for a while.

 

[Shigure]

Just to make sure you are aware, a properly designed vdev of HDD's will blow the SSD's out of the water. Of course a properly designed SSD would kick some butt too. So do not think that HDD's are slow in this application, they are not. The major downside to a SSD is it's limited number of erase cycles, where the HDD does not have this limitation. This is key to know if you are developing a NAS that will be writing a lot of data, and if you do want SSD's then there are some designed for this high write capability.

A little more about my story, I started with four 2TB HDD's in many different configurations and after goofing around with them for several months, I ended up rebuilding my pool into a RAIDZ2 using six 2TB HDD's. Four years later when my drives warranty had expired by 1 year, I changed my setup around to four 6TB HDD's. I gained a little more capacity even though I didn't actually need it, and lowered my power consumption and heat generation. The bad side was if I need to replace a drive, it now takes up to 3 times longer to resilver and this is a vulnerability. I also slightly lost some throughput speed for data but if I didn't test it, I would never had known in my day to day use. My use case does not require ultra high speed access. I did still maintain full 1 Gbit throughput over Ethernet for both reading and writing large files.

So you could change things up if you feel like it at a later date but the downside is you need to backup the data or transfer it somehow. I backed up my data twice, once to an internal HDD, the second to a USB external hard drive. The first was of course much faster and what I used to restore my data to the pool. Again, I didn't have a lot of data. If you have 20TB, that is a lot of data so these things should be considered.

I highly recommend that you play around with TrueNAS and the various vdev configurations before placing much data on your pool.

We are throwing a lot at you but you are asking questions and I think you want the best bang for the buck, like most of use desire. And the best design. But the best design is subjective and depends on how you use it.

Take your time and make a good decision, after all you will need to live with it for a while.

Thanks and I totally agree. My claim about SSD faster than HDD is more talking about the theoretical preformance of the examples in that post with 4x 1TB HDDs and 2x 1TB SSDs. In real world it will be much more complicated when taking vdev config, drive type and interface into account.

Now I have a 4x SSDs stripe mirrors pool and a 3x 8TB HDDs(I will return them to my friend eventually) RAIDZ1 pool running for my test drive. When I finally got the new HDDs I plan to run the disk burnin (https://github.com/Spearfoot/disk-burnin-and-testing) first then config it to the way I want(RAIDZ2 for sure).

As for the backup/transfer plan, what I thought was: data will be uploaed to TrueNAS over network. When it's done and I need a copy for transfer, I will add a new pool with a single drive vdev then send data to it(might need multiple single drive pools though, which is a little risky). After that I can do what ever tweak for my original pool, when it is back online/a new pool is added I can copy the data over. Is it the correct way to do it? I think it's better than download files via SMB to a Windows machine then upload them back, but I will keep the SMB way as an alternative.

 

[joeschmuck]

Is it the correct way to do it?

There is more than one way to get the job done. I would never try to transfer a large amount of data over Ethernet (SMB) when an option exists to transfer to a hard drive directly. As for risk, there is always risk and all you can do is try to minimize it. For example I told you that I created two copies of my data before replacing my drives and creating a completely different vdev. I was able to use the internal drive alone but I did have a second backup just in case some files were not readable.

Another option, and I'm not sure what hardware you have but if you can attach your new pool while the old pool is present, you could transfer all the files that way. That would be ideal of course but most of us don't have that many ports available unless we add-on an extra HBA card, and then figure out how to provide the extra power connectors. Could be a mess. If I have a 12 bay drive then I could do stuff like that, but then again I don't want a 12 bay drive case. In my next iteration of my replacement pool, I might just go with a three drive mirror. It depends on the cost of the drives and the capacity I desire. Since my drives are two years out of warranty, I should start considering what I want.

 

[DigitalMinimalist]

Interesting topic:
I would like to make a backup of my video (several >4GB) and photo library from a ZFS pool. I consider it nearly static, as my whole local movie collection (thx Netflix) will not be extended and old pictures will not be changed.
I would love to copy directly from ZFS pool via SATA to the backup drives and it would be perfect if these backup drives will be readable later in Windows too.

NTFS seems the only simple solution / filesystem to achieve that… correct?
Speed is not important- only compatibility

 

[joeschmuck]

NTFS seems the only simple solution / filesystem to achieve that… correct?
Speed is not important- only compatibility

So I'm going to have to claim ignorance here as to how best move those files. I'm not a FreeBSD nor Linux expert. I would expect you would be able to create an NTFS file system on a drive, attach the drive to TrueNAS, and then copy the files over, but since we are dealing with TrueNAS I'd actually have to do a transfer myself to ensure it actually works. I know in the distant past it could be done, I just don't know with today's products but honestly, I'd give it a try. You may need to mount the drive manually. But if time is not an issue then maybe just placing a large drive into a networked computer and copy the data over? That should be very simple, and time consuming but you would maintain your compatibility. Sometimes the simplest solutions are the best solutions, but maybe not the fastest.

 

[Shigure]

There is more than one way to get the job done. I would never try to transfer a large amount of data over Ethernet (SMB) when an option exists to transfer to a hard drive directly. As for risk, there is always risk and all you can do is try to minimize it. For example I told you that I created two copies of my data before replacing my drives and creating a completely different vdev. I was able to use the internal drive alone but I did have a second backup just in case some files were not readable.

Another option, and I'm not sure what hardware you have but if you can attach your new pool while the old pool is present, you could transfer all the files that way. That would be ideal of course but most of us don't have that many ports available unless we add-on an extra HBA card, and then figure out how to provide the extra power connectors. Could be a mess. If I have a 12 bay drive then I could do stuff like that, but then again I don't want a 12 bay drive case. In my next iteration of my replacement pool, I might just go with a three drive mirror. It depends on the cost of the drives and the capacity I desire. Since my drives are two years out of warranty, I should start considering what I want.

Ideally it will be great if I can mount the new pool with the old one but yes I will need another HBA for that. And sadly people also run out of PCIe slot on their MB. With copies of all the data, in the future maybe I will choose the replace larger drive one by one route unless I want to reconfigure the vdevs...