General upgrade questions

[Kosta]

Hello,

please forgive if some questions are trivial, but I just want to make sure I am going the right way:
My current setup revolves around TrueNAS and X11-based server, which holds two NVMEs which I use as ESXi datastores (no RAID, just simple storages). Note: home setup. TrueNAS is virtualized as a VM, and accesses 8 disks via an HBA directly assigned. Current HBA is LSI 9211-8i.
Current drives are WD Red and some older Seagates combined, I only wanted to test with TrueNAS at first, as I have an older 8-bay Synology which has been serving me a long time well (since 2014). TrueNAS has been fairly stable, I had 1-2 troubles with it, and that is 1-2 troubles more than I had with Synology (which is like *none*). However:
I need lots of storage for general use, and only part of that needs to be backed up. Synology is fine for that. What I need is a performing expandability. And ZFS is a good idea I reckon. Besides, I can always upgrade the hardware, which isn't possible with Synology, I have to buy the whole bay. Besides the server performs way better.
Now, to make room for future upgrades, I would go this way:
I would buy a big 4U storage case like Fantec SRC-4240X07. Can fit 24 drives, has decent cooling. And I would need 2 additional HBAs (or an SAS expander).

Now let's get into ZFS:
Currently I have a RAIDZ2 with 8 drives (single VDEV), which I can still reconfigure. Meaning: I can still move the data somewhere else, and do with it whatever I want.
Configuration is 8x2TB, which with RAIDZ2 gives 8TB of free space. I do wonder however, online calculators tell me I should have 10 of free space, yet I only have 8TB? (using this calc https://wintelguy.com/zfs-calc.pl)
However if I would reconfigure it:
With expandability in mind and if understand it correctly, I can only add full VDEVs based on the original VDEV (or simply put all VDEVs must have same sized disks). If I build one VDEV with 5 drives, I can only add full VDEVs, containing of 5 drives (5 because apparently this is the minimum for RAIDZ2). So the route might be 5+5 disks each in their own VDEV, and then adding 5 disks in 3rd and 5 disks in 4th VDEV. However, my understanding is that this is not an optimal setup when it comes to performance and space availability? The more VDEVs I have, to less free space I have for the same amount of disks. Is this correct?
With 5 disks in VDEV, I only get 50% of space availability, so out of 5x2TB, I would really only get 5TB (or most likely less on TrueNAS, see above). This is true also for 8x2TB disk VDEV, which I have now (and I still wonder why is that).

The only way to get "more" (or most?) out of a single VDEV would be going 10x4TB disks, which should get me 26TB of space. Is this correct? 4TB are best value I think currently, but I would needs the case and 2nd HBA (or an expander), to be able to test it out.

In future steps, I would then sooner or later want to upgrade this array of 20 disks. Can I simply upgrade single disks in a VDEV? Say I have a single ZPOOL over 2 VDEVs (10x2TB and 10x4TB). Can I go ahead and replace a disk after disk in either VDEV with larger disks? (thus increasing the size of the whole ZPOOL?)

Or would you rather advise me to build two separate ZPOOLs, each having 10TB disks, disks having different sizes in each VDEV (but same size per VDEV)?
To be honest, I always liked the idea of one big storage rather than more smaller ones. But I could live with two ZPOOLs.
Is there a performance increase/penalty when using one big ZPOOL as compared to two smaller ones?

And finally, would you rather buy two additional HBAs, like Intel M1115 (each going for about €100) or an SAS expander, which is some €120 on ebay (Intel RES2SV240)?

Finally, caching: I read and looked at some videos about write and read cache, and just a simple question: does it make sense to add two single 256GB SATA SSDs, connect them to the mainboard directly and (hopefully be able to) add them to the TrueNAS. Would that improve performance, especially writes?

I hope this covers all my questions :) Sorry for the uber-long post.
Thank you.

 

[danb35]

(or simply put all VDEVs must have same sized disks)

I'm not sure whether you mean that the disks in all vdevs must be the same capacity, or that all vdevs must contain the same number of disks and be of the same type, but neither is correct--though in the latter case the GUI will warn you. It's a valid, though not a recommended, configuration to have, say, one 6x 2 TB RAIDZ2 vdev and one 8x 4 TB RAIDZ3 vdev in a single pool.

5 because apparently this is the minimum for RAIDZ2

No, four disks is the minimum for RAIDZ2.

The more VDEVs I have, to less free space I have for the same amount of disks.

Assuming the same RAID level in any vdev, yes, this is correct. However, increasing the vdev count increases performance, specifically IOPS.

Can I go ahead and replace a disk after disk in either VDEV with larger disks?

Yes. Once you replace every disk in a vdev with a larger one (but not before then), the capacity of the vdev, and thus of the pool, will increase.

Is there a performance increase/penalty when using one big ZPOOL as compared to two smaller ones?

Probably a slight increase, presuming the big pool has more vdevs than either of the two smaller ones.

Finally, caching: I read and looked at some videos about write and read cache, and just a simple question: does it make sense to add two single 256GB SATA SSDs, connect them to the mainboard directly and (hopefully be able to) add them to the TrueNAS. Would that improve performance, especially writes?

Probably not. Particularly for writes, a SLOG (which isn't really a cache at all) has very limited usefulness.

 

[Kosta]

I'm not sure whether you mean that the disks in all vdevs must be the same capacity, or that all vdevs must contain the same number of disks and be of the same type, but neither is correct--though in the latter case the GUI will warn you. It's a valid, though not a recommended, configuration to have, say, one 6x 2 TB RAIDZ2 vdev and one 8x 4 TB RAIDZ3 vdev in a single pool.

This is actually what I meant, one big ZPOOL with VDEVs that have different sizes, eg. 10x 2TB and 10x 4TB. If not recommended, I can do two pools, that would be fine.

No, four disks is the minimum for RAIDZ2.

Sorry, apparently I misread somewhere.

Assuming the same RAID level in any vdev, yes, this is correct. However, increasing the vdev count increases performance, specifically IOPS.

The whole thing is less about IOPS, more about throughput and storage (more space is more important. Since I do need one larger storage and one smaller storage with possibly more performance, it could create 10x4TB for storage, and two 5-disk VDEVs for a 2nd pool, which might be used for ESXi datastore as a data-dump (images, offline VMs etc). Productive VMs are exclusively on NVMEs. Backups of VMs (Veeam) will go to the TrueNAS store, and backup of certain things on TrueNAS will go to Synology and possibly one external disk (what's big and less important).

Yes. Once you replace every disk in a vdev with a larger one (but not before then), the capacity of the vdev, and thus of the pool, will increase.

OK, so it pays off only to buy the whole number of disks for the VDEV, no step-by-step upgrade like I can do with Synology now.

Probably a slight increase, presuming the big pool has more vdevs than either of the two smaller ones.

This is defeated by the previous statement: I will never buy 20 disks at once. 10 is already a hard thing to push with my wife

Probably not. Particularly for writes, a SLOG (which isn't really a cache at all) has very limited usefulness.

Too bad, but thank you.

And thank you for taking time to answer everything!

 

[danb35]

one big ZPOOL with VDEVs that have different sizes, eg. 10x 2TB and 10x 4TB

This is not ideal, but it's perfectly usable (that's how my pool's built).

whole number of disks for the VDEV, no step-by-step upgrade like I can do with Synology now.

You can upgrade disk-by-disk over as much time as you like, but you won't see any increased capacity in the vdev until all disks are upgraded.

If your concerns are performance and capacity, you're kind of at cross-purposes. Performance typically means lots of vdevs, and mirrors are normally recommended there. See also https://www.truenas.com/community/resources/picking-a-zfs-pool-layout-to-optimize-performance.101/ and https://www.truenas.com/community/r...and-why-we-use-mirrors-for-block-storage.112/ (edit: see also https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/).

For capacity, though, you want wider vdevs, though 10 disks wide is about the limit for performance reasons.

 

[Kosta]

For capacity, though, you want wider vdevs, though 10 disks wide is about the limit for performance reasons.

Does that mean that performance suffers as I add more disks to a single VDEV? Currently my 8-disk VDEV is totally OK for what I am intending it to use: multimedia files data-dump. Reads do not happen at larger rate than 100MB/S (Gbit-Network to the Clients), and Crystalmark shows me over 1000MB/s in SEQ read. Writes on this volume will happen without me looking really, so the speed isn't the issue, as long as it's in Gbit range.

BUT: how come that my 8x2TB single VDEV shows only 8TB free? According to calcs, it should be 10TB. I already ordered 2x2TB, just to test out if it will show more when I add them (hopefully adding via SATA will work).

 

[danb35]

how come that my 8x2TB single VDEV shows only 8TB free?

How much is used on it?

 

[Kosta]

How much is used on it?

How do you mean?
Here is a screenshot:

 

[danb35]

So, 1.83 TiB used + 8.38 TiB free = 10.21 TiB. Where's the problem?

 

[Kosta]

Hmm, right. I was reading the DATA, where it says Available. This must be due to iSCSI Volume though.
But, I was extremely sure that as I created the Pool, only 8GB was shown... maybe turning 43 recently killed some grey cells

 

[danb35]

where it says Available

That can be a little confusing. As Free/TrueNAS uses it, "available" is equivalent to "free", which would equal "total" - "used."

 

[Kosta]

Awesome! Thank you. So I have as expected, 10GB. If I add two 2TB disks, with total of 10, I should be in the area of "largest free space for a single VDEV Z2". At least according to the calc. In other words, for my data-dump enough performance, and it is faster than Synology. When 8TB disks become cheaper, I can simply upgrade the whole pool to 8TB and be done with it for a longer while.
The 2nd pool could consist of 8-12 disks in final stage, but I would begin with 2 VDEVs, both Z2. I would also give it Pro-Drives. I would expect better performance from that array, especially if I add 3rd VDEV to it.
Last two bays I could use for hotswaps.
Am I understanding this right?

 

[danb35]

If I add two 2TB disks

How do you propose to do that? You can't add devices to a RAIDZn vdev.

 

[Kosta]

How do you propose to do that? You can't add devices to a RAIDZn vdev.

Create new pool. Data is already down, tomorrow disk arrive and I'll then delete everything and start creating new pool, which will become "productive", so to speak.

 

[danb35]

Yes, that'll do it.

 

[Kosta]

One question though: will I get into problems if I first create the pool with two disks connected to SATA ports on the motherboard (provided that works), and when my SAS expander arrives, move them to the expander? They should the actually automatically appear in TrueNAS...

 

[danb35]

That shouldn't be a problem; TrueNAS is designed to tolerate disks moving around.