Need advice on volume options

[Apollo]

I currently have a volume composed of 6*4TB WD RED in RAIDZ2 configuration.
My setup is somewhat fine, but I think I would benefit from a higher iop/s.

I use my Freenas server for pretty much everything, but more recently I have been using Owncloud/Nextcloud more extensively and I believe a different pool configuration might help improve responsiveness of the system in general.

I need to increase my pool size a bit, but also need the extra iop/s.

I am going to order extra drive so that I can change my pool configuration. I have the entire pool replicated to backup drives and my intent is to destroy my current 6*4TB RAIDZ2 and reconfigure it as multiple vdev when I get the new drives and replicate to the new pool via replication from backups.

I am looking primarily at redondency and capacity but I would like some feedback and advice to explore other possible alternatives.

I have the following two options in mind:

1) Two vdev comprising of 5 disk RAIDZ2 (3 data + 2 redundancy) = Twice iops/s for 24TB data storage
2) Three vdev comprising of 3 disk RAIDZ1 (2 data + 1 redundancy) + 1 spare = Three times iop/s for 24TB data storage

I have been looking at the various forum threads over the years as well as the PDF/powerpoint presentation about ZFS from Cyberjock, but I am still unsure about the best approach to my setup.

With solution 1) I maximize redundancy while solution 2) improves iop/s to the cost of redundancy. However, because of the lower drive count of solution 2) redundancy may not have to be so stringent or should it?

My plan is to increase the drive count from 6 to 10 drives to achieve increase in capacity but also iop/s on my existing Xeon system using only the motherboard SATA ports.
This will require a mix of heterogeneous LSI 8 port with 2 of the Intel. I do not expect issue there as I am using the non LSI port, but I am unsure if I will get a performance hit.

It is my understanding that having increase Vdev will make use of higher number of CPU, which is fine.
I am also going to encrypt my new volume (I have been playing with that as well).
So overall, going with larger number of Vdev would improve performance.

What is the opinion of the more experience members here?
Owncloud/Nextcloud is used mostly to store photos and videos and have them shared between many relatives so increase in iop/s should prove beneficial in this aspect and the amount of stored data is increasing.

Any advice in the better course of action?

 

[Spearfoot]

I currently have a volume composed of 6*4TB WD RED in RAIDZ2 configuration.
My setup is somewhat fine, but I think I would benefit from a higher iop/s.

I use my Freenas server for pretty much everything, but more recently I have been using Owncloud/Nextcloud more extensively and I believe a different pool configuration might help improve responsiveness of the system in general.

I need to increase my pool size a bit, but also need the extra iop/s.

I am going to order extra drive so that I can change my pool configuration. I have the entire pool replicated to backup drives and my intent is to destroy my current 6*4TB RAIDZ2 and reconfigure it as multiple vdev when I get the new drives and replicate to the new pool via replication from backups.

I am looking primarily at redondency and capacity but I would like some feedback and advice to explore other possible alternatives.

I have the following two options in mind:

1) Two vdev comprising of 5 disk RAIDZ2 (3 data + 2 redundancy) = Twice iops/s for 24TB data storage
2) Three vdev comprising of 3 disk RAIDZ1 (2 data + 1 redundancy) + 1 spare = Three times iop/s for 24TB data storage

I have been looking at the various forum threads over the years as well as the PDF/powerpoint presentation about ZFS from Cyberjock, but I am still unsure about the best approach to my setup.

With solution 1) I maximize redundancy while solution 2) improves iop/s to the cost of redundancy. However, because of the lower drive count of solution 2) redundancy may not have to be so stringent or should it?

My plan is to increase the drive count from 6 to 10 drives to achieve increase in capacity but also iop/s on my existing Xeon system using only the motherboard SATA ports.
This will require a mix of heterogeneous LSI 8 port with 2 of the Intel. I do not expect issue there as I am using the non LSI port, but I am unsure if I will get a performance hit.

It is my understanding that having increase Vdev will make use of higher number of CPU, which is fine.
I am also going to encrypt my new volume (I have been playing with that as well).
So overall, going with larger number of Vdev would improve performance.

What is the opinion of the more experience members here?
Owncloud/Nextcloud is used mostly to store photos and videos and have them shared between many relatives so increase in iop/s should prove beneficial in this aspect and the amount of stored data is increasing.

Any advice in the better course of action?

Have you maxed the memory on this system? I recommend doing that before going to all the trouble of re-configuring your pool.

 

[Apollo]

I have, as the limit on Xeon E3 is 32GB.
Unless I go to Xeon E5, which I have been exploring from afar, I will not be able to do much about it.
Beside increasing the RAM, the only benefit could be somewhat marginal, as most photos are in the 10-25MB in size. So RAM would need to be substantially larger.
Price and availability of parts, here in Canada is not in my favor. Damn Canadian government for that and accessing high end technology is not as strait forward as it would seem.
I have played with a much older Core Duo and much older drives but I found increasing the number of vdev would help in LAN throughput as well (From the old Core Duo prospective). My current LAN throughtput is close to being maxed out on write to server, but on read there is a drop of maybe 20% on my Xeon system. Not really an issue, but for smaller files, especially when and if I use SVN over the network, or Owncloud/Nextcloud, LAN throughput drops much lower.
I am hoping by braking the volume in 2 or 3 Vdev, that I would be able to gain better responsiveness.

 

[Spearfoot]

I have, as the limit on Xeon E3 is 32GB.
Unless I go to Xeon E5, which I have been exploring from afar, I will not be able to do much about it.
Beside increasing the RAM, the only benefit could be somewhat marginal, as most photos are in the 10-25MB in size. So RAM would need to be substantially larger.
Price and availability of parts, here in Canada is not in my favor. Damn Canadian government for that and accessing high end technology is not as strait forward as it would seem.
I have played with a much older Core Duo and much older drives but I found increasing the number of vdev would help in LAN throughput as well (From the old Core Duo prospective). My current LAN throughtput is close to being maxed out on write to server, but on read there is a drop of maybe 20% on my Xeon system. Not really an issue, but for smaller files, especially when and if I use SVN over the network, or Owncloud/Nextcloud, LAN throughput drops much lower.
I am hoping by braking the volume in 2 or 3 Vdev, that I would be able to gain better responsiveness.

I confess I'm a little stumped.

Is there anything unusual about your network setup? What kind of switch are you using? Have you configured LACP/LAGG or anything else that's exotic?

I won't deny that you'll get more IOPS from multiple vdevs than you're presently getting. But I'm afraid you might be disappointed - that's why I'm inclined to eliminate any other 'gotchas' first, before proceeding with tearing down and rebuilding the pool. If the bottleneck is your network, speeding up the disk subsystem probably isn't going to help.

FWIW: RAIDZ1 isn't recommended for 'large' drives (>= ~1TB), so I recommend using two RAIDZ2 vdevs.

 

[DrKK]

Well, Owncloud/Nextcloud are known to be slow as a matter of mathematics, especially when using an SQLite database. Is Owncloud/Nextcloud the only situation where you have objectionable performance, sir?

 

[Apollo]

I ran iperf and I would get almost flat out throughput in excess of 900Mbit/s, but I will have to confirm if the test was done both ways.
I general, I don't have much to complain about in regards to file transfers, especially large files, but when I try to download several files from the same computer, then the network becomes a bit erratic.
With Owncloud/Nextcloud, I can reach top speeds around 600-800MBit/s from files in excess of 100MB (it would seem, but not hard values), but when it comes to smaller files, a few MB and under it slows down drastically. I Know Owncloud/Nextcloud are using both DB and files datasets and that could affect performance. Some of it is also linked to the client operation.
I still need to expend my data storage but I feel reluctent to do so from a 6 disk RAIDZ2
to 7 disk RAIDZ2.
Scrubing with the 6 disk RAIDZ2 does nearly 2 days to complete, more obviously if server is more active. And as mentioned on the forum RAIDZ2 configuration tends to increase wasted space.
So my gola is to break it down to at least 2 Vedvs.