VMware NFS 3 data store variations

[JoeAtWork]

Hi All,

I have a 48 drive mirror set that spans 4 external HP ds 2600 shelves. The 4 sas cables go into a LSI 2116 external 4 port card.

Some mirror members have as little as 674G of data while others are as high as 963G of data, the average is 821 gig. :-(

Is this normal or is it causing performance issues?

The server has 192 gb of ram and two 10 core processors, lz4 compression for the data set and is running TN Core 12.0-U8

Thanks,
Joe

mirror 743G 1.99T 11 2 2.55M 972K
mirror 800G 1.94T 12 2 2.80M 1.01M
mirror 844G 1.89T 13 2 2.93M 1.07M
mirror 839G 1.90T 14 2 2.91M 1.06M
mirror 920G 1.82T 15 2 3.20M 1.16M
mirror 847G 1.89T 12 3 2.94M 1.08M
mirror 920G 1.82T 15 3 3.20M 1.17M
mirror 835G 1.90T 12 3 2.90M 1.07M
mirror 802G 1.94T 13 3 2.80M 1.02M
mirror 833G 1.90T 14 3 2.89M 1.06M
mirror 956G 1.79T 15 2 3.33M 1.21M
mirror 795G 1.94T 12 3 2.78M 1.01M
mirror 963G 1.78T 16 3 3.34M 1.22M
mirror 844G 1.89T 13 3 2.93M 1.08M
mirror 836G 1.90T 13 3 2.90M 1.07M
mirror 688G 2.05T 12 3 2.39M 910K
mirror 921G 1.82T 15 2 3.20M 1.16M
mirror 805G 1.93T 12 3 2.81M 1.02M
mirror 777G 1.96T 12 2 2.71M 1010K
mirror 847G 1.89T 13 2 2.95M 1.07M
mirror 848G 1.89T 14 2 2.94M 1.07M
mirror 679G 2.06T 12 3 2.36M 900K
mirror 674G 2.06T 11 3 2.34M 895K
mirror 703G 2.03T 11 2 2.43M 920K

 

[blanchet]

Yes it is normal especially if you have progressively expanded the pool by adding new vdevs,
because ZFS does not rebalance the existing data when adding new vdevs.

There is no major unbalanced between the disks so it should not causing performance issues.

 

[JoeAtWork]

Hi Blanchet,

This was a pool I started with 8 vdevs and kept adding vdevs but this new NFS mount point and disk usage is many months after that.

Should I destroy it and re-make the pool?

Thanks,
Joe

 

[blanchet]

This was a pool I started with 8 vdevs and kept adding vdevs

So it is normal that the older vdevs have more data than the others.

but this new NFS mount point and disk usage is many months after that.

The age of the NFS mount point does not matter for the data balance. Only the vdevs ages matters.

Should I destroy it and re-make the pool?

It does not worth recreating the pool. Just keep it and do not worry about disk balance.
Disk usage will naturally rebalance over the time when the data are rewritten.

 

[JoeAtWork]

So I have done more looking into this issue.

I have now provisioning all the disks at once I still get unbalanced data per vdev.

I attribute this mechanical hard drives are not identical.
(different firmware on the disk, different manufacturers, different cache size on some disks, some disks set to WCE=1 while others in the same pool are set to WCE=0 and some disks doing re-write in place. I am still unsure if the stupid dell background scan is also not causing issues on the Seagate drives.)

Even with one of my filers using all Dell drives with WCE I have variation of 23%.

Where do I see this as a problem? When doing a scub the last 2 hours is scrubbing just ONE vdev! But most important when using VEEAM to backup to tape, the reading of blocks slows down to the speed of a single vdev. :-(

How to resolve this?
1) allow us to re-balance vdevs
2) impose a disk write max thoughput so that all mechanical disks only write at XX megabytes per second.
3) make zfs auto throttle load to the pool so that all disk writes run at the speed of the slowest member in the slowest vdev
4) tell me to go get a VTL so I do a veeam copy to the VTL and then a copy to the LTO tape.

Thanks,
Joe

 

[lucky644]

So I have done more looking into this issue.

I have now provisioning all the disks at once I still get unbalanced data per vdev.

I attribute this mechanical hard drives are not identical.
(different firmware on the disk, different manufacturers, different cache size on some disks, some disks set to WCE=1 while others in the same pool are set to WCE=0 and some disks doing re-write in place. I am still unsure if the stupid dell background scan is also not causing issues on the Seagate drives.)

Even with one of my filers using all Dell drives with WCE I have variation of 23%.

Where do I see this as a problem? When doing a scub the last 2 hours is scrubbing just ONE vdev! But most important when using VEEAM to backup to tape, the reading of blocks slows down to the speed of a single vdev. :-(

How to resolve this?
1) allow us to re-balance vdevs
2) impose a disk write max thoughput so that all mechanical disks only write at XX megabytes per second.
3) make zfs auto throttle load to the pool so that all disk writes run at the speed of the slowest member in the slowest vdev
4) tell me to go get a VTL so I do a veeam copy to the VTL and then a copy to the LTO tape.

Thanks,
Joe

Scripts like this exist, but I wouldn't personally worry about it unless you are experiencing severe performance issues.

It'll sort itself out over time, and data is moved/written.