Out of Space

[Dudde]

Hi Im having problem grasping how i should setup my dataset and ZVOL in the correct way.
Im having 4 1TB WD RED mirrored accordingly:

The Volume and dataset look like this:

As you can see the dataset HVG-DATA is at 94% capacity.
I have created two ZVOL's one "HVG-DATA" see belove:

I thought this should give me 500GB of free space for my snapshot but i guess the snapshot is stored in the free space of the ZVOL?

This is how the iSCSI share look inside ESXi:

Here's another picture with more details.

I must have got some of the basics wrong so could anyone explain to me how i should setup everything correct so can get more then my 91.6GB of free space for snapshots?
I know you should always try to keep the zpool at maximum 60% preferable 50% of it's capacity.

How should i have done from the beginning to get this right?
I need 1TB of storage for my ESXi datastore the rest was supposed to free for snapshot etc.
The HVG-EKO is a last minute ZVOL at 60GB that was created when we found out that a software we use needed a block device to be able to install and it needed to be installed on a separate disk then the DATA volume or the SYS volume.

Please help me uderstand the basic around how i should setup my VDEV's zpools and dataset

 

[Vito Reiter]

Well, you've mirrored 4x1TB Drives, You're total RAW space available is only going to be about 2TB and then some room for ZFS to work because it is a little space hungry. The volume itself will show the total space so ~4TB, the datasets themselves will show that lower amount of space because of the redundancy you have setup.

You have found yourself in a situation that I've been in before > You need a different form a redundancy and have no space to transfer and move stuff around. You have a few options here:

• Increase the volume size by replacing those drives with bigger drives one-by-one
• Invest in a new volume such as a 5 disk RaidZ2 (Although this doesn't follow the maximum space algorithm) it'll provide you with much more space and the same amount of redundancy for the cost of a 1TB hard drive

Your space is equivalent to mirroring 2x2TB drives and the ~1.7ish TB that you have is an expected amount. The total space of the volume won't increase until every 1TB drive is replaced, but unless you have HDD space to create a new volume and move everything over replacing drives is your only option. I did notice that the Raid-5 variances such as RaidZ1/2/3 usually provide more space than expected whereas mirroring provides less space than expected.

When I had accidently run a 5 disk volume in Raid-0 for a few months I moved everything to a single drive temporarily in your case 1x2TB drive, created a new volume (5-Disk RaidZ2) and moved everything back to the NAS with SSH/rsync commands (Note: rsync should only be used when dealing with your data solely in the same filesystem i.e ZFS)

Anyway, good luck man, but from what I see the amount of space in your dataset is what's to be expected from a ZFS mirror and I would start taking advantage of the maximum storage redundancies like RaidZ2/3 I can promise you it's worth the extra drive(s) you'll need and you'll see an increase in performance and if using RaidZ3 (Which will only be more space if you use 6 Drives+) better redundancy as well.

 

[Dudde]

I dont want to run RaidZ 1,2,3 since i read mirrored devices should give better iops, now i know i dont run VM's under this ZVOL but it's devided between 4 diffrent companies and has lots of read and writes. Plus the data integrity is important

 

[Vito Reiter]

I dont want to run RaidZ 1,2,3 since i read mirrored devices should give better iops, now i know i dont run VM's under this ZVOL but it's devided between 4 diffrent companies and has lots of read and writes. Plus the data integrity is important

Alright, so your best option is to replace the disks with bigger ones. Your volume doesn't seem to be too far off what is expected but I may be missing something here. ZFS does require a little extra room to work opposed to other filesystems but if someone else could chime in and tell me if I'm wrong then that would help us both.

Hope you figure this out,
Vito

 

[Dudde]

I thought i did save some space when i created my HVG-DAD ZVOl at 1.16TB and HVG-EKO at 60GB in a zpool with a capacitive of 1.8TB?

 

[Jailer]

If you just need more space add another mirror vdev to your pool using the appropriate size disks to give you the free space you need.

 

[Dudde]

Well I know I can replace all the disks in a vdev to expand it I have tried it before. But 4x1TB = 1.8TB usable space should be enough. The data take up around 450GB so we thought double that space should be enough for a good few more years before having to update the machine. But the problem is that I only have 90GB of free space for my snapshot even do you can see in the picture above that 1.4TB is free out of 1.8TB. I thought I should at least have 5-700GB free since the ZVOLs reserves around 1.2TB

 

[Robert Trevellyan]

data take up around 450GB

the ZVOLs reserves around 1.2TB

You're using about 1.65TB out of 1.8TB, not including filesystem overhead, with about 0.1TB free. Everything appears to add up. You need more capacity.

 

[Dudde]

So the zvol HVG-DATA where the 450GB of data is stored use 1.65TB of storage space? I thought when i create a ZVOL with a size of 1.17 TB and then inside ESXi create a datastorage LUN out of 1TB from that ZVOL, how can that add up to 1.65TB of used space?
Shouldnt the 1TB data storage be about 45% full and the vdev should still have about 700GB of the space for snapshots etc?

 

[Robert Trevellyan]

Here's another picture with more details.

The image you posted shows 424GB in use (USEDDS) and a further 1.2TB reserved (USEDREFRESERV).

 

[Dudde]

Shouldn't the 1.2TB reserved space (HVG-DATA ZVOL) contain the 424GB? And the free space outside the ZVOL should be ~600GB?
Theres no data stored on the disk except for the data inside the ZVOL where a .vmdk file at 1TB is created and there by ~200GB of free space should exist inside the ZVOL. The virtualdisk file at 1TB is filled with 426GB of data.

 

[Robert Trevellyan]

Take a look at some of these threads:
https://forums.freenas.org/index.php?search/8677367/&q=USEDREFRESERV

 

[ccssid]

https://jsfiddle.net/Biduleohm/paq5u7z5/1/embedded/result/

 

[Dudde]

Take a look at some of these threads:
https://forums.freenas.org/index.php?search/8677367/&q=USEDREFRESERV

Thank you that was very helpful.