Help with the ZFS 80% limit

[Rickinfl]

I've read and read and still don't understand this. What I'm doing is not complicated, but every way I do it I get those warnings over 80%. So I figured I'd explain what I'm building and what I'm doing and someone can help me.

I built a FreeNAS box with 12 - 8TB drives that I'm using for Camera's to write too. (I create a separate partition for each camera.)

Now my Pool's name is NVR.

Name Type Used Available Compression
NVR dataset 4.35 MiB 64.2 TiB lz4

So....

79% of (64 terabytes) =
50.56 terabytes

So how to I set my dataset to not exceed this and email me freaking warnings daily?
I've tried Quota for this dataset and reserved space for this dataset and still get warnings.

What I'll do is create 1.5TB datasets under NVR that for each camera.

Now I've done this similar setup and still get warnings.

So if someone can explain how to do this it would be great.

I created the RAIDZ2 with the 12 - 8TB drives. I have 64.2TB available. I know I can't exceed 80% of that. I need to create 1.5TB partitions for the camera's. AND not get freaking warnings.

I'm just so tried of fighting this and have been for almost a year trying to figure out the configuration that works. If I was a programmer I'd rip that freaking warning out of the code.... LOL (I know it's there for protections) But I've gotten so angry at this whole project I wanted too.

 

[PhiloEpisteme]

I built a FreeNAS box with 12 - 8TB drives that I'm using for Camera's to write too. (I create a separate partition for each camera.)

Based on the rest of your post I think you mean you created datasets for each camera, correct?

So how to I set my dataset to not exceed this and email me freaking warnings daily?
I've tried Quota for this dataset and reserved space for this dataset and still get warnings.

Do you have other datasets on the pool? The 80% relates to the usable space on the pool, not the individual datasets which live inside a pool. Consider this case where I have two datasets within a pool, setA and setB. If I set quotas on both to be 10TB and fill them both up the pool will have 20TB stored on it, 10TB for each of setA and setB.

I created the RAIDZ2 with the 12 - 8TB drives. I have 64.2TB available. I know I can't exceed 80% of that. I need to create 1.5TB partitions for the camera's. AND not get freaking warnings.

FreeNAS can be confusing at first if you're coming from other systems. My guess is what you actually want is something more like the following. One pool made up of your 12 drives. It sounds like you're using those 12 drives in a single RAIDZ2 vdev. Within that pool create a dataset, call it whatever you like but here I'll call it cameras. Set a quota on that dataset which is 80% of your available space in the pool. Then, within that dataset you can create child datasets for each camera, perhaps camera1 and camera1. You don't have to set any quotas etc on these child datasets because the quota on the parent dataset will keep them from overfilling the available space.

Using this setup your cameras can all fill their datasets as they please so long as the total data used by all cameras is less than the quota you set on camers. In this scenario one camera could hog all the space. If you want to prevent that you can set individual quotas on the child datasets for each camera. Check out the user guide for your version of FreeNAS or any of the documentation out there explaining quotas vs reserved space in ZFS.

Caveat: If you create SIBLING datasets to cameras and fill THAT up you could certainly go above 80% capacity. If you add datasets as siblings in this way you may want to adjust the quotas on all top-level datasets as you see fit.

I hope this helps. Let me know if you have other questions.

 

[Rickinfl]

Ok. Here is my setup at home.

I have 4 - 4TB hard drives. I created and Raidz1 (Don't really care about the video data)
7.90 was 79% of the total of the Raidz1 it was a little over 10TB.

I have a Pool called NVR, Under that I have a DataSet called NVR.
Under the settings for Dataset NVR I have it set to "Quota for this dataset and all children" 7.90TB.
Then under that dataset I have individual datasets for each camera.
Cam09 with "Quota for this dataset" 490GB

And I still get this email:

Refquota exceed on dataset NVR/Cam09.
Used 95.93% (504.7 GB of 526.13 GB)

What's confusing is this shows under pools in Freenas..
Name, Type, Used, Available.
Cam09 dataset 470.18 GB 19.82 GB

Notice the email Refquota is showing different GB sizes, but under pools it shows it correct.

 

[Chris Moore]

The place to control this is the camera. Configure your camera to use less space.

 

[PhiloEpisteme]

@Chris Moore do you know exactly why the reported numbers are different? I suspect that it is related to reporting raw space vs usable space but I'm not 100% sure.

 

[Rickinfl]

Well first Chris I don't think you understand how this works.

I create a datastore (Cam01) of say 500GB. I login to my camera (Hikvision) I go to Storage Management. I connect it to an Server IP address. Then add the path to the folder/partition I created. I go to HDD Management and in there it shows me a 500GB partition and ask me if I want to format it.

So to your response to me.. No I can not configure my Camera to use less space.

 

[Chris Moore]

Well first Chris I don't think you understand how this works.

I create a datastore (Cam01) of say 500GB. I login to my camera (Hikvision) I go to Storage Management. I connect it to an Server IP address. Then add the path to the folder/partition I created. I go to HDD Management and in there it shows me a 500GB partition and ask me if I want to format it.

So to your response to me.. No I can not configure my Camera to use less space.

I don't do surveillance cameras, so I am sure I don't understand. It sounds like you are sharing raw storage by something like iSCSI if the camera is asking to format the storage. Can you give us some more to go on?

 

[Chris Moore]

@Chris Moore do you know exactly why the reported numbers are different? I suspect that it is related to reporting raw space vs usable space but I'm not 100% sure.

It is probably like the difference between TB and TiB. Just a math conversion.

 

[Rickinfl]

Ok. Let me break this down Simple because this Quota stuff is so confusing.

I have 65 TIB. ZFS will not allow you to use more than 80% or it starts crying like a baby and annoys you continues to email you.

So I created an DataSet called NVR and set the "Quota for this dataset and all children" to 55 TiB.

Now I created Datasets to each camera (which varies per camera) But I set Quota on Cam01 to 1.5TiB.

Not sure about Quota and Quota and children. (ZFS is confusing) but I think I have that part right.

 

[jgreco]

ZFS will not allow you to use more than 80% or it starts crying like a baby and annoys you continues to email you.

ZFS will allow you to use 99+% of your pool storage. It will simply be extremely sucky. In the worst case scenario you can actually get into a situation where you cannot perform metadata updates because ZFS is a CoW filesystem, and if you literally cannot write anything new to the disk, this also means you cannot make any changes, so your pool might be "permanently full."

Because this has the potential to be a catastrophe, FreeNAS warns you about this. It will also warn you if you exceed the threshold on a dataset, because that has a different but also potentially problematic set of issues for administrators. You can disable the warnings if you wish. For datasets, I don't believe it warns except when you actually cross the threshold. I have a mirrors archive dataset that often sits at 100% full without incident, and the only spurious warnings are if I restart the filer.

 

[Rickinfl]

The issue is the 1.5TiB I have the camera write to it. Now it sees the 1.5 and formats it perfect and the camera is set to overwrite when full, but Freenas has an issue with it exceeding 80%. I get the Refquota email, but it's set as Quota only.