Problems with FreeNAS crashing and losing all data in ZFS volume

[Ken Davis]

For the 4th time, I've had a FreeNAS device crash and lost all the data in the ZFS volume. Can someone please let me know what I'm doing wrong here?

I've got a 45Drives enclosure full of 3TB Western Digital Red drives for a total unformatted available space of 132TB. I've created a single ZFS volume on the device so far of 60TB with 1.2TB reserved. I am presenting the 60TB volume this to my Windows 2008 Server as a iSCSI volume. The server sees the space, formats it as NTFS and can write to it as an iSCSI device. It's being used as library storage space (replacement for tapes) for a CommVault/Sympana backup system.

The volume reaches 95% capacity and then the device crashes - the only fix I've been able to use to get the device back online is to reinstall it from scratch and reformat and reconfigure the device completely over again. Several weeks later when it fills up again it just does the same thing. Is there something I need to do to tell the device to reserve space for something it needs that it does not have enough of that is causing it to crash this way?

Can someone tell me what is being done wrong here?

 

[anodos]

You shouldn't fill a volume to 95% capacity.

 

[gpsguy]

You have several problems.

1) 45 drives is far too wide ...

2) For best performance with iSCSI you should be using striped mirrors.

3) For iSCSI you shouldn't use more than 50-60% of your volume.

Do a Google search of the forum for iSCSI in the past 60-90 days.


Sent from my phone

 

[cyberjock]

In fact, more than an 11 disk RAIDZ3 is too wide... LOL. And you are more than 4 times that.

 

[Ken Davis]

In fact, more than an 11 disk RAIDZ3 is too wide... LOL. And you are more than 4 times that.

The zpool is made up of four 11 disk RaidZ3's presented as one volume.

 

[pschatz100]

I can't help but wonder whether or not the OP is being serious. 45 drives with 132 unformatted capacity would mean 44 drives plus 1 redundant drive. Four 11 disk raidZ3 would nominally be 24Tb x 4 = 96 capacity before formatting. Am I missing something here? Where does a 60Gb volume come from?

It is hard to imagine somebody doing something like this over and over 4 times.

He didn't list his hardware configuration. Maybe he has less than the necessary amount of memory.

 

[panz]

Please post a detailed description of your configuration (hardware + software). We're not going to question you for every detail we need to help you ;)

 

[rs225]

Is it on the latest release? Has a memory test been run?

Are scrubs ever run?

Is it losing the pool, or just your NTFS volume appear to vanish or lose data on the Win server?
What is sync set to on the zvol, and do you have a separate SSD SLOG? Are you formatting the NTFS with 64K or larger clusters and creating the zvol with a 64K+ blocksize (otherwise you really do need to be using mirrors)?

What disk controller are you using?

 

[Ken Davis]

It's vanishing from the windows server not visible as an iSCSI device.

Machine has 32GB of ECC RAM.
Scrubs are run every two weeks.

 

[Ken Davis]

Also, I'm using iSCSI to publish the volume directly using a device extent to a Windows 2008 server. The hardware is a 45 drives appliance purchased from www.45drives.com.
It's a Supermicro motherboard with 32GB of RAM and an i3 processor. All the disks are 3GB WD RED.

 

[Mynorx]

did you find a solution?

 

[cgill27]

I also have a 45Drives enclosure, not as many drives, (36) 3TB drives in (3) 11 disk raidz2's. I don't do any iscsi stuff with it but mine has been pretty rock solid since initial install close to 1 year ago.
I know this doesn't help anything, just throwing it out there that there is someone else with similar hw and running ok.

 

[Ken Davis]

No not yet on the solution. I'm not certain how to do it, but I need to create an 85TB zvol and then a dataset with a 60TB quota on that volume so that Windows can't ever fill up the full 85TB as it only sees 60TB of the available space? Does that sound correct?

What's happening here is that Windows server uses all of whatever is presented. The Zvol fills to more than 85% and then it will become unstable and crash. Can a dataset of a zvol be presented with a quota of say 80% of the zvol using iSCSI?
This space is being presented to windows using iSCSI for large amounts of data storage (backup library).

 

[cyberjock]

zvols aren't volumes. If you set a zvol to 85TB, then you'd better have the required disk space free to hold 85TB of data. I was actually just talking about this subject to someone I was training earlier today. Specifically I was explaining how this kind of mistake can happen and how you must plan ahead or you get to plan twice- once for the first setup, and once for the second (and correct) setup. ;)

Windows (and all iSCSi clients for that matter) are supposed to use all of whatever is presented. If your system can't swallow that much data then you need to figure out what your limit actually is based on disk space, quotas, reservations, snapshot sizes, etc and redo your iSCSI extent accordingly.

You *might* be able to resize the partition down, but no guarantees that will work. I don't do iSCSI on Windows so I have no idea if it will let you or not.

 

[KempelofDoom]

I would suggest you create a dataset and specify a quota that is less than 80% of your entire capacity. Then contact Commvault for advice on configuring it to clear space if it's current backup will exceed the current available space. I've used commvault in the past and their support is top notch.