Special vdev to speed up deleting large files (1TB+)?

[Videodrome]

The majority of discussions regarding special vdevs seem to focus on read and write performance, leaving me uncertain about what to anticipate in terms of delete performance.

Occasionally, I need to delete rather large files, exceeding 1TB in size.
From my past encounters, this action has the potential to cause an existing server to lock up, possibly even disconnecting clients.

Would adding a special vdev potentially resolve this issue?
Or are there other solutions available to address this problem?

Thank you.

 

[Arwen]

First, a Special Metadata vDev has to be added before the files are written, so that the metadata for those files can be written to the Special Metadata vDev. Pre-existing files would not benefit from a Special Metadata vDev as much.

Next, the redundancy of any Special vDev, (Metadata, small file or De-Dup), should be the same as main data vDevs. So a RAID-Z2 data vDev, would suggest a 3 way Mirror Special vDev, (which would be able to loose 2 disks without data loss).

If I remember correctly, there was an OpenZFS feature being developed similar to async destroy. I guess you could call this "Async Delete". Basically it puts files to be deleted on a hidden list, to be deleted as I/O time permits. This might help in your use case. Of course, for small files, "Async Delete" would be worthless. I don't know the status of this feature.

As for the original question, will a Special Metadata vDev help deleting 1TB files, I don't know. Perhaps someone else can answer the main question.

 

[Peced1955]

Locking up during large file deletes suggests another issue might be at play. It could be CPU limitations, insufficient RAM, or even a storage controller bottleneck.

 

[Videodrome]

My Synology NAS used to drop the 10GbE connection whenever I attempted to delete a file larger than 1TB. I made two changes to address the issue, although I wish I had implemented them separately to identify the exact fix. The changes were:
1. Pinning the metadata to an NVMe cache.
2. Using an SSH connection to the 1GbE NIC to delete the file.

Since ZFS lacks the NVMe cache feature found in Btrfs, I'm concerned that the issue might return.
That's why I'm considering whether a special vdev mirror could be beneficial, or if it's just a waste of resources and deletion would work fine without it.

 

[Davvo]

I don't think a special VDEV would help you much. What are you system's specs and your use case?

@Peced1955 answer is not trivial.

 

[Videodrome]

Xeon E5-2699 v3,
96GB ECC,
Intel X540-T2,
LSI 9300.
12-disk raidz2 (16TB HDD)
12-disk raidz2 (14TB HDD)

A single user primarily storing video files, with occasional file sizes reaching up to 10TB, those files are re-encoded and then deleted.

My inquiry about special vdevs is purely speculative at this point; The system isn't ready yet, so I can't conduct any testing.

My main concern is ensuring uninterrupted operation during the deletion of these large files. I aim to prevent any interruptions to upscaling or encoding processes on my client machine, as this could result in the loss of potentially days of processing work. Slowing down is fine, disconnection is not.

 

[winnielinnie]

A single user primarily storing video files, with occasional file sizes reaching up to 10TB, those files are re-encoded and then deleted.

Files that can be as large as 10 TB? You sure you didn't mean 10 GB? Regardless, you might want to use a larger recordsize. I'd suggest 1 MiB recordsize if you primarily deal with gigantic media files.

 

[Davvo]

Special VDEVs generally help with small files. How are you sharing the datasets? SMB? You might want to try deleting a file using ssh: see if you experience the same freezing.
I have a hunch it might be SMB related... You are not using SMR drives, are you?

 

[Videodrome]

10 TB files yes, plan is to use 1MiB recordsize without any compression.
No SMR drives.

I am sharing it with SMB, but I'll use ssh to delete the large files like I do now.
And FTP for transferring everything from the old to the new system.

Good to know the Special VDEV won't matter in this case. Saves me about $1000.