SOLVED Same file in different folders with different names without wasting space?

[CheeryFlame]

EDIT: I made this script with the help of MageMinds and ChatGPT to do exactly what's described down here.

https://github.com/gravelfreeman/qbit-hardlinker

--
Hello!

Id like to build the ultimate Plex Media Server and Torrenting setup on my server and wondered if this kind of setup has already been achieved. Basically the idea is to share back the torrents anonymously while having a clean Plex library without wasting disk space and resources.

Goals to achieve altogether

• Use the same server/storage for the media files
• Torrent client, jDownloader and NZB must go through a VPN for privacy
• Direct access to Plex Media Server outside VPN
• Share torrents AND add media files to Plex library without duplicating data*

* ZFS Deduplication is discouraged here as it wouldn't be viable with CPU/RAM/SSD requirements.

TLDR; Is there a way to have the same file in 2 different folders with different names without wasting storage space and resources?

 

[winnielinnie]

If within the same dataset, you can use "hardlinks" to achieve this.

 

[CheeryFlame]

If within the same dataset, you can use "hardlinks" to achieve this.

Thank you for replying, I didn't knew about hardlinks! I tried to find some information and some BT clients seem to do hardlinking but I'm afraid it's the other way around I'm interested. Given the fact that one day I'll delete the torrent files and stop sharing whereas I'll keep my organized media library.

Basically I would like to download with qBittorrent in

Code:

/mnt/Temp/Shared/TrackerName1
/mnt/Temp/Shared/TrackerName2

Then after download completion, move the source file to

Code:

/mnt/Temp/Downloaded/Music
/mnt/Temp/Downloaded/Movies
/mnt/Temp/Downloaded/TV Shows

Then automatically create a hardlink back in

Code:

/mnt/Temp/Shared/TrackerName1
/mnt/Temp/Shared/TrackerName2

Then when I'm ready I could manipulate the filenames in

Code:

/mnt/Temp/Downloaded

and move them freely in

Code:

/mnt/Media/Music
/mnt/Media/Movies
/mnt/Media/TV Shows

Would this be achievable?

 

[winnielinnie]

Hardlinks don't have a "main" file. Once a hardlink is created, you have multiple pointers to the same file (common inode). You don't even need the same filenames.

For example, "movie1.mp4" and "Home Movie 1.mp4" share the same inode number (as they are hardlinked). Everything is preserved: modification time, ownership, permissions, ACL, etc. (Keep this in mind, especially in regards to permissions and ownership.)

This is only possible within the same filesystem/dataset. Once you traverse datasets, you cannot use hardlinks.

From your use-case, it's getting granular. Are "Shared" and "Downloaded" different datasets in the pool "Temp"?

 

[CheeryFlame]

Hardlinks don't have a "main" file. Once a hardlink is created, you have multiple pointers to the same file (common inode). You don't even need the same filenames.

For example, "movie1.mp4" and "Home Movie 1.mp4" share the same inode number (as they are hardlinked). Everything is preserved: modification time, ownership, permissions, ACL, etc. (Keep this in mind, especially in regards to permissions and ownership.)

This is only possible within the same filesystem/dataset. Once you traverse datasets, you cannot use hardlinks.

From your use-case, it's getting granular. Are "Shared" and "Downloaded" different datasets in the pool "Temp"?

Thank you for your detailed reply. I made a dataset for everything, even subfolders so if I understood correctly I won't be able to use hardlinks. Is there the equivalent of hardlinks but that's able to traverse datasets? Here's a screenshot of my current structure, any advice are welcomed!

 

[winnielinnie]

There's a lot of back and forth.

Some of this is achievable with symlinks, to traverse beyond filesystems. However, symlinks depend on the original file to retain its name and path.

What you could possibly do is minimize some of these steps.

Let qBittorrent download to /mnt/Eden/Temp/Shared/

This can be where the real "source" files will forever live.

Then you manually (or cleverly?) have a folder of the same name created under /mnt/Eden/Temp/Downloaded/ for each of the torrents. (qBittorrent allows you to "run a command" upon completion, making this possible.) Whether manually, or with a script, you'll now have a folder under "Downloaded". From here you can create symlinks to each relevant source file, and name them anything you want.

You can then create symlinks of the source files for the "Media" folders. (Symlinks can be renamed as many times as you want. As long as the source files stay put.)

This means the Plex and qBittorrent jails need the proper mountpoints and permissions.

Your setup makes this impractical, though. Especially if you want to change the names of folders for multifile torrents, while using symlinks.

 

[CheeryFlame]

Thank you for providing all that information. I'll think about it but it looks like I should rethink my directory structure since hardlinks would be easier to work with. They just make more sense since I'll keep my organized media library longer than the torrents. Probably something like so would be more viable;

Code:

/mnt/Media/Music
/mnt/Media/Movies
/mnt/Media/TV Shows
/mnt/Media/Shared/Tracker1
/mnt/Media/Shared/Tracker2

I have one last question, what will happen when I'll do a Google Cloud Sync task on that media dataset? Will it understands it's hardlinks or will it upload all the files twice?

 

[winnielinnie]

Will it understands it's hardlinks or will it upload all the files twice?

Another wrench in the machine! I believe it will view them as separate files (and hence upload them "twice, thrice, etc".) This is because the cloud storage service has no clue/support about hardlinks on your local filesystem. (I don't know of any popular cloud storage service that preserves hardlinks, except maybe rsync.net?)

If you were using rsync or ZFS replication, then hardlinks can be preserved. But for Google Cloud, Amazon, Dropbox, Backblaze, etc? Nope.

In your case, you would have to use the "Exclude" option in your Cloud Sync Task to skip entire folders. Perhaps, skip the "Shared" folder entirely.

 

[CheeryFlame]

except maybe rsync.net?)

$1500 USD per month, I can't afford this solution haha!

In your case, you would have to use the "Exclude" option in your Cloud Sync Task to skip entire folders. Perhaps, skip the "Shared" folder entirely.

I guess that's the best way to do it.

Well it's a week of changes. I was really decided on Core but yesterday I decided to go with Scale even with the performance hit. Today I'm changing my whole dataset structure! Thank you for your time, was really helpful!

 

[winnielinnie]

Just a heads up, due to the nature of multifile torrents and how often torrents use their own folder/subfolder structure, it'll be tricky to try to "automate" hardlink creation upon download completion. You might just have to resort to manually doing it when needed in the command-line.

So let's say you download with qBittorrent "[COOLRIPS] Public Domain Two-Part Movie (2023)". (This does not take into account permissions or the mountpoints of jails!)

It downloads to:
/mnt/Eden/Media/Shared/Tracker1/

Which yields the new folder like this:
/mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/

And within this folder are some files:
/mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/PDTPM-part1.mp4
/mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/PDTPM-part2.mp4
/mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/PDTPM-subtitles.zip
/mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/info.txt

You then create a new folder to accept this newly acquired media, using a cleaner name:
/mnt/Eden/Media/Movies/"Public Domain Two-Part Movie (2023)"/

And now in here is where you start creating hardlinks, using "clean names", and for only the files you care about:

Code:

cd /mnt/Eden/Media/Movies/"Public Domain Two-Part Movie (2023)"/

ln /mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/PDTPM-part1.mp4 "Public Domain Two-Part Movie (2023) Part 1 of 2.mp4"

ln /mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/PDTPM-part2.mp4 "Public Domain Two-Part Movie (2023) Part 2 of 2.mp4"

 

[CheeryFlame]

I thought qBitorrent could just create hardlinks of everything in the torrent in a temporary folder and then from that folder I could just delete the files I don't care about and then freely move and rename my files to whatever I want. That way I don't have to create the hardlinks myself and I can manage the files as I want to. Once I'm done seeding, I could just delete the files in Shared and no hardlinks would be left.

I also use an Archive folder for data hoarding about artist I like and collect stuff about. Mostly it's stuff that wont end in Plex but the official VHS, Laserdisc, DVD and Blueray will. I'll hardlink those as well. Pretty sure I'm on the right track with these hardlinks.

 

[CheeryFlame]

Just a heads up, due to the nature of multifile torrents and how often torrents use their own folder/subfolder structure, it'll be tricky to try to "automate" hardlink creation upon download completion. You might just have to resort to manually doing it when needed in the command-line.

So let's say you download with qBittorrent "[COOLRIPS] Public Domain Two-Part Movie (2023)". (This does not take into account permissions or the mountpoints of jails!)

It downloads to:
/mnt/Eden/Media/Shared/Tracker1/

Which yields the new folder like this:
/mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/

And within this folder are some files:
/mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/PDTPM-part1.mp4
/mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/PDTPM-part2.mp4
/mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/PDTPM-subtitles.zip
/mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/info.txt

You then create a new folder to accept this newly acquired media, using a cleaner name:
/mnt/Eden/Media/Movies/"Public Domain Two-Part Movie (2023)"/

And now in here is where you start creating hardlinks, using "clean names", and for only the files you care about:

Code:

cd /mnt/Eden/Media/Movies/"Public Domain Two-Part Movie (2023)"/

ln /mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/PDTPM-part1.mp4 "Public Domain Two-Part Movie (2023) Part 1 of 2.mp4"

ln /mnt/Eden/Media/Shared/Tracker1/"[COOLRIPS] Public Domain Two-Part Movie (2023)"/PDTPM-part2.mp4 "Public Domain Two-Part Movie (2023) Part 2 of 2.mp4"

Just finished installing and getting used to qBittorrent app and I understand more your message here. Since qBittorrent is able to run a script after torrent completion would it be possible to create a script that analyzes directory structure of

Code:

/share/torrentname
   /share/torrentname/season 1
   /share/torrentname/season 2
   /share/torrentname/season 3

and replicate it in

Code:

/temp/torrentname
   /temp/torrentname/season 1
   /temp/torrentname/season 2
   /temp/torrentname/season 3

and then create hardlinks of all files in

Code:

/share/torrentname
   /share/torrentname/season 1
         /share/torrentname/season 1/episode 1.mkv
         /share/torrentname/season 1/episode 2.mkv
         /share/torrentname/season 1/episode 3.mkv
   /share/torrentname/season 2
         /share/torrentname/season 2/episode 1.mkv
         /share/torrentname/season 2/episode 2.mkv
         /share/torrentname/season 2/episode 3.mkv
   /share/torrentname/season 3
         /share/torrentname/season 3/episode 1.mkv
         /share/torrentname/season 3/episode 2.mkv
         /share/torrentname/season 3/episode 3.mkv

I'm pretty sure someone could code such a script, I'm even willing to pay for this as I really need this feature.

Let me know if you've got any ideas!

 

[CheeryFlame]

I made this script with the help of MageMinds and ChatGPT to do exactly what's described in this thread.

https://github.com/gravelfreeman/qbit-hardlinker

 

[winnielinnie]

Did you actually try it out?

I'm floored at how seamlessly AI is being used to write programs!

I would test it out myself, but I have already have an established way to manage my torrents.

And of course I would also like to remove or increase the limit of 3 levels of directories.

What is this limit of 3 levels?

I would like this script to be able to send the hard linked files into a given subfolder based on the tags and/or category of the torrent.

Could you not use the %L and %G variables? I would think it gets complicated if more than one tag is assigned to a torrent.

 

[CheeryFlame]

Did you actually try it out?

It's in production since 2 days and haven't failed me a single time yet!

I'm floored at how seamlessly AI is being used to write programs!

I am as well, this feels like black magic to me.

What is this limit of 3 levels?

The script re-creates up to 3 levels of subfolders and when you ask chat gpt for more the script breaks.

Could you not use the %L and %G variables? I would think it gets complicated if more than one tag is assigned to a torrent.

Yes that's the idea. I already thought about multiple tags. Actually it would be best to have a config file as well to set the destination directory and set tags and categories to go in a certain folder only if set in that config file. The script would ignore tags and categories not specified in the config file. I have great ideas but I don't know how to bring them to life.