Seani
Dabbler
- Joined
- Oct 18, 2016
- Messages
- 41
Peace.
My new RAM (the old one was faulty) is in the testing-stage (memtest86+ with at least 5 passes (ETA 44hrs)) and I am currently working out how I will store, backup and access ~45TB of mostly media files in a smart manner (given my limited resources) in FreeNAS. My family consists of hoarders and tech noobs (bad combination when applied at the same time; that's why I am the one to "make it work, make it easy, make it cheap, make it safe-and don't you dare bother me, I don't understand any of it" - thank you very much, my pleasure)...
I am trying to collect my thoughts on several topics (this is going to be a lot of text since this has been floating around in my head for months) and share them with the community to see if my thinking makes sense and also to see where it all leads. I used the forum search for most of the topics in advance to get an overall image. If I missed something, feel free to link it and I will gladly read it.
Backup:
Is-state:
Right now all files are stored on windows machines (for example: my tower holds 4x 3TB and 1x 8TB and I have 2x 3TB, 1x 6TB and 1x 8TB as external USB 3.0 drives attached, which are only powered on when I perform a backup (and yes, I know how stupid and vulnerable that setup is... hence, FreeNAS). The miracle being that in the 12 years we have been using this concept, we never had a major loss of data.
Since I cannot afford to buy a second, equally sized NAS to simply backup using rsync or zfs-send, I will have to rely on the existing drives for backup purposes (additional to the "safety" of raidZ2, snapshots and UPS).
Could-be-state:
If my understanding is not flawed I should be able to create a dataset on the NAS matching each of my existing drives (5 datasets, 5 backup drives for example).
I could create a CIFS share for each dataset (accessed by a backup user with complete recursive read rights) and then simply mirror the content of these shares to the local drive attached to my windows machine over the network using a software solution for windows.
Possible software:
I read a lot about Macrium Reflect to backup entire disks as one file (not my primary concern; setting up windows is easy and quick) and 2BrightSparks SyncBack for general backup purposes (sounds very promising though pricy). Experiences?
At the moment I am using Beyond Compare to mirror my internal drives to their external counterparts.
Other software suggestions are very welcome.
Upside:
- fairly straight-forward
- datasets including content of the same kind can be united once an external drive dies and gets replaced by a bigger one
- very important data like RAW-pictures can be mirrored to 2 external drives for extra safety (one stored off-site)
- NTFS can be read by most OS
- easily expandable on the backup side
- the backup drives can be stored separately and do not rely on one another
Downside:
- creating the backups is manual labor
- unnecessary datasets due to the limited space of external drives. E.g.: 3 datasets: "MKV1", "MKV2", "MKV3" (to match the drives) where "MKV" would otherwise suffice.
- only one backup at a time (Gigabit LAN)
- no snapshots of the backup drives
- no redundancy of the backup drives
- no detection of bit-rot or other loss of file-integrity
- perils of USB (could be avoided by buying an eSATA cage)
Media streaming:
Is-state:
General
My PC shares the frequently used files (music/movies/tv-shows) via windows shares (read-only). That means that the files are only accessible when I am at home and my PC is on (that was one of the major reasons for a NAS/Server "oh no, he's out, I can't listen to music").
Audio
Music is accessed with Mediamonkey locally on all computers (the music shares are mapped in windows and scanned by Mediamonkey when desired).
Video
Video files are either accessed directly via Windows Explorer and VLC Mediaplayer or via KODI or Plex (both can read metadata stored in the individual movie directory which I manually add with tinyMediaManager).
Could-be-state:
Audio
Local:
Same setup that I am using right now. CIFS music shares on the NAS, mapped on windows machines and scanned and accessed by Mediamonkey.
Internet:
I am looking for a program that will enable me to access the music over the internet (a friend of mine uses Madsonic but I dislike the way it displays music and also it could not handle my amount of tracks (6-digit numbers)).
Does anyone know a way to stream music over the internet and encode it on-the-fly on FreeNAS (mp3 for mobile access)? The solution must be able to handle huge amounts of tracks.
Video
Local:
Same setup that I am using right now. CIFS video shares on the NAS, mapped on windows and scanned and accessed by KODI/Plex/Windows Explorer+VLC Mediaplayer.
OR
Plex/Emby/KODI Server in a Jail or as a Plugin in FreeNAS (I do not know what's better: Jail or Plugin) with Plex/KODI Clients.
Internet:
I read that Plex is a common choice to stream movies over the internet with on-the-fly encoding. Are there any alternatives to Plex (I read the GTC of Plex and disliked the thought that they have information on what files I have, what I watch, when I watch it and even where I watch it.)
Since I am new to streaming over the internet (until september 2015 I had 60kbyte/s UL so I never bothered to look into the subject; now I have 1,7Mbyte/s) I am open to suggestions, places to inform myself, experiences and so on. For testing purposes I installed both Plex and KODI on my tower and accessed it via local client PCs. I liked KODI a bit better for its custumization possibilities.
Temporary Files:
TV recordings:
I use a windows machine with DVB-Viewer (great piece of software btw.) to record movies and TV-shows over satellite. I wondered if it's an issue to store the recordings on the NAS instead of a local HDD in the recording PC (I would dump the 2 old HDDs and instead install a 120GB SSD for the OS to save power). I read that FreeNAS, as a COW System, has issues with fragmentation:
Would using the NAS as a temporary storage for recordings (between 6 and 25GB per file) lead to serious fragmentation or doesn't it matter at all?
Handbrake:
Has anyone got Handbrake running in FreeNAS (or in a Linux VM in a Jail)? Since the server will be online all the time and has a decent CPU, I figured: Why not run Handbrake on it to convert files?
JDownloader:
Most of the time I am using JDownloader under windows and linux to download files. Any of you using your NAS to download files and if so, what software are you using and is it easy to integrate a good VPN for the download software? Also, same question as for TV recordings: Does it lead to serious fragmentation?
That's all I can think of for now. If anything else pops up in my mind tonight, I will add it.
If you have read it all you earned my deepest respect. Thank you all for your time.
My new RAM (the old one was faulty) is in the testing-stage (memtest86+ with at least 5 passes (ETA 44hrs)) and I am currently working out how I will store, backup and access ~45TB of mostly media files in a smart manner (given my limited resources) in FreeNAS. My family consists of hoarders and tech noobs (bad combination when applied at the same time; that's why I am the one to "make it work, make it easy, make it cheap, make it safe-and don't you dare bother me, I don't understand any of it" - thank you very much, my pleasure)...
I am trying to collect my thoughts on several topics (this is going to be a lot of text since this has been floating around in my head for months) and share them with the community to see if my thinking makes sense and also to see where it all leads. I used the forum search for most of the topics in advance to get an overall image. If I missed something, feel free to link it and I will gladly read it.
Backup:
Is-state:
Right now all files are stored on windows machines (for example: my tower holds 4x 3TB and 1x 8TB and I have 2x 3TB, 1x 6TB and 1x 8TB as external USB 3.0 drives attached, which are only powered on when I perform a backup (and yes, I know how stupid and vulnerable that setup is... hence, FreeNAS). The miracle being that in the 12 years we have been using this concept, we never had a major loss of data.
Since I cannot afford to buy a second, equally sized NAS to simply backup using rsync or zfs-send, I will have to rely on the existing drives for backup purposes (additional to the "safety" of raidZ2, snapshots and UPS).
Could-be-state:
If my understanding is not flawed I should be able to create a dataset on the NAS matching each of my existing drives (5 datasets, 5 backup drives for example).
I could create a CIFS share for each dataset (accessed by a backup user with complete recursive read rights) and then simply mirror the content of these shares to the local drive attached to my windows machine over the network using a software solution for windows.
Possible software:
I read a lot about Macrium Reflect to backup entire disks as one file (not my primary concern; setting up windows is easy and quick) and 2BrightSparks SyncBack for general backup purposes (sounds very promising though pricy). Experiences?
At the moment I am using Beyond Compare to mirror my internal drives to their external counterparts.
Other software suggestions are very welcome.
Upside:
- fairly straight-forward
- datasets including content of the same kind can be united once an external drive dies and gets replaced by a bigger one
- very important data like RAW-pictures can be mirrored to 2 external drives for extra safety (one stored off-site)
- NTFS can be read by most OS
- easily expandable on the backup side
- the backup drives can be stored separately and do not rely on one another
Downside:
- creating the backups is manual labor
- unnecessary datasets due to the limited space of external drives. E.g.: 3 datasets: "MKV1", "MKV2", "MKV3" (to match the drives) where "MKV" would otherwise suffice.
- only one backup at a time (Gigabit LAN)
- no snapshots of the backup drives
- no redundancy of the backup drives
- no detection of bit-rot or other loss of file-integrity
- perils of USB (could be avoided by buying an eSATA cage)
Media streaming:
Is-state:
General
My PC shares the frequently used files (music/movies/tv-shows) via windows shares (read-only). That means that the files are only accessible when I am at home and my PC is on (that was one of the major reasons for a NAS/Server "oh no, he's out, I can't listen to music").
Audio
Music is accessed with Mediamonkey locally on all computers (the music shares are mapped in windows and scanned by Mediamonkey when desired).
Video
Video files are either accessed directly via Windows Explorer and VLC Mediaplayer or via KODI or Plex (both can read metadata stored in the individual movie directory which I manually add with tinyMediaManager).
Could-be-state:
Audio
Local:
Same setup that I am using right now. CIFS music shares on the NAS, mapped on windows machines and scanned and accessed by Mediamonkey.
Internet:
I am looking for a program that will enable me to access the music over the internet (a friend of mine uses Madsonic but I dislike the way it displays music and also it could not handle my amount of tracks (6-digit numbers)).
Does anyone know a way to stream music over the internet and encode it on-the-fly on FreeNAS (mp3 for mobile access)? The solution must be able to handle huge amounts of tracks.
Video
Local:
Same setup that I am using right now. CIFS video shares on the NAS, mapped on windows and scanned and accessed by KODI/Plex/Windows Explorer+VLC Mediaplayer.
OR
Plex/Emby/KODI Server in a Jail or as a Plugin in FreeNAS (I do not know what's better: Jail or Plugin) with Plex/KODI Clients.
Internet:
I read that Plex is a common choice to stream movies over the internet with on-the-fly encoding. Are there any alternatives to Plex (I read the GTC of Plex and disliked the thought that they have information on what files I have, what I watch, when I watch it and even where I watch it.)
Since I am new to streaming over the internet (until september 2015 I had 60kbyte/s UL so I never bothered to look into the subject; now I have 1,7Mbyte/s) I am open to suggestions, places to inform myself, experiences and so on. For testing purposes I installed both Plex and KODI on my tower and accessed it via local client PCs. I liked KODI a bit better for its custumization possibilities.
Temporary Files:
TV recordings:
I use a windows machine with DVB-Viewer (great piece of software btw.) to record movies and TV-shows over satellite. I wondered if it's an issue to store the recordings on the NAS instead of a local HDD in the recording PC (I would dump the 2 old HDDs and instead install a 120GB SSD for the OS to save power). I read that FreeNAS, as a COW System, has issues with fragmentation:
Would using the NAS as a temporary storage for recordings (between 6 and 25GB per file) lead to serious fragmentation or doesn't it matter at all?
Handbrake:
Has anyone got Handbrake running in FreeNAS (or in a Linux VM in a Jail)? Since the server will be online all the time and has a decent CPU, I figured: Why not run Handbrake on it to convert files?
JDownloader:
Most of the time I am using JDownloader under windows and linux to download files. Any of you using your NAS to download files and if so, what software are you using and is it easy to integrate a good VPN for the download software? Also, same question as for TV recordings: Does it lead to serious fragmentation?
That's all I can think of for now. If anything else pops up in my mind tonight, I will add it.
If you have read it all you earned my deepest respect. Thank you all for your time.
