General ZFS Question

[Kortex]

Greetings all,

I do a lot of photography and thus have a lot of images. Some of my images are recent (~20MB per image) and some are older (~30KB per image) from back when digital cameras were not so good. I'm aware that ZFS doesn't need to be "defragged" like some windows filesystems, but will placing thousands and thousands of images on my ZFS freenas box increase the likelihood of corruption or anything? Should they be in their own dataset? Any advice here would be appreciated. I've been hesitating moving my images to it.

 

[MrToddsFriends]

As fas as I can tell, ZFS is the most extensively tested self healing file system available.
http://doc.freenas.org/11/zfsprimer.html

Having that said, one should be familiar with concepts and features both of ZFS and FreeNAS before relying on it. Make also sure to use suitable hardware. Read the manual from cover to cover and browse the Resources section of the forum.
https://forums.freenas.org/index.php?resources/

Of course, feel free to ask questions in the forum before making mistakes.

 

[Jailer]

I've been hesitating moving my images to it.

Once you educate yourself about FreeNAS and ZFS you won't want to store them anywhere else.

 

[garm]

Storing millions of blocks that cannot be allowed to degrade is the point of ZFS. Images is a good example where I wouldn’t trust anything else.

 

[Ericloewe]

Is there any filesystem out there that'll reach a certain threshold and go "hmm, time to corrupt some data!"? Other than btrfs. I expect this sort of thing from btrfs.

 

[Jailer]

Is there any filesystem out there that'll reach a certain threshold and go "hmm, time to corrupt some data!"? Other than btrfs. I expect this sort of thing from btrfs.

NTFS??? :D

 

[Arwen]

Is there any filesystem out there that'll reach a certain threshold and go "hmm, time to corrupt some data!"? Other than btrfs. I expect this sort of thing from btrfs.

Linux' BTRFS.

 

[Chris Moore]

I've been hesitating moving my images to it.

Does this mean that you have already built a system, or are you here looking for information regarding a system you are thinking of building? Details?

Also, please review these as they will help you in all your future endeavors with FreeNAS and ZFS:

Updated Forum Rules 4/11/17
https://forums.freenas.org/index.php?threads/updated-forum-rules-4-11-17.45124/

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

 

[PhilipS]

I store photography on my home system - I keep the RAW files in one dataset and the cooked JPEGs in another. The JPEGs get replicated offsite to another FreeNAS server. The RAW files are not backed up, just can't spare that kind of bandwidth.

I have Lightroom setup to keep the catalog and smart previews on a local SSD and the RAWs are stored on the FreeNAS. I then use Jeffrey Friedl's jfFolder publisher plugin to cook the JPEGs and transfer to the other FreeNAS dataset which will then get replicated offsite.

 

[Chris Moore]

I have Lightroom setup to keep the catalog and smart previews on a local SSD and the RAWs are stored on the FreeNAS. I then use Jeffrey Friedl's jfFolder publisher plugin to cook the JPEGs and transfer to the other FreeNAS dataset which will then get replicated offsite.

That sounds like a pretty good setup. I had to read about Jeffrey Friedl's jfFolder publisher plugin because I used to do some photography my self.

The RAW files are not backed up, just can't spare that kind of bandwidth.

Have you considered a second local FreeNAS server, or even a second storage pool, to backup the raw files to? I don't know how you handle the raw files. Once you go JPG, do you ever go back to the RAW?

 

[PhilipS]

Have you considered a second local FreeNAS server, or even a second storage pool, to backup the raw files to? I don't know how you handle the raw files. Once you go JPG, do you ever go back to the RAW?

I do occasionally go back to the RAWs - and sometimes we don't get to them immediately so losing them wouldn't be ideal, but not worth the expense of building and operating another server to us (I wouldn't pay $1000+ to recover them - so I'm not going to spend that to protect them). Most of the photos are hobby/personal, but my wife does have some professional work on there - I may consider setting up a separate catalog for those that store to a different dataset and back those up offsite, although she processes those pretty quickly for her clients so the JPEGs are enough at that point.

 

[ethereal]

i have home video and some photography.

i have two raidzf2.

i copy the original files to my first pool - i have a script that runs hourly (can also run it manually) and copies the files to the second pool.

i then attach an external hdd to my windows machine and a program syncs the files on the first pool to the external hdd.

this way i could loose 4 hdd in freenas - 2 in each pool - and not loose any data. the external hdd is in a fire proof safe.

i also burn to blurays - now i have 4 copies in 4 different places

 

[Ericloewe]

Linux' BTRFS.

Are you serious? I... am somehow not surprised and yet I am shocked.

 

[garm]

Are you serious? I... am somehow not surprised and yet I am shocked.

its a feature https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-RAID-56-Is-Bad

 

[Ericloewe]

its a feature https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-RAID-56-Is-Bad

That part is not new, it was broken from the start... Amazing how little thought went into it.

 

[Chris Moore]

Greetings all,

I do a lot of photography and thus have a lot of images. Some of my images are recent (~20MB per image) and some are older (~30KB per image) from back when digital cameras were not so good. I'm aware that ZFS doesn't need to be "defragged" like some windows filesystems, but will placing thousands and thousands of images on my ZFS freenas box increase the likelihood of corruption or anything? Should they be in their own dataset? Any advice here would be appreciated. I've been hesitating moving my images to it.

Did you get the answer you were looking for? Do you still have questions?

 

[Arwen]

That part is not new, it was broken from the start... Amazing how little thought went into it.

It's quite frustrating. ZFS on Linux can't be a bundled with the kernel due to licensing. Yet BTRFS is bundled with the Linux kernel, and is for some production purposes, useless.

It's even worse than some Linux or BTRFS fan boys think. ZFS is available on several *nix flavors. Even a MS-Windows port has started, (though I doubt it will get anywhere quickly, if ever). But, BTRFS is mostly Linux only. (There is an exception with some MS-Windows clone OS that does support BTRFS. But that's not really production ready software, yet.)

From a comparison point of view, (MY opinions!):

Stability

• ZFS >98%
• BTRFS <90%, (if NOT using RAID-5/6! Or have any un-expected power fails!)

Features

• ZFS 90%, (at least until we get some of the newer features installed)
  
• BTRFS 65%, (missing stable RAID-5/6 for more than 5 years!)

Code release & managability

• ZFS 90%, (having code developed on 3 different platforms does create problems...)
  
• BTRFS 20%, (you have to run cutting edge kernels to be SURE of getting all the features and bug fixes! But those are the kernels that may have the most bugs!)

Here is what someone else wrote about BTRFS:
https://kethryverisblog.wordpress.c...s-a-sad-day-for-file-system-that-had-promise/