From QNAP to DYI TrueNAS?

[BlackEternity]

Hi there,

I bet this is the gazillionth post of this sort.
I feel somewhat prepared for this.

First and foremost: I'm a Home-User and Enthusiast.
I know that this is no "out of the box solution" and requires some work and thoughts before doing stuff.
This is why I'm here and asking maybe stupid questions questions.

Preface:
I currently own a QNAP TS-879 Pro with 8x 2TB Drives in it.
- The NAS is Stock and utilizes an i3 and 2GB of RAM (Non-ECC as far as I know).
- It runs RAID5 and is nearly 10 years old now (Dang how time flies...)
- Connected via Gigabit-Ethernet to my network.
I had some hickups with it and to be honest: I'm done with it.
The Flash-DOM started acting up two years ago. It wouldn't boot properly, needed to be restarted 5 times to even POST. Sometimes I needed to re-seat and juggle the flippin' thing.
This is the main pain-point. When I really hammer it with thousands of small files I can make it hard crash and restart.
This is a no-no for me.

This system hosts all kinds of stuff you would expect at home.
Backups of movies, music, Documents and most importantly: My Raw Files and Photos that I take as a Hobby-photographer.

I have a redundant offsite-backup on a regular old HDD that I copy on a regular basis.
I understand that a NAS is not a Backup so I understand the difference.

Use-Case:
This storage-solution is used for just archiving my data, streaming and presenting it to different computers around the space.
There is no transcoding taking place. The NAS just has to present everything, accept all new stuff and work.
Main Stress-Point:
As a Hobby-Photographer I use Adobe Lightroom.
The Library itself is stored locally on my PC and a Backup is done to the NAS once a month.
But the RAW-Files are stored on the NAS and I edit them directly there.
Again - this is not a heavy workload IMHO like Video editing or something like that. The Files just have to be there in a "snappy" fashion.

I don't expect huge File-Dumps like 2 TB in one go - Just the ocassional Copy job or SD-Card from Lightroom.
Because the Disk space is relatively cheap nowadays and we are over the "2TB Disk size" that I was stuck back then, I want to reduce the number of spinning disks from 8 to maximum 4. That allows me to use RAIDZ1 with 3 in total (I can't expand later, I know) or RAIDZ2 with 4 in total for twice the parity.

Right now my Disk-Space is running pretty low and I need to upgrade my Storage-capacity anyways and I don't want to dump money into 8 drives when I ditch my QNAP anyways.
Right now I have around 12TB usable space from the 8x2 disks. I aim towards 20TB+ - depending on what my ZFS-Config might be and how expensive everything gets. 20TB minimum, more would be nice as piece of mind - Again 4 disks would be the target I aim for in terms of weight, noise and heat (SSDs for OS or whatnot excepted).

Now to you people:
My first thought was to use my old i5 and 16GB of RAM that I had kicking around. But reading on the "Get started" page, that was shattered pretty quickly.
I got in a heated debate with a good friend of mine that uses UnRaid and I am not really willing to go that route - I think.
Regardless of reading "you should use ECC and beefy stuff" I gave it a whirl and it worked fine. So I know that I don't need to throw 128GB of RAM at that thing to just do it's job.

Buuuut there is still so many things I don't get. And I hope to find answers because I want to make an educated decision here before I pull the trigger and move my precious data to any place.

1) Why the "high RAM requirements" and how much is enough for my Use-Case above?
I'm new to ZFS and I did some reading. I understand that it's taxing on the hardware to properly get to speed. But what does the RAM do with it?
I understand that it's the buffer (ARC?) for your data. And if it's full it needs the next best thing and that would be the Disks itself / the System-Disk?
So how do I know when "enough is enough"? I tested with my 16G of Non-ECC Memory and it got filled up. But it copied happily from and to the system with near full saturation of my Gigabit-Ethernet (100 Megabytes/sec).

2) Does ECC make a huge difference and what Metrics are important: Clock-Speed, Single or Dual-Rank, How many modules?
I know I know - I question ECC. I'm not saying I don't want ECC - I just want to learn and understand.
I know that ECC is error correcting and that it prevents corrupt data. Which I really like. But why for example is my PC not hard crashing or my QNAP working fine?
I read that the TrueNAS / ZFS-Filesystem stores data in the RAM before it gets written to Disk (Which basically all systems do at one point or another). But how long is that data stored in there "for real?" that makes it "vulnerable" to this problem?
I've read Cyberjock's FreeNAS / TrueNAS Guide and to be honest: It's a bit frightening and I don't want to be a statistic.

3) What type of CPU should I target?
I fully understand, that this is a server and I shouldn't put janky hardware in it when I appreciate the safety of my data.
Newer AMD boards for example can utilize ECC-Memory. But so do older Xeon processors.
Again - in regards to my preface, I want to understand what makes sense. I don't want a 100W Processor heating my home in the winter - I got central heating for that. And from my quick and dirty testing, the i5 that I used seemed pretty chill with 20% max load for a couple of seconds in the dashboard.
Does it make sense to search for older used Server-hardware or just grab a cheap modern CPU with a Board that is more Server/SOHO oriented?

4) RAIDZ1 or RAIDZ2?
With the Preface that I aim around 3-4 disks, I am not sure what to pick. RAIDZ1 reflects my current RAID5 setup and I know that I can take one hit before it crashes and burns. Which was fine in the past - I never lost or replaced more than one drive (knock on wood).
The benefit of RAIDZ2 is appealing to me but with "only" 4 disks I feel like I would just setup a RAID1 in terms of utilization. "Wasting" half the space for Parity seems a bit overkill and is from my standpoint really beneficial when using a setup like my old QNAP or more? Again - constructive feedback appreciated because I've read Cyberjock's guide and it made me question my life-choices :D

5) Please forgive me for asking and don't throw a pitchfork but I want to understand this: What are fundamental benefits of TrueNAS over Unraid?
As I've said in the Preface, I had an argument with a friend of mine over this. He uses Unraid but he couldn't explain the benefits of me shelling out money for it.
I don't rummage through my closet and find a dusty 1TB disk and think "Well great, let's plonk it in my NAS". Since I built my QNAP, I never added a single hard disk to it. I bought it, I bought 8 Disks and shoved them in. Replaced them when they failed, but never needed to extend the Disk-Array.
I don't use PLEX so this is another "advantage" of Unraid gone. "Runs on low end hardware" is one - but again: No idea what I want. Still reading up on it and wanting to understand the solution I might choose. On the surface Unraid and TrueNAS seem "identical". Both offer Storage solution. One uses ZFS, the other XFS. One uses a single parity-disk for everything in order to "save the disk and spin unused ones down", the other takes the more classic "Raidlike approach". I don't need to spin single disks down. For the past 10 years, 8 disks have been running nearly 24/7.
I read up on the documentation and on the technical side but I want to understand from users themselves: What makes this solution worthwile and great?

6) Hardware-Issues - How to deal with it?
I'm really frustrated by the shortsighted approach of QNAP. I bought an 8-Bay NAS that is NOT a Home-Use product specifically so I could upgrade it and fix an issue without throwing everything in the garbage. My Flash-DOM is finnicky and QNAP tells me to punch dirt and buy a new QNAP instead of selling me a replacement-stick. The unit even has a second slot in it for a backup-unit and is replaceable.
So how do I deal with Hardware-Faults? How does TrueNAS and the ZFS respect changes to Hardware?
Can I replace my CPU without killing everything?
Can I replace faulty RAM without destroying everything?
What about a new system when my Mainboard is bricked or I need more power? Can I transfer my installation and the disks to another system without rebuilding it? Or do I need to have the same storage-space AGAIN in order to transfer / copy stuff? Yes - Backups will be done but I ask specifically about the things I can do without rebuilding everything. Where does it end? I would expect that I can drop replacement parts in it without the ZFS completely acting up - as long as the config of the TrueNAS OS is still fine?
I want to make sure that my next solution can survive another 10 years from the moment I have transferred my data and I don't need to babysit the machine.


If you aren't dead by now and can still read this: Thanks and I hope my questions and thoughts aren't boiling your blood. I'm an enthusiast and love tech and I want to understand it before I commit to things. I want to learn and make educated decisions and if everything here is obvious to me: Please understand that I have been searching online for the past 3 days and I always end up short on these questions / or am too stupid to find everything I need.

Thanks and cheers,

 

[jgreco]

Why the "high RAM requirements" and how much is enough for my Use-Case above?

For 20TB of storage? 16GB may be fine.

So how do I know when "enough is enough"? I

It's largely subjective. "When it's fast enough" is not a bad summary. There's a solid component to it, and the topic is frequently discussed; most recently in this thread:

What metrics can I use to determine if I need more ram?

I am running Freenas on a C2550D4i and while this cpu is pretty weak it seems to me that performance of the web interface should be much better than it is. I am running an 8*8TB Raid Z2 setup with 8GB of ram and a plex docker instance. I am getting tons of conflicting information on whether or...

www.truenas.com

There's some danger in going wildly crazy too small. 8GB of RAM for 20TB might be pushing it, but 16GB isn't. It might not be PERFORMANT in the desired manner, but it is not unsafe.

But so do older Xeon processors.

Actually virtually ALL Xeon processors.

I've read Cyberjock's FreeNAS / TrueNAS Guide and to be honest: It's a bit frightening and I don't want to be a statistic.

That's good; you've virtually guaranteed you won't be a statistic, by virtue of learning about the pain points.

But why for example is my PC not hard crashing or my QNAP working fine?

You know how you get in your car, and you (hopefully) buckle up your seat belt, and the air bag indicator lights up, but you go driving around and you never actually seem to find yourself in an accident?

ECC's the seatbelt. You can drive around all day without it and not get into trouble. This doesn't make not wearing the seatbelt a good idea. It isn't actively saving you on a minute-to-minute basis. But data corruption is a real thing. It happens. ZFS tries to minimize this, and ECC works to address a common weakness in servers/PC's. ECC will very rarely save you, but when it does, oh boy, then you're happy. And you might not even know when it saves you.

i5 that I used seemed pretty chill with 20% max load for a couple of seconds in the dashboard.

So find the Xeon equivalent, because it will probably work about the same.

Parity seems a bit overkill and is from my standpoint really beneficial when using a setup like my old QNAP or more? Again - constructive feedback appreciated because I've read Cyberjock's guide and it made me question my life-choices :D

Awesome. The main problem with ZFS is that there are no fsck/chkdsk type tools. Once a block is corrupted or lost, that's that. Therefore it is advisable to build sufficient redundancy into your pool. I talked about this in a different topic here:

DDR5 “ECC” and ZFS

I am planning on a homelab upgrade once DDR5 prices settle into something more normal (scalping really does blow…), but was curious on what the merits of DDR5’s internal checksumming was vs what I believe is called the current standard of full “chip to RAM and back” checksumming verification...

www.truenas.com

RAIDZ2 gives you the potential to lose a full disk, and then to encounter additional disk errors during rebuild, and your data still isn't at risk.

What are fundamental benefits of TrueNAS over Unraid?

It's subjective. ZFS is oriented towards large servers in a commercial/enterprise setting, and has several things working against it for small home hobbyists, where UnRAID has a certain popularity for being more readily able to take a pile of random crap and make a pile of slightly-less-crap out of it. But if you're interested in actually building high quality storage and don't mind spending some money, ZFS has the potential to be extremely performant, very stable, high levels of protection for your data, etc. I like ZFS because I don't have to worry about things I used to have to worry about. TrueNAS is just a shell that makes ZFS easier.

I bought an 8-Bay NAS that is NOT a Home-Use product

Ah, yeah. I hope you can take it in good humor when I say that this was ... naive. Understandable, but you misunderstood the business model of QNAP/Syno/etc. They are in the business of selling hardware. It is how they make their money.

So how do I deal with Hardware-Faults? How does TrueNAS and the ZFS respect changes to Hardware?
Can I replace my CPU without killing everything?
Can I replace faulty RAM without destroying everything?
What about a new system when my Mainboard is bricked or I need more power? Can I transfer my installation and the disks to another system without rebuilding it? Or do I need to have the same storage-space AGAIN in order to transfer / copy stuff? Yes - Backups will be done but I ask specifically about the things I can do without rebuilding everything. Where does it end? I would expect that I can drop replacement parts in it without the ZFS completely acting up - as long as the config of the TrueNAS OS is still fine?

If you're using appropriate hardware, i.e. all the recommended stuff, it's usually problem-free. This isn't like Windows 95. It's an actual professional operating system.

If you use a RAID controller (don't do that), you can run into weird problems. But you were going to run into other weird problems, and then the bad problems, and the horrible problems too, because RAID controllers often obscure the disks from the OS. So let's assume you don't do this.

Switching your ethernet adapter type may require some re-do in the networking department, which can also sometimes affect any jails or VM's that you have set up.

However, the underlying storage should always move cleanly. The boot device can be an SSD or high endurance thumb drive, or a pair of them, and are easily replaced. You do want to make sure you download your NAS configuration before moving between platforms. The configuration should drag all the important bits along, with the exception of the networking stuff if you change ethernet adapter types.

I want to make sure that my next solution can survive another 10 years from the moment I have transferred my data and I don't need to babysit the machine.

Well, the oldest FreeNAS box here has passed its 10 year birthday. Its oldest snapshot is from 2011 (the year). It has had its disks replaced/upgraded once, it has been hosted on several different hardware platforms, it went from FreeNAS-with-nanoBSD to FreeNAS-with-ZFS, through versions since FreeNAS 8, and has basically been in continuous operation for a long time.

I don't generally need to think about it unless something goes wrong. Storage should just sit there and do its job. FreeNAS has been pretty good in that regard.

I'm an enthusiast and love tech and I want to understand it before I commit to things. I want to learn and make educated decisions and if everything here is obvious to me: Please understand that I have been searching online for the past 3 days and I always end up short on these questions / or am too stupid to find everything I need.

Do feel free to spend a bunch of time over in the Resources section. I do this stuff professionally, and I've spent a bunch of time trying to write articles to bring people up to speed on unfamiliar topics, and lots of other knowledgeable people have too. Large, high quality NAS involves lots of rabbit holes, and can be intimidating. A stupid question is the one not asked that ends up killing you. Not knowing things is no crime, and as long as you don't mind being redirected to the knowledge, you should be able to get up to speed.

As a side note, I find that omnibus messages such as yours get a poor response on these forums, perhaps because they are overwhelming to the average participant. I've taken a good bit of time to provide some answers, potentially a bit trite, in hopes that we can get you moving in the right direction.

 

[NugentS]

1. RAM. TN needs 8GB minimum and will scale as high as you like. The important thing to relaise is that RAM is your read cache. TN, when you read a file, loads it into ARC (RAM) and then serves it from there (until or unless it gets displaced). So make sure that you have enough RAM to cover your working set + some for TN to run in. Start with something like 16GB, but make sure you can expand as required to say 64GB

2. Long argument. ECC is not required. TN will run just fine without it. Having ECC does protect against a random bit flip in RAM then being written back to the disks and possibly corrupting things as the RAM is trusted

3. You don't need that much CPU for file serving / standard NAS functioins. You have eliminated transcoding as a requirement so a basic CPU will be fine. Fewer cores at a higher GHz is sensible as SMB is a single threaded app so you want a decent (not silly) single core performance.

4. RAIDZn - another long discussion. Z1 is considered unsafe for large disks (above 2TB or was it 4TB) as the resilver takes too long during which you could lose another disk, and there is no parity left during the resilver so go RAIDZ2 or mirrored. Mirrored has advantages and disadvantages:
Cheaper to expand (2 disks at a time)
Much shorter resilver times
Faster IOPS
Not as "safe" as Z2 as you can only lose one disk in a mirror (unless you use a 3+ way mirror, but thats getting silly)
For four disks, that are new - I would use mirrors personally.

5. In one word (set of letters) ZFS

6. IN TN everything is based around ZFS. The disks can be picked up and moved to another machine also running ZFS (as long as its the same version or newer) and imported and you get to keep all your data in the event of a total motherboard / system failure. If your boot drive fails, then replace it, reinstall and upload your config and everything is back again (you will keep a copy of the config file safe won't you). Even if you don't have the config file and can't retrieve it from thew syste dataset then just import the pool and just setup shares again. All the datasets will be there - just permissions and such will be wrong.

 

[BlackEternity]

First of all:
Thank you all for giving me valuable feedback and being welcoming.
Sorry if I rip some info together but I want to understand it.
On my QNAP and my PC and whatnot the RAM is always just a temporary dump for stuff that is needed now and "in a few moments".
But from the info I gathered here:

DDR5 “ECC” and ZFS

I am planning on a homelab upgrade once DDR5 prices settle into something more normal (scalping really does blow…), but was curious on what the merits of DDR5’s internal checksumming was vs what I believe is called the current standard of full “chip to RAM and back” checksumming verification...

www.truenas.com

it seems that this is not the case with ZFS. Again: Sorry if this is obvious but this really changes my thinking of ECC memory.
If stuff is stored "long term" in the RAM for frequent access and is kept alive there in some shape or form, I understand the impact that this can have. And of course - with an uptime of >4 years on my QNAP I really don't want anything to happen in this case on ZFS.

For 20TB of storage? 16GB may be fine.

Regarding the really "grand scheme of things" that this will cost some money and is not easily solved by chucking a disk at it and be done, 100 bucks more or less in the total bill won't spoil that. So I'm looking for 32GB just to be "safe". I would opt for two Sticks in case I want to upgrade when I feel comfortable with the system and might deploy a VM for fun and giggles to tinker with something.

Actually virtually ALL Xeon processors.

Well yes - of course. I was all over the place in my mind. I meant to say that if I would opt for an older, second hand Xeon Processor instead of something new from AMD or so.

RAIDZ2 gives you the potential to lose a full disk, and then to encounter additional disk errors during rebuild, and your data still isn't at risk.

Yikes. Thanks for glaringly pointing me to that issue. I never thought about that when I rebuild my QNAP when a Disk failed. Of course it doesn't run ZFS but I never bat an eye about the "what if". I know this is a rabbit hole without ends but yes, this is something I could get a sleepless night over.

It's subjective. ZFS is oriented towards large servers in a commercial/enterprise setting, and has several things working against it for small home hobbyists, where UnRAID has a certain popularity for being more readily able to take a pile of random crap and make a pile of slightly-less-crap out of it.

Nicely said, I like that. Slightly-less-crap.

Ah, yeah. I hope you can take it in good humor when I say that this was ... naive. Understandable, but you misunderstood the business model of QNAP/Syno/etc. They are in the business of selling hardware. It is how they make their money.

Yeah I can take that slap. Took me some years but as I've said - after I got the info from QNAP that I should add another piece of electronics to the landfill with their name on it, I realized that. That's why I'm here now.

As a side note, I find that omnibus messages such as yours get a poor response on these forums, perhaps because they are overwhelming to the average participant. I've taken a good bit of time to provide some answers, potentially a bit trite, in hopes that we can get you moving in the right direction.

I really appreciate the time and effort you put in here. And thanks for pushing me to different stuff.
I am currently falling from one rabbit-hole to the next and I know that I am still bloody new to this.

4. RAIDZn - another long discussion. Z1 is considered unsafe for large disks (above 2TB or was it 4TB) as the resilver takes too long during which you could lose another disk, and there is no parity left during the resilver so go RAIDZ2 or mirrored. Mirrored has advantages and disadvantages:

As I already said above to jgreco's statement: This really slapped me into reality. I "just" run 2TB drives but because I'm now in the 10-12TB territory for disk space, I would literally jump out the window when anything happened during resilvering.

It can only be installed on a usb stick, which are not as reliable as hard drives and ssd's. It is a hassle to get your license transfered to a new usb stick.

Yeah my friend thinks it's not an issue but I am really turned off by the thought of leaving a USB Stick on a system for 10 years and expect it to keep my data safe and intact...

Also it is not raid... hence the name UNraid. You get the read performance of a single disk.

This might be a bit obvious but ... I never thought about the name like this. *duh*


Thanks for the infos all. I really appreciate the help here.
I scanned through the list recommended HW-List and for now (without any reason to why other than it "reads good") I would probably use this:
- Xeon E3 1220v6 (Can't find any of the lower end i3's anymore so Xeon we go).
- Supermicro X11SSM-F
- 32GB RAM. And there is an issue.

Tested Memory List | Supermicro

www.supermicro.com

I looked through this site for the supported RAM and they only list their own RAM anymore. And I can't get that where I live.
Sooo ... is there an archive of supported RAM for this board? I couldn't find anything quickly.
I found this Samsung RAM: M391A2K43BB1-CTD that I could buy but without the QVL List from SuperMicro I have no clue if it's okay.


Cheers,

 

[Jessep]

Please keep in mind that TrueNAS (RAID including ZFS) is not a backup and start thinking about your backup strategy.

"3-2-1"
3 copies
2 media types
1 offsite

 

[BlackEternity]

Please keep in mind that TrueNAS (RAID including ZFS) is not a backup and start thinking about your backup strategy.

"3-2-1"
3 copies
2 media types
1 offsite

Thanks for pointing that out. I already have Backups of the most critical data of my storage offsite just in case. So that's okay.

 

[jgreco]

RAM is your read cache.

And your write cache, too. Transaction groups get built there, so if you have a really fast pool but a small amount of RAM, you lose write performance too.

 

[NugentS]

On memory the X11 takes

• 2400/2133/1866/1600MHz SDRAM 72-bit, 288-pin gold-plated DIMMs

My X10 takes

• 2400/2133/1866/1600MHz ECC DDR4 SDRAM 72-bit

I got my memory from Bargain Hardware but its gone up in price from when I bought some. Another alternative is Ebay, maybe you can find a system dismantler in Europe who has memory to sell.

 

[HarryMuscle]

I only skimmed through your post so I'm not sure if you've considered this ... how about installing TrueNAS Scale on your QNAP unit?

Thanks,
Harry

 

[jgreco]

how about installing TrueNAS Scale on your QNAP unit

At a minimum, you'd have to upgrade the memory. In theory, it's a lovely idea, and you do have the advantage of it being an Intel platform, so it's not impossible that it could be done, but it may not be a good idea as a primary NAS.

 

[NugentS]

It does work on suitable hardware - I do it - but wouldn't consider it as a main NAS. I use it as a snapshot repository overnight, so don't really care how fast / slow it is.