Home NAS solution - How to contain costs

[ChrisRJ]

Thanks both, so here it's my reasoning on TrueNAS.

- CPU 8/16 threads max (2018 XEON probably)
- 128GB ECC DDR4

Depending on the use-case this may very well be overkill.

Use case
I'll use my NAS for projects (software development), media etc, but also VM backups, basically data that I don't want to lose.
A lot of writing every day, let's say 100-150GB a day but big blocks, not a big drama for an HDD...

Can you be more specific? Are you working with your IDE against an SMB share, or is the NAS more of a backup for source code not yet checked in?

If it's better, I can just take snapshots on daily basis, but backups once a week for example... Something tells me that this is better... I just need to find a proper strategy.

Do you mean daily ZFS snapshot and a weekly backup with a different technology (e.g. rsync)?

Where's my real backup?
On a secondary disk, powered by the NAS anyway which is powered by an UPS, or maybe from a Laptop connected to the UPS (laptop batteries are hateful), we'll see.

In case of a power surge, a defect of the PSU etc. that USB will likely be gone as well. So the USB disk should only be connected during the backup.

Plus my cloud account, but I think I should use another specific service for TrueNAS just to make it easy, I still didn't evaluate this point...

I can recommend the cloud sync task with encryption. Personally I use MS OneDrive from Office 365 Family, because the latter is often on sale. So I end up with 5-6 Euros per month for 6 TB of cloud storage.

Plus, I don't want to install garbage drivers and this is not even allowed in TrueNAS due to your image, right?
So, shall I rely on the IPMI or similar?

That (preferably) or whatever built-in GPU comes with your CPU (if that is the case at all).

How to spot HDD issues
I have a fantastic case, but it's not meant for hot swap of disks, so I wonder how I can be effectively notified for broken disks or pool issues...

You can enable notifications for scrub and smart tests. Plus there is a fantastic script maintained by @joeschmuck (sorry, I don't have the URL right now and need to run).

 

[artlessknave]

Are you aware of electricity prices in Europe?

nope. I am in Canada.
I blame crypto-currency mania

 

[Torrone]

Are you aware of electricity prices in Europe? 50 cents per kWh is not a bad price these days here in Germany. And from what I have read it is not very different in the UK.

It's very expensive! In France we pay 18,41 cent /KWh full price and 14,70 cent /KWh during the night (but the government regulates prices)

 

[danb35]

I blame crypto-currency mania

It's more a result of EU policies demanding "renewable" energy that isn't (the second law of thermodynamics guarantees there's no such thing), combined with a little fracas in an oil-producing country to the east of them.

 

[TDi39]

1. Depending on the use-case this may very well be overkill.

2. Can you be more specific? Are you working with your IDE against an SMB share, or is the NAS more of a backup for source code not yet checked in?

3. Do you mean daily ZFS snapshot and a weekly backup with a different technology (e.g. rsync)?

4. In case of a power surge, a defect of the PSU etc. that USB will likely be gone as well. So the USB disk should only be connected during the backup.

5. I can recommend the cloud sync task with encryption. Personally I use MS OneDrive from Office 365 Family, because the latter is often on sale. So I end up with 5-6 Euros per month for 6 TB of cloud storage.

6. That (preferably) or whatever built-in GPU comes with your CPU (if that is the case at all).

7. You can enable notifications for scrub and smart tests. Plus there is a fantastic script maintained by @joeschmuck (sorry, I don't have the URL right now and need to run).

1. Yes, I strongly think it will be overkill most of the time, but if I need to compile a complex project even just once a year, I want it to be quick.
CPU is certainly overkill, but is RAM what mostly concerns me, the cost of 128GB of these builds is almost the same as 64GB.
Regarding the CPU, I need to use it for the next 5y minimum (if it doesn't break), so I think to the future too.

2. My projects will reside in the 2TB M2, the pool on the NAS is only for backups, or at least I'll have different mount points for different machines.
As said previously, this is not the only backup I'll have.

3. Daily snapshots are for VMs (the entire block), Time Machine on my Macs and some other machine.
Backups are heavier, so I'm planning to take them once a week for the VMs, this process doesn't apply to Time Machine for macOS.
The automation for backups still need to be planned, but rsync could be the starting point.

4. Yes, the USB disk will be used only for backups, powered on when necessary.
My electrical line is protected, not only at home, but also on the way, so I think I should be safe,it's more than a decade that I don't see any issues with power surges, unless if it comes from the PSU :D .

5. Microsoft offers the family subscription with 1TB for each person, not on one account...
I could use any other service anyway, the issue to automate all this is the file size of the VMs storage blocks... Not for the script I need to write, but for all the crazy writing on the M2, I could use the NAS to avoid the burnout, but it's additional load on the NAS :D , or maybe not, it depends where it stages the files... I need to check.

6. XEONs don't have a built-in GPU if I'm not wrong... I think the only way is to have an IPMI or I need to play with the drivers...

7. Is this script part of the official release?
I could check it out by myself, but I'd expect that the software has already something like that.

Thanks a lot for your help.

 

[TDi39]

It's very expensive! In France we pay 18,41 cent /KWh full price and 14,70 cent /KWh during the night (but the government regulates prices)

LOL, we in UK pay around 34, coming from 20...

It's more a result of EU policies demanding "renewable" energy that isn't (the second law of thermodynamics guarantees there's no such thing), combined with a little fracas in an oil-producing country to the east of them.

Let's simplify, Europe doesn't have enough energy and that's it, that's why they always purchased foreign energy and underground resources from other continents in order to keep up with them...

 

[DigitalMinimalist]

I can recommend the cloud sync task with encryption. Personally I use MS OneDrive from Office 365 Family, because the latter is often on sale. So I end up with 5-6 Euros per month for 6 TB of cloud storage.

One question: do you Pool the different 1TB spaces?
As OneDrive is heavily integrated into Windows (and the respective MS account), it‘s not easy to use the different cloud spaces from a family account

 

[ChrisRJ]

One question: do you Pool the different 1TB spaces?
As OneDrive is heavily integrated into Windows (and the respective MS account), it‘s not easy to use the different cloud spaces from a family account

No, I use a manual "sharding" approach. In other words, I use multiple accounts in such a way that for none the 1 TB threshold will be reached. For me this caused surprisingly little problems so far. In fact, there was none, only the one-time additional effort to set up multiple accounts in TrueNAS. But those 15 minutes I can live with, esp. relative to the price of roughly 1 Euro/TB/month.

EDIT: I am fully aware that this approach will not be suitable for all possible requirements out there.

 

[DigitalMinimalist]

No, I use a manual "sharding" approach. In other words, I use multiple accounts in such a way that for none the 1 TB threshold will be reached. For me this caused surprisingly little problems so far. In fact, there was none, only the one-time additional effort to set up multiple accounts in TrueNAS. But those 15 minutes I can live with, esp. relative to the price of roughly 1 Euro/TB/month.

EDIT: I am fully aware that this approach will not be suitable for all possible requirements out there.

Hi,

thank you so much for that hint :) I didn't consider that option - I have a family account and I use 3 accounts right now and was looking into merging the space, e.g. CBackup, for offsite backup. Your route seems feasible to me.

PCs?Laptops, etc -> onsite Backup to TrueNAS -> offsite Backup of important files to OneDrive

 

[TDi39]

Assuming I need to do a deployment low power, no development usage, what could be the best hardware? (cost-wise too)

I was digging a bit about the hardware, cool solutions are:
1. https://www.supermicro.com/en/products/motherboard/X10SDV-4C-TLN2F
2. https://www.supermicro.com/en/products/motherboard/x11ssh-f

The first model is more commonly found with a CPU already, not bad price-wise.
They both have IPMI, so no videocard required.
I could cut down the average consumption to 65W (from 90W of a proper machine for development), I don't think I can go lower with 4 disks.

Is there any reason why you wouldn't use one of these 2 motherboards?

I want to use TrueNAS Scale due to KVM, but if I won't use VMs in future, I could use TrueNAS core, I just need to be careful with the HW choice from now.
Btw, can I migrate my pool from Scale to Core? I don't think it's OS related, right?

----------------------------

At the end, I still think I'll go with a single big machine, but at the moment, it doesn't look a great investment, because I already have an old DEV machine, so I'm exploring cheap alternatives.
One thing is certain, I need a NAS and I want it now :D .

Have a nice end of the year everyone!

 

[ChrisRJ]

Assuming I need to do a deployment low power, no development usage, what could be the best hardware? (cost-wise too)

What do you mean with "no development usage"?

I was digging a bit about the hardware, cool solutions are:
1. https://www.supermicro.com/en/products/motherboard/X10SDV-4C-TLN2F

Mini-ITX is very limiting. On the plus side there is RDIMM support.

2. https://www.supermicro.com/en/products/motherboard/x11ssh-f

UDIMM-only would be a show stopper for me in terms of price. RDIMMs are usually considerably cheaper. Example: While the price has gone up quite a bit, I paid 20 Euros for 32 GB DDR3 RDIMMs in September 2020.

I could cut down the average consumption to 65W (from 90W of a proper machine for development), I don't think I can go lower with 4 disks.

I have about 90 Watts with 8 drives, so your looks quite ok.

Is there any reason why you wouldn't use one of these 2 motherboards?

No. But I think this is the wrong question. First I would ask myself: What is important to me? Do I want to be flexible in terms of the CPU? How many SATA ports do I want on the motherboard because an additional HBA will consume at least another 10 Watts? In my case this made me end up with an X9 SRi-F.

I want to use TrueNAS Scale due to KVM, but if I won't use VMs in future, I could use TrueNAS core, I just need to be careful with the HW choice from now.
Btw, can I migrate my pool from Scale to Core? I don't think it's OS related, right?

I would create the pool with Core, then install Scale. This way you can be sure that this change is possible in terms of ZFS version/feature.

At the end, I still think I'll go with a single big machine, but at the moment, it doesn't look a great investment, because I already have an old DEV machine, so I'm exploring cheap alternatives.
One thing is certain, I need a NAS and I want it now :D .

I had also thought about this long and hard. I ended up with the following setup:

• TrueNAS Core on bare metal (details in signature) running 24x7
• Low-power ESXi (Celeron G3900 with 2x2.8 GHz) for VMs always running but not needing a lot of CPU power (e.g. Unifi Controller, pfSense, Redmine, Nextcloud) with old 480 GB local SSD, running 24x7
• 2 Supermicro 1U servers (also X9 SRi-F with 8 cores and 128 GB each) running XCP-NG for power-hungry VMs that are not running 24x7 (local 1 TB SSD with backups to TrueNAS Core)

Have a nice end of the year everyone!

Thank, you too!

 

[TDi39]

@ChrisRJ thanks a lot, but you can't stop from making me wondering about your setup :D .
I have a couple of questions:

1. Why with 8 disks you use RAIDZ2? Just disk space needs? I would expect to have 3 disks as fault tolerant with 8 disks setup.
2. I see that you have the boot drive mirrored, but how? Is it through TrueNAS or you did some custom/3rd_party implementation?
3. Why do you have 64GB of RAM on the TrueNAS used as a NAS? I thought that 16GB are enough...

Let's jump on my previous arrangments:
3. I see that I may change my mind in future, taking a better equipped MB will help, so no mini-ITX, damn good point :D
4. In terms of development machine, I use VMs to develop (they are nicely portable), but I'm not 100% sure if I'll use the TrueNAS Scale box for that (with KVM), definitely not until my old machine breaks. So I thought to start with recent hardware, a good motherboard but low consumption CPU, just to keep the consumption down for now, which is the primary goal in general by the way.
5. Sorry but I don't remember if anyone here suggested the max size of an HDD, does it depend on the vendor or it's a general evaluation on disk experience when it's too late (breaking :D).
Price suggests that it make sense to take 4x14TB disks, but due to availability and pricing, it may fall between 12-16TB.

 

[artlessknave]

Btw, can I migrate my pool from Scale to Core?

yes. as long as the openzfs version of the pool is not newer/more featured than the one installed, any OS with openzfs can access the same pool

I would expect to have 3 disks as fault tolerant with 8 disks setup.

raidz3 is overkill in most cases, particularly if you have a backup. raidz2 on 8 disks is pretty standard, good reliability with good space efficiency, lose a bit of speed (slower random access/small files, better streamed large files.

I see that you have the boot drive mirrored, but how?

standard function of TrueNAS since the FreeNAS 8 days. simply attach another disk to a stripe boot pool in the webUI

Why do you have 64GB of RAM on the TrueNAS used as a NAS? I thought that 16GB are enough...

16GB is the recomended minimum (8GB is the absolute minimum) to not have your system choke trying to survive with not enough RAM. more RAM is ALWAYS, in all cases, better than less, with ZFS. I would never run ZFS on anything less than 32GB.
on scale, this is worse, because scale aparently limits ZFS to exactly 1/2 the installed RAM, so on 16GB that would mean only 8GB for ZFS to use.

suggested the max size of an HDD,

whatever size you can afford and fits how you want to use the server, though I would avoid the largest drives as they tend to be bleeding edge and the most likely to have unexpected failures.

also, note that because you kind of snuck this into another thread, you only have about 2 people who know about your questions.

 

[ChrisRJ]

1. Why with 8 disks you use RAIDZ2? Just disk space needs? I would expect to have 3 disks as fault tolerant with 8 disks setup.

A RAIDZ2 with 8 disks is generally considered a proper configuration. There may be special circumstances where RAIDZ3 will be more appropriate (e.g. the server is located in a remote data center), but none of those apply here. A typical RAIDZ3 vdev would have 10-12 drives.

2. I see that you have the boot drive mirrored, but how? Is it through TrueNAS or you did some custom/3rd_party implementation?

That is a ZFS mirror done when TrueNAS was installed. It will therefore not necessarily give me HA for booting the system (a separate HW RAID would be needed here). But I have a copy of configuration and log files. That is sufficient for me.

3. Why do you have 64GB of RAM on the TrueNAS used as a NAS? I thought that 16GB are enough...

There is a certain ration between storage capacity and RAM. For lower RAM levels it is roughly(!) 1 GB RAM for 1 TB storage, but that is only a rough guidance. Otherwise it largely depends on the use-case. Also, since the RAM not needed otherwise (at least on Core) is dynamically used as cache.

In addition, in my case the RAM was a bargain so I just went for more.

5. Sorry but I don't remember if anyone here suggested the max size of an HDD, does it depend on the vendor or it's a general evaluation on disk experience when it's too late (breaking :D).
Price suggests that it make sense to take 4x14TB disks, but due to availability and pricing, it may fall between 12-16TB.

It is indeed a combination of requirements and price. If you want more IOPS with conventional HDDs, more but smaller HDDs are better (ignoring power consumption). For storage that does not require a lot of IOPS the best price-capacity ratio is what I would go for.

 

[TDi39]

A RAIDZ2 with 8 disks is generally considered a proper configuration. There may be special circumstances where RAIDZ3 will be more appropriate (e.g. the server is located in a remote data center), but none of those apply here. A typical RAIDZ3 vdev would have 10-12 drives.

That is a ZFS mirror done when TrueNAS was installed. It will therefore not necessarily give me HA for booting the system (a separate HW RAID would be needed here). But I have a copy of configuration and log files. That is sufficient for me.

There is a certain ration between storage capacity and RAM. For lower RAM levels it is roughly(!) 1 GB RAM for 1 TB storage, but that is only a rough guidance. Otherwise it largely depends on the use-case. Also, since the RAM not needed otherwise (at least on Core) is dynamically used as cache.

In addition, in my case the RAM was a bargain so I just went for more.

It is indeed a combination of requirements and price. If you want more IOPS with conventional HDDs, more but smaller HDDs are better (ignoring power consumption). For storage that does not require a lot of IOPS the best price-capacity ratio is what I would go for.

Thanks for all the help.

Regarding the RAM requirement, here I found the following:

• Add extra RAM (in general) if more clients will connect to the TrueNAS system. A 20 TB pool backing lots of high-performance VMs over iSCSI might need more RAM than a 200 TB pool storing archival data. If using iSCSI to back VMs, plan to use at least 16 GB of RAM for reasonable performance and 32 GB or more for optimal performance.

I'll have 5-6 clients but the mount points may be 8, with every client connecting to max all of them, so 5 clients connecting to all the 8 mount points (that's the worst case so far)
What actually affects the number of GB of RAM in this case?

In the other case, is the pool or the total amount of TB to affect the RAM requirement?
So, in 4x16TB disks, it should be 32GB or 64GB of RAM? (in case of data pool, no VMs or high IO)

• Add 2 GB of RAM for directory services for the winbind internal cache.

I'm not sure if I need to use it.

• Add the suggested 5 GB per TB of storage for deduplication that depends on an in-RAM deduplication table.

What's this?

• Add approximately 1 GB of RAM (conservative estimate) for every 50 GB of L2ARC in your pool. Attaching an L2ARC drive to a pool uses some RAM, too. ZFS needs metadata in ARC to know what data is in L2ARC.

Shall I use it? I don't think so.

 

[artlessknave]

64GB RAM min before even considering L2ARC. L2ARC uses RAM, RAM that could be used for ARC, which is always faster.

again, ZFS is always better with more RAM. no contest. it uses RAM constantly, for everything, and will cache often used data in unused RAM for the quickest access, as well as about 5 seconds of writes before flushing them to disk. the bigger that cache can be, the faster everything has the potential to be.

I would consider 32GB a reasonable minimum for any ZFS build, as RAM is that not expensive. 64GB is a much better number and also not unreasonably priced. for 4x16TB either of these is probably acceptable. more will still give you more ARC, but probably be diminishing returns.

 

[Davvo]

I would consider 32GB a reasonable minimum for any ZFS build, as RAM is that not expensive.

If you only want storage of large files without jails/containers, virtualization, deduplication or VMs and you are fine with 1Gbps performance, 16GB is enough.
Anything else dictates an increase of RAM.

 

[TDi39]

If you only want storage of large files without jails/containers, virtualization, deduplication or VMs and you are fine with 1Gbps performance, 16GB is enough.
Anything else dictates an increase of RAM.

I can't go beyond 1 Gbps anyway due to my FW, 10 GB/s firewalls are very expensive, not only for the network card itself, but also the hardware necessary that needs to process all that traffic...

I need to seriously evaluate deduplication...

The real challenge though is to find components in UK...
Plus Xeon low power are not common at all...

 

[Davvo]

I can't go beyond 1 Gbps anyway due to my FW, 10 GB/s firewalls are very expensive, not only for the network card itself, but also the hardware necessary that needs to process all that traffic...

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

I need to seriously evaluate deduplication...

You sure? It's an expensive feature.

 

[danb35]

10 GB/s firewalls are very expensive, not only for the network card itself, but also the hardware necessary that needs to process all that traffic

Who said anything about 10 Gb/sec (not 10 GB/s; that would be 80 Gb/s, which doesn't really exist) firewalls? If you're needing to pass traffic at 10 Gb/sec between networks, even then a proper switch could do it, and those can still be had at quite reasonable prices.