Home NAS solution - How to contain costs

[Torrone]

I rely on cloud, but I use encryption, which means that cloud sync is not for me anyway, unless I encrypt the data before.

For cloud encryption, I use Rclone.
It's a very powerful tool for that but not easy to handle at first.

 

[ChrisRJ]

While @danb35 and @HoneyBadger bet me to it, I would like to reiterate the "used enterprise gear" topic. Back in September 2020 I built my current NAS (see signature) and couldn't be happier. The system uses around 90 W idle, which I think is reasonable. Only the case and the required fans (i.e. their noise) for 8 HDDs are less than ideal. But I can live with that.

 

[danb35]

For cloud encryption, I use Rclone.

...which is what TrueNAS uses for its Cloud Sync feature anyway.

 

[Torrone]

Cool!
This will make it easier for me to get started with TrueNAS!

About possible (and potentially less expensive) alternatives to RAIDz, here is a comparison (be careful of biases, it's made by SnapRAID) :

SnapRAID

A backup program for disk arrays. It stores parity information of your data and it recovers from up to six disk failures

www.snapraid.it

 

[TDi39]

Thanks everyone so far.

I actually need to buy another machine to develop properly, hence, I could use that with TrueNAS and run VMs in it, but it's not the same as running VMs on Linux, right? It's gonna be another challenge, maybe it will get me stack for 3h, who knows...
In any case, I should migrate my VMs, they run on KVM, I hope I won't meet any blocker.

If I go on that road, I'll spend 550 for a machine and 600 for disks, in that case I take all for less than 1200.

I don't want to expose my NAS on internet, I'll just need the VMs to connect to internet, which probably will have a dedicated network card for that specifically, so I can reduce the network attack surface, or config mistakes mostly.
Just to clarify, I wouldn't use the storage pool for VMs, I have an M2 that boots with Linux currently, so I can use that M2 for VMs and the back it up to the ZFS pool.
I could do 1:1 mirroring or I can convert the M2 into a data disk where there are only VMs, because currently I have OS and VMs there, at the end it will be always a 1:1 mirroring, I just don't know if the current partition layout could cause a problem.
Anyway, this story of all in one machine doesn't make me feel great.

My concern was having the issue of an all in one solution, but also, such machine would need a lot of wattage, it will produce a lot of dust and it may be also noisy (this is crucial, because it's sitting around my workbench :D ).
It will run in my living room unfortunately, so it can be a serious concern to have a big machine like that running all the time.
I currently have a cluster which will be decommissioned soon, honestly it's not so noisy, I think I can find a similar one on the market.

What about summer time? Well, this one was quite crazy :D . I wouldn't be able to cool down the disks properly, I don't have a proper AC system unless I purchase one separately, like one to just plug in (I rent and I can't fit one).

The cluster specs would be as following:
10-12 physical cores Xeon 26xx... (1 CPU socket)
128GB RAM ECC
2TB M2
3x12TB disks CMR
1-2 additional network cards, this could cost a bit... (1x 2.5 Gbit is not necessary anymore because all runs on the same machine)
Good case, noise and dust proof, cooling shouldn't be a problem unless in the summer, but that's a problem with many other devices...

 

[Torrone]

I actually need to buy another machine to develop properly, hence, I could use that with TrueNAS and run VMs in it, but it's not the same as running VMs on Linux, right?

Have you looked at what is possible with TrueNAS SCALE?
It should fit your needs I think: https://www.truenas.com/truenas-scale/#_

It's based on Debian stable and you can run Docker, KVM, Kubernetes...

 

[TDi39]

Having extra services on the NAS to serve the local LAN should be safe. Security issues come with exposing the NAS to the wider world. Use a VPN to access the NAS and its services remotely.

You can run publicly accessable services in jails and VMs. The security implications are no different than with any other virtualisation platform. Many users seem to run Nextcloud in a jail.

I don't need to expose it on internet, so it's fine, my concern was to have all in one machine for security reasons even on my LAN only.

Plus, during night is pointless to run something so powerful because the systems would be in idle doing nothing, but the wattage would be still too high. Shutting it down could cut 1/3 of my electricity bill, at the moment I'm not sure if it's a viable solution.

 

[TDi39]

Have you looked at what is possible with TrueNAS SCALE?
It should fit your needs I think: https://www.truenas.com/truenas-scale/#_

It's based on Debian stable and you can run Docker, KVM, Kubernetes...

It seems I don't know quite a lot of this product then :D .
That's really cool, I won't have issues to move my VMs.

The only thing is that my current hardware is the following:

- AMD 1600
- MSI motherboard 4x SATA, 1 M2, 1 eth
- 32GB gaming RAM, not slow ones :D , but they are not ECC..................
- 1x2TB M2
- 256GB SSD for boot
- Decent case, but I can buy a better one

In the case of an all in one solution for VMs and NAS, the RAM is not sufficient and it's not even ECC anyway, my MB doesn't support ECC memory, this is an old gaming PC.
Reason why I was thinking to spend 400 pounds for a proper machine and add up disks as needed.

I could keep the old gaming PC only for the NAS solution, but no ECC memory anyway.

So, I think that the best is to buy another machine anyway for the NAS, the decision now is if to have a NAS and VMs in the same machine.
I want to clarify that the NAS portion on the HDDs will not be used by the VMs, but only for backups, I have the M2 to make them run fast :D .

Electricity wise, it would be more expensive to run 2 different machines, and also more expensive by the cost of the items.

Purely cost wise, the best choice seems to have an all in one solution, but after only I assessed the security implications to have an all in one machine, this will be a longer research.

Noise? Well, better one machine, but I'll probably invest in a proper silent gaming PC case, because the server ones don't come up with the intent to keep them silent, even though I've seen a couple of silent workstations out there and I have one under my nose.

Operation issues:
1. When the electricity goes down and the UPC starts to work, the host goes in stand-by and the VMs are still running, I hope I don't run into some corruption :D . I tried it in the past, apparently all was fine, but I didn't test it properly, nor I've read proper resources online, this must be verified...

2. Driver issues for MSI?

3. Is TrueNAS Scale actually the same product as TrueNAS? It runs on Debian, but is there any reliability compromise?


Thanks everyone so far, really helpful.

 

[MisterE2002]

3. Is TrueNAS Scale actually the same product as TrueNAS? It runs on Debian, but is there any reliability compromise?

SCALE is quite similar but somewhat new. Performance is currently lacking a bit (see Lawrence Systems youtube video) but will probably better in the future. iX declared it Stable (opinions vary) but i do not have noticed real issues. Note: only CORE supports jails currently.

 

[Etorix]

It seems I don't know quite a lot of this product then :D .
That's really cool, I won't have issues to move my VMs.

Note that you can run a Linux VM on TrueNAS CORE as well…

The only thing is that my current hardware is the following:

- AMD 1600
- MSI motherboard 4x SATA, 1 M2, 1 eth
- 32GB gaming RAM, not slow ones :D , but they are not ECC..................
- 1x2TB M2
- 256GB SSD for boot
- Decent case, but I can buy a better one

Not optimal but serviceable for a first try at a small NAS if the NIC is Intel and not Realtek. Some Ryzen boards do support ECC unofficially.
The lack of ECC is not redibitory, but if you're buying to build it's indeed better to go for ECC.

2. Driver issues for MSI?

It's an issue for components, not for the motherboard as a whole, or its maker.

3. Is TrueNAS Scale actually the same product as TrueNAS? It runs on Debian, but is there any reliability compromise?

The ZFS part is supposed to be on par, but obviously SCALE does not have the same amount of field testing as CORE.

 

[TDi39]

Note that you can run a Linux VM on TrueNAS CORE as well…

Yes I know, but the migration could have some issue, for example, config issues...
Currently I run my VMs on Linux (KVM).
Having Debian will be so sweet.

Not optimal but serviceable for a first try at a small NAS if the NIC is Intel and not Realtek. Some Ryzen boards do support ECC unofficially.
The lack of ECC is not redibitory, but if you're buying to build it's indeed better to go for ECC.

Unfortunately I have Realtek, as every gaming board...

Now the choice is with a single mega machine or 2 machines, one cluster and one small for the NAS.

Well, I have enough info, thanks everyone so far, now let's make the choice

 

[ChrisRJ]

Unfortunately I have Realtek, as every gaming board...

Not a big issue, just get yourself a used Intel NIC and you are fine.

 

[TDi39]

Not a big issue, just get yourself a used Intel NIC and you are fine.

That's right, I forgot that I needed only one port, not many like in a switch :D , it's gonna be cheap

 

[TDi39]

Can someone tell me what's the difference between TrueNAS Core and SCALE when it comes to the following statement:
CORE

High Availability	No

SCALE

High Availability	Yes

Source: https://www.truenas.com/compare-editions/

I thought that CORE was the primary supported version, therefore I would expect high availability, or is it referring to the case of not having a pre-built hardware solution?

 

[Patrick M. Hausen]

High availability commonly means clustering of multiple machines and seamless failover without service interruption in case one of them fails.

 

[artlessknave]

if you just want to serve some files and maybe a container or 2:
x9scm from ebay: 70$ (the BIOS MUST be updated due to a bug)
e3-1230 v2 from ebay: 40$
32GB ECC ddr3 from ebay: 100$
optional: SAS card from ebay: preflashed: 100-150$, unflashed 50-100$
do 2 m
NOTE: do NOT use raidz1 on 3 12 TB drives. do 2 mirrored drives or get 4x12TB and raidz2/mirrors.

nothing says you cant fire up the board you already have and try things out. you can also use that consumer board to make a backup system, because raidz is NOT a backup. no ECC is still not ideal, but it's an "oh shit" copy anyway.

 

[TDi39]

Thanks both, so here it's my reasoning on TrueNAS.

- CPU 8/16 threads max (2018 XEON probably)
- 128GB ECC DDR4
- The rest will be accordingly

Disks | I have a few options:
- 3 disks in RAIDZ1 (I think I can have 2 drives failure with this type of RAIDZ, but only one drive is used for the active pool...)
From the documentation: A RAIDZ1 uses one disk for parity while all other disks store data. A RAIDZ1 requires at least three disks.
It seems strange to me that it uses 2 disks for data and only one for parity... Does it mean that I can lose 2 disks actually? (one from the pool and one of parity???)

OR 4 disks in RAIDZ2 (2 drives failure, in this case I should have more performance)
- More disks doesn't make much sense for home setup.
- Each disk will be minimum 12 TB, but I'll try to get 16 TB if I find a good price, any problem with the size of each disk??

Use case
I'll use my NAS for projects (software development), media etc, but also VM backups, basically data that I don't want to lose.
A lot of writing every day, let's say 100-150GB a day but big blocks, not a big drama for an HDD...
If it's better, I can just take snapshots on daily basis, but backups once a week for example... Something tells me that this is better... I just need to find a proper strategy.
Otherwise, it will be in idle...

I'll use VMs on the same systems because it seems that I won't have better security running 2 different systems, but they'll run on the M2, only file backups will be on the NAS.
The VMs may be accessed through RDP or similar, so audio issues should not impact me because I'll have another machine where to stream audio...

Where's my real backup?
On a secondary disk, powered by the NAS anyway which is powered by an UPS, or maybe from a Laptop connected to the UPS (laptop batteries are hateful), we'll see.
Plus my cloud account, but I think I should use another specific service for TrueNAS just to make it easy, I still didn't evaluate this point...

What kind of backup is this?
This is the most ridiculous backup, but it works well for me for only one reason, it's just a copy paste (or rsync) of critical data mainly + VM backups (which I may afford to lose one of 2 days of work, in any case I use remote repositories)
Why did I choose this method for the beginning? Because it's very simple, that's it.
I can't mess up with restoring the entire pool, I may end up in a situation that I don't know how to fix... Reason why this may become my permanent backup method + the one on cloud.
I may also get an hardware issue and it will take a lot of time to fix.
(I'll try to practise on a VM, but I need to get started with a production environment asap)

About containing costs...
The cluster will cost me 22 pounds of energy per month average (with the current prices estimated for 90W average 24/7), any suggestion on how to lower down consumption? :D

I know something about UEFI config, but I wonder if the XEON could have better C States or similar options...

Unfortunately, I decided to build only one machine, instead of having one NAS and one cluster (used development in VMs), it's simply cheaper (on the day 1), but not that much more expensive if I had 2 machines instead, even on the long run, so I thought it was a better idea.
Plus the energy prices should go down in future, probably as before at some point, which could lower the cost at 13 pounds monthly.

GPU or NO GPU?
I don't see the reason, but it's also true that I still don't deploy the software and I have a complex network here...
I think that it will be necessary at least in the beginning, but then I'd remove it due to the electricity consumption.

But what GPU? I don't see many recommendations...
Plus, I don't want to install garbage drivers and this is not even allowed in TrueNAS due to your image, right?
So, shall I rely on the IPMI or similar?

How to spot HDD issues
I have a fantastic case, but it's not meant for hot swap of disks, so I wonder how I can be effectively notified for broken disks or pool issues...

 

[Patrick M. Hausen]

- 3 disks in RAIDZ1 (I think I can have 2 drives failure with this type of RAIDZ, but only one drive is used for the active pool...)
From the documentation: A RAIDZ1 uses one disk for parity while all other disks store data. A RAIDZ1 requires at least three disks.
It seems strange to me that it uses 2 disks for data and only one for parity... Does it mean that I can lose 2 disks actually? (one from the pool and one of parity???)

RAIDZ1 means you can lose one disk and still be good. Lose two and the data is toast. RAIDZ2 means you can lose two disks and still be good. Lose three and the data is toast.

Unless you want to run transaction heavy workloads like databases or VM "virtual disks" directly on those pools, RAIDZ2 is generally considered the sweet spot for resiliency/cost considerations in home or self-employed environments. For example 6 disk RAIDZ2 gives you a real good capacity/cost ratio. Backups and "read mostly" workloads are perfectly OK.

If you intend to not only do backups of VMs (via NFS I assume?) but run VMs via NFS or iSCSI directly on the NAS volume, a set of mirrors is preferrable.

 

[artlessknave]

The cluster will cost me 22 pounds of energy per month average

that seems really high. that sounds more like the yearly power costs, not monthly, unless maybe you are running prime95 all month?

 

[ChrisRJ]

that seems really high. that sounds more like the yearly power costs, not monthly, unless maybe you are running prime95 all month?

Are you aware of electricity prices in Europe? 50 cents per kWh is not a bad price these days here in Germany. And from what I have read it is not very different in the UK.