New to TrueNAS, building a system

[MatMK]

Hi,
my old Synology's best days are over, and after days browsing the web I decided to go with TrueNAS for my new main (and only) storage. Now I am in the process of choosing components, and I would greatly appreciate any tips. I am somewhat proefficient in IT, but a complete noob in storage, so please bear with me.

The main use case will be to store critical data, both personal and work, that I am keen on protecting. The data will be accessed almost exclusively through SMB from Windows, both locally (2.5 Gbps NIC) and remotely (100 Mbps up/down) through OpenVPN. My data consist of file-sizes from all over the spectrum, both a lot of very small and reasonably big. I also have a lot of copies of the same files all over the place, for versioning and backups that I am too lazy to go through . The system will be mainly used by two users; me and one other person with similar usage, but occasionally by others as well. I don't need it to be incredulously fast, just fast enough and reliable.
The second, less important usage, is going to be hosting a media server (just DLNA at the moment), and making use of some utilities offered by the system/applications, e.g. a download manager. I am not using any virtualized systems as of now, but don't want to lock myself out of it if I find the need later. Some ideas I might want to do in the future might include hosting a game server, light web server, home automation, or the like.

I am aiming my build for energy efficiency, for various reasons. The planned power usage should ideally go as low as ~15W on idle. For that reason, I decided to choose Fujitsu D3222 motherboard (one of the recommendations on forums specializing on PC efficiency). I plan to max out its memory (DDR3, non-ECC) to 32GB, more on that later. I bought two server-grade 16 TB HDDs that I plan to mirror for the data, and a second (single) 8TB HDD for the media files that I can miss. I hope the storage will last me for quite a while, as currently I barely fill 3 TB. For OS drive, I plan to use an old 24 GB SSD I have laying around. As for the CPU, I already have an Intel Core i3-4340 that I wanted to utilize. The system will be connected to an UPS at all times.

Despite listing my components above, I am not certain about the built and have some doubts that I hoped some of you may help me answering:

1. Will the CPU be powerful enough? The i3 only has two cores, but I heard that samba is single-threaded anyway. I can buy a better 4th gen CPU with 4 cores, but single-thread performance does not improve that much. To go further, I would need a newer platform.
2. Is the usage of server-grade components important? Another option for me is to ditch this build altogether, in favor of (similarly efficient) workstation-grade system (D3417-B), with ECC support and higher supported RAM sizes, not mentioning more modern platform with higher performance. It makes no wonder that building it would be more expensive, of course. If my specs are good enough, I would prefer to use it as I already have some of the components at hand.
3. As I described, my data contains a lot of duplicates. One of the reasons I chose to go with TrueNAS in the first place is deduplication, but now I found it seems to be discouraged on this forum. Will my CPU be enough? Will the RAM be enough? I read that the 5GB rule per TB is obsolete now, and that you can greatly reduce the RAM usage by increasing the record size, as modern data seem to be larger in general (source). Can someone please comment on this?
4. Would you recommend TrueNAS Core or Scale, based on the usage I described in the first paragraphs?
5. Does the system SSD needs to be reliable? What happens in case of its failure, or the failure of the motherboard? I understand the data won't be lost, but how easy is it to get it up and running?
6. Do you think it would be an advantage for my workflow to use an extra SSD(s)? Whether for some form of cache, dedup table, dedicated metadata?
7. Can the OS put the disks in sleep, if not used for some period of time? I know that spinning down the disks lower their lifespan, but so does letting them spin, and for example the HDD for media does not need to run at all times when it will only be used once a week.
8. For those who have some experience with picoPSU, would you say it is reliable enough to use for this project? Or should I sacrifice some efficiency for a standard ATX PSU?
9. Is using the NAS as a router as bad an idea as it sounds? I could make it the center of my network, DHCP, DNS, etc. and ditch the consumer router, but I am of course worried about security.

These are just some of the things that I figured are worth asking. If you have another remark or suggestion, I would be glad to hear it and discuss with you.

Thank you for your responses!

 

[Davvo]

The thing with the really low power system builders out there is that data safety comes second. We suggest you to use server grade components because A) they have been proven to be reliable B) they can (more or less easily) be found on the used market C) they have great features like IPMI and ECC RAM.

I wouldn't store critical data without ECC RAM.

Hardware Guide:

Hardware Recommendations Guide

This is the latest edition of the FreeNAS Community hardware recommendations guide. The current major version is R2, dated January 2021, with the last minor update on 2021-01-24. The format has moved away from the forum post form factor, to...

www.truenas.com

Introduction to ZFS:

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

Knowing this:
1) it's enough, great CPU.
2) as previously said, we prefer server grade hardware with supermicro being the prime choice since their boards are used in iX's Enterprise solutions. A word about 2.5Gb NICs: here the suggestion is either go big or stay 1Gbps due to a few things. I leave you a resource about it.

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

3) 32GB of RAM won't be enough, that's for sure. There are plans for an improved dedup system, dynamically allocated but it's a thing that, at best, will ship next year. I am no expert about it, but it's generally not worth it. There are a few things you can do that we will see in a few points.
4) I am biased, but I suggest CORE: in terms of reliability, resource efficency and flexibility it has no competitor in SCALE yet... But you can't use plugins and need to manually setup jails (there are a few scripts made by the community that take care of that for some apps); SCALE is the new thing, it has better power management and in theory better flexibility... but there are a few caveats, nanely a kubernet issue that consumes more resources than necessary and a crippling use of RAM that basically halves the ARC available for caching to your system (thanks Linux). I'd say start with CORE and see if it suits your needs, if it doesn't try SCALE.
5) It doesn't. As long as you have a config backup (easily sent to you as part of the great multi_report script for example) you can easily and effortlessly restore your system to the state of art with a simple swap and clean install. Your data is on your drives.
6) A special vdev dedicated to hold the dedup table is worth considering if you choose to go that way. I am not overly confident of my knowledge on the matter, but I think it needs the same level of redundancy of your pool (same as a metadata special vdev). Wait for experts here. L2ARC is also a thing, but I don't think you need that performance boost. I would suggest to bring your RAM to at least 64GB (CORE, double for SCALE) in order to even consider either option.
7) Yes. We even have user scripts for this purpose. https://www.truenas.com/community/resources/hdd-spindown-timer.122/
8) The idea of a picoPSU doesn't exactly inspire me serenity. Please read the following resource.

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can...

www.truenas.com

9) Yes, it really is as bad as it sounds.

You can find more resources in my signature or directly in the resource section of the forum. If you have further questions please ask.

P.S.: make sure the drives you bought are not SMR.

List of known SMR drives

Hard drives that write data in overlapping, "shingled" tracks, have greater areal density than ones that do not. For cost and capacity reasons, manufacturers are increasingly moving to SMR, Shingled Magnetic Recording. SMR is a form of PMR...

www.truenas.com

 

[Etorix]

1/ For storage it's good enough. Depending on what extra duties are asked, more cores and/or more RAM may be useful.
2/ If the data is important, ECC RAM would be a plus.
In the same theme, if data is important I would personally NOT use a 2-way mirror but 3-way mirror or raidz2 (≥ 4 drives).
3/ Do NOT use dedup. The 5 GB/TB rule of thumb is perfectly applicable, especially if you have small files. From personal experience, to enable dedup on a 16 TB dataset you'll need 96-128 GB RAM (for CORE; double that for SCALE!) not to slow to a crawl.
4/ CORE. Not only does SCALE have bad ARC management, it also comes with Kubernetes which takes 10-15% CPU just to stay idle. The memory issue is supposed to be addressed in the next release; the stupidly high idle will not be addressed because Kubernetes developers do not even acknowledge there's an issue.
5/ Always keep a recent copy of the configuration file (exported from GUI) outside of the NAS. If the boot drive fails, plug in a new one, install TrueNAS anew and load the configuration file. If the motherboard fails, just move all drives, including boot, to a new motherboard. (May still need to fix network settings in either case, but that should be all.)
6/ A (metadata-only) L2ARC could help for dedup (which you should NOT use…), but would still require at least 64 GB RAM. Special/dedup vdev would require 3-way mirror if data is important.
7/ Yes, but think hard about wear and actually using more power if the drives come up too often.
8/ Why a pico-PSU to begin with? It doesn't use less power because it's physically smaller.
9/ Having a dedicated OPNsense appliance should be both safer and easier to maintain. And to virtualise everything in a single box you'd really need server-grade components, more cores and more RAM.

 

[Davvo]

Overall, the D3222-B is a good board: good PCIe, good SATA ports number, has a USB internal port for a USB (I use one as boot drive)... pity it doesn't support ECC RAM and hasn't IPMI, it would have made a killer board.

The D3417-B is a good choice if you wanna stick with Fujitsu: up to 64GB of ECC RAM, single m.2 slot for L2ARC or SLOG, good PCIe and SATA ports number... still doesn't have IPMI, and they apprently removed the internal USB port, but it's not that bad for home use... especially for the price (€45 used).
If it works, obviously.

 

[Heracles]

You already received great advices on your questions. I would also add that server-grade parts are designed to run 24x7, as opposed to workstation parts. They are also meant to endure more vibration, offer more monitoring interfaces, etc.

About dedup, do not touch this below 128G of RAM and even then. Depending of what is causing your duplicate, you can adjust your data management accordingly. Ex : you have 5 users starting from the same basis but who then change their instance in whatever way they wish. For that, you create that common starting point, snapshot it, clone that snapshot 5 times and use one clone per user. That way, you start with a single copy of that base and only the differences will be written on disks.

The reason for not using dedup with less than 128G is that the entire dedup table must stays in RAM 100% of the time. Not only does it need to fit in the RAM but that RAM is not available for caching anymore. Again, never touch dedup.

About putting disks to sleep, that is a no-go. You will kill your drive in no time. Drives are designed to stop and start only that many times. TrueNAS by itself will do a lot of access just for housekeeping and will wake these drives all the time. Drives will last much longer when they are keep hot. The cost of replacing a single drive is way more than the cost of power required to keep your few drives up all the time. There is no savings doing that.

I decided to go with TrueNAS for my new main (and only) storage.

This is probably the most important point :
No matter how robust a TrueNAS server be, a single server will always be a single point of failure. Destroyed by a single physical incident like fire or by a single logical incident like human error, your data will always be on the edge and one step from complete lost. Backups are not optional, even less for critical data. See my signature about backups...

 

[MatMK]

Thank you all for your advises. I read them all and noted your suggestions.
I decided in the end to go for the ECC after all, for better data protection as you said. I am also going to install Core, as per your recommendation.

Now I am deciding on whether to go with the mentioned D3417-B (from workstation-grade Fujitsu Celsius W550), or choose D3373 (from server-grade TX1320 M3). Both have ECC, are reasonably cheap and are using the same chipset. The latter does also have the mentioned IPMI. But there is still the question of power efficiency.

Do you think IPMI is really necessary for a home system? Aren't the remote management features of TrueNAS enough? From another point of view, the IPMI is drawing additional constant power that can't be turned off (based on what people say online for this exact unit, which is otherwise quite efficient).
The board also features SAS controller, which I don't really need, and provides only 4 SATA ports if I understood correctly. It is meant to be used with add-on card for another SAS controller, but that alone is so power inefficient it looks like it needs cooling on its own.
Also, do you think I might have some use for the second Gbit port? Would it be possible for example to aggregate them to use 2Gbps?

As for the dedup. Thank you all for your recommendations. I will try to learn more about it and create a new thread later, if I have more questions.

A word about 2.5Gb NICs: here the suggestion is either go big or stay 1Gbps due to a few things. I leave you a resource about it.

I read the resource you mentioned (all of them in fact, thanks). But I can't seem to find a lot of info on 2.5Gb NICs. If I understand you correctly, you recommend to go with SFP. The thing is, there are already CAT 5e cables in walls in our house, so going above 2.5, maybe 5 Gbps is sadly out of the question. Could you please elaborate on why 2.5 Gb NIC (or a 5 Gb one), would not be recommended? Thank you

Why a pico-PSU to begin with? It doesn't use less power because it's physically smaller.

The efficiency does not come from it being small, it comes from DC to DC conversion. It is true that AC from the wall has to be converted at some point, but commonly available 12V adapters seem to have better efficiency than ATX PSUs (don't exactly know why). Some people report nearly 2x reduction in power usage for sub 15W systems. Anyway, it looks that the Fujitsu PSUs are also pretty efficient, so I will keep it simple and use it for the moment.

Backups are not optional, even less for critical data.

I wasn't clear on my first statement that you commented on. I have an additional external USB drive that I back up on, but as it is kept offline, I didn't consider it in my original post.

 

[danb35]

it also comes with Kubernetes which takes 10-15% CPU just to stay idle.

BS. This is a meaningless figure, not least because people run TrueNAS on everything from 2-core Celerons to dual E5s (and perhaps higher). As such, it's just FUD. I agree with the recommendation of CORE here, but this is a poor argument for it.

My system reports an average CPU idle of 94.5% over the past 24 hours--that's the entire system, not just k8s, and the system certainly hasn't been idle during that time. And that's with 20+ apps running, too.

 

[Davvo]

Do you think IPMI is really necessary for a home system?

It's not a necessity, just a very nice thing to have.

From another point of view, the IPMI is drawing additional constant power that can't be turned off (based on what people say online for this exact unit, which is otherwise quite efficient).

imho it becomes relevant only when talking about very small consumption... I doubt the board itself will consume more than 25W.

The board also features SAS controller, which I don't really need, and provides only 4 SATA ports if I understood correctly. It is meant to be used with add-on card for another SAS controller, but that alone is so power inefficient it looks like it needs cooling on its own.

According to what I have been able to find (these fujitsu boards manuals seem to be hard to find... about this I wans't able to find even the datasheet of the board itself, just one regarding a server with that inside) the board does not feature a SAS controller, and as such needs one if you want to conncect more drives than the SATA ports can manage: namely, an HBA; your points regarding the increased power consumption as well as heat generated are true. About HBAs, please read the following resource.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Also, do you think I might have some use for the second Gbit port? Would it be possible for example to aggregate them to use 2Gbps?

Not for a single user. You could use it in failsafe mode (if that's called) but I believe that the switch your NAS will be connected needs to support it as well. The rabbit hole called networking is deep.

I read the resource you mentioned (all of them in fact, thanks). But I can't seem to find a lot of info on 2.5Gb NICs. If I understand you correctly, you recommend to go with SFP. The thing is, there are already CAT 5e cables in walls in our house, so going above 2.5, maybe 5 Gbps is sadly out of the question. Could you please elaborate on why 2.5 Gb NIC (or a 5 Gb one), would not be recommended? Thank you

I reccomended you to either stay 1Gbps or go 10Gbps (cabled, not SFP since you already have the cables) because 2.5Gbps cards (or their drivers) are... not state of the art to put it kindly, and even Intel ones have caused issues to users. My suggestion? Stay 1Gbps, especially since you seem to have power saving very close to your top priorities. Also, it's not obvious that your pool will be able to go beyond that number... it depends on your files sizes.

I would say, for your use case and if you want to stick with the fujitsus, go with the D3417-B. It has enough SATA ports for you to use in any case you will resonably pair with it, with space to add an HBA, a network card or whatever you want in the future if needed. It doesn't have IPMI, but if you are careful (and with a bit of luck) you won't have a need for it... especially since your CPU has a iGPU. It also has a m.2 slot.
And if something goes wrong you won't be suffering a big loss since it's pretty cheap (assuming you buy used).

The efficiency does not come from it being small, it comes from DC to DC conversion. It is true that AC from the wall has to be converted at some point, but commonly available 12V adapters seem to have better efficiency than ATX PSUs (don't exactly know why). Some people report nearly 2x reduction in power usage for sub 15W systems. Anyway, it looks that the Fujitsu PSUs are also pretty efficient, so I will keep it simple and use it for the moment.

You should really do your work before choosing such PSUs since you risk undersizing it. The D3417-B has a surge input of 180W for "<10ms" and one of 120w for "<100s". Add drives, fans, and the SSD... you really don't want to undersize it. I already linked the proper PSU giodance, make sure to do your homework... and consider looking at staggered spinup.

In which case were you thinking of putting this exotic thing?

---

About putting disks to sleep, that is a no-go. You will kill your drive in no time. Drives are designed to stop and start only that many times. TrueNAS by itself will do a lot of access just for housekeeping and will wake these drives all the time. Drives will last much longer when they are keep hot. The cost of replacing a single drive is way more than the cost of power required to keep your few drives up all the time. There is no savings doing that.

Reasonable spin down is not as serious as you make it be. But it's a long time discussion, one which @joeschmuck (if I'm not wrong we talked about it recently?) very well knows... generally, it's best not to spindown the drives in order to make them last as long as possible, but reasonable spindown is not of concern (TL;DR: do not spin them up and down dozens of time every day).

BS. This is a meaningless figure, not least because people run TrueNAS on everything from 2-core Celerons to dual E5s (and perhaps higher). As such, it's just FUD. I agree with the recommendation of CORE here, but this is a poor argument for it.

My system reports an average CPU idle of 94.5% over the past 24 hours--that's the entire system, not just k8s, and the system certainly hasn't been idle during that time. And that's with 20+ apps running, too.

Quite a few users reported such number (or close to it), not really sure I'd call that BS. Especially since the issue is undoubtedly there.

 

[Etorix]

Namely, the issue is discussed in this tread, with post #23 reporting on the lack of response from the Kubernetes team (no direct link to the GItHub issue, unfortunately).
Fore heavy deployments or very active containers it may be of little significance, but not for little home servers with few or no containers—especially if there's an explicit focus on "energy efficiency", as here.

 

[MatMK]

I would say, for your use case and if you want to stick with the fujitsus, go with the D3417-B.

Thank you for your responses and recommendations in general. The D3417-B it is.

You should really do your work before choosing such PSUs since you risk undersizing it. The D3417-B has a surge input of 180W for "<10ms" and one of 120w for "<100s". Add drives, fans, and the SSD... you really don't want to undersize it. I already linked the proper PSU giodance, make sure to do your homework... and consider looking at staggered spinup.

In which case were you thinking of putting this exotic thing?

I will definitely look into it if I decide to use the pico. For now, I will be staying with the original ATX PSU that comes with the board (as I bough it as a whole PC from the seller).