Low capacity AMD build

[tupui]

Hi all,
I have spent the last days looking around this great forum to try build my first TrueNAS (also a first with zfs) system. I think I have a good overview now and need some last advices to close the loop.

My needs

I have only around 500GB of data now and plan for at most 1-2 TB in 5 years. I only store important docs, code and RAW photos (and spend a lot of time to only keep ones I care).

I would use this box as the main storage for that plus a copy of my current system with TimeMachine (500GB to 1TB max). So connection to a MacBook Pro and sometimes a Windows laptop.

I don’t want to do any VM, Docker, etc. on this. It’s just for data storage/viewing. Most I could consider doing is having a Lightroom library on it. So light load.

What I want

A secure and fast setup. By fast I mean low latency and close to saturating a 10gb link (I only need to connect the box to my MacBook Pro with a 10gb adapter: SPF for thermals). The secure part would be raidz2 I think as I don’t like the 3 way mirror solution. (As for backup, I plan on having a replication job to a drive I would rotate offsite. I also consider building the same box twice.)

I will be buying new and don’t want second hand stuff (except maybe for a network card).

Plan so far

• Eaton UPS Ellipse pro 1200
• Fractal design node 804

• Motherboard
  • ASRock X470D4U
  • ASRock X570M Pro 4
  • ASRock B550M Steel legend (2.5gbe Dragon?)
  • Asus TUF B550M-Plus (2.5gbe)
• AMD Ryzen 5 5600(G if not integrated)
• x2 Kingston server premier ECC UDIMM 32GB 2666
• EVGA 550W gold full modular
• 10Gtek (Intel x520 SPF)

• Boot drive: Kingston A400 120GB x2 in mirror (M2 or Sata to USB3 cable to spare ports and use USB3 ones?)
• Data: 1 raidz2 vdev (4 to 6 drive)
  • 1TB SSD Samsung 870 evo
  • 2-4TB iron wolf

Question points

1. I am not sure about the motherboard. I don’t need IPMI as I will be all the time with this box and don’t want it to be on any network. From my list and reading here, the x570M looks fine. But some other cards might do and even have 2.5gbe. Only hard requirement is ECC
2. I would start with x2 32GB ECC UDIMM. Do I need 3200 MHz or 2666 is enough? If I read some MB’s diagrams correctly, if one day I want to update, then I need the 2666 and not the 3200. This is also half the price!
3. Is my network card fine? I just need 1 port.
4. I’ve read this nice config, I don’t think I need nvme, but maybe SSD over HDD to achieve speed and low latency.
5. What do you think about the mirror boot on sata to USB3? Shall I just go with M2?
6. Since my storage needs are slim, I think 1 pool with 1 vdev raidz2 is fine. I am not sure about having a working pool and a storage pool (unless it’s easy to automate copy from working to storage. In which case I could do 1 pool of nvme or ssd in mirror and 1 storage pool of hdd in raidz2). Then only Lightroom would be on the fast pool. But maybe just going full ssd is enough and I don’t need to bother with 2 pools. I am really open to discuss this.

Thanks in advance!

 

[MisterE2002]

Why did you choose a AMD setup? Just curious. Most people use the trusted and proven Intel setup. It just works. Although the CPU, motherboard and memory seems to have ECC support try to find forum post of people with similar setups.

And if still correct nowadays. Intel (power/idle) support support is usually better in Linux/BSD.

 

[tupui]

Why did you choose a AMD setup? Just curious. Most people use the trusted and proven Intel setup. It just works. Although the CPU, motherboard and memory seems to have ECC support try to find forum post of people with similar setups.

And if still correct nowadays. Intel (power/idle) support support is usually better in Linux/BSD.

Thanks for the reply Main reason would be price I suppose. Also I recently build some AMD setup and I am pretty happy with these. (Added bonus is making good competition for intel). Supermicro, which seem to be an obvious choice here, is not easy to find new in EU (at least I did not find these), and ECC makes the CPU choice hard. But maybe I missed a nice combo and would be happy to reconsider!

 

[MisterE2002]

Sure who does not like the underdog. (although that is currently Intel). I really had no trouble finding Supermicro stuff here in the Netherlands.
What do you find difficult with ECC? Basically Pentium, i3 and Xeon. It is all described in the guides.

Beware between sandy bridge en coffee lake. They use the same socket but they are not compatible. X11SC. The "C" marks the "generation"

• https://www.truenas.com/community/resources/hardware-recommendations-guide.12/
• https://www.truenas.com/community/resources/freenas®-quick-hardware-guide.7/

 

[tupui]

Thanks I will have a closer look at the i3, as it should be equivalent to a Ryzen in some config. (I did read all these guides, for some reason I found it hard still to parse and make a concrete decision out of them. Would be handy to have a community listing of existing working config. Usually I use pcpartpicker and with ECC it just does not really work, some ECC ram is not marked ECC, same for MB.)

Setting aside the MB and CPU, what do you think about the rest?

 

[MisterE2002]

You can also check users footers. But everyone has different requirements.

Note: if you use server boards you get VGA connectors, so IPMI is really nice to have. In my case a requirement and had bad experience with the java versions. Only the > X10 line are using the newer ASPEED 2500 BMC. So, the choice was easy.

That said, i bought the X11SCL-F-O Retail. This is "Low cost" board. Compared with your board the price seems similar: 300 Euro.

Later i discovered some disadvantages. I must use my x8 HBA in the x16 slot to have enough lanes.
1 PCI-E 3.0 x8 (in x16),
2 PCI-E 3.0 x4 (in x8)

And no video through the VGA port. Not sure my crappy monitor can not handle the resolution or this is impossible because of the chipset. But i am using IPMI anyway.

https://www.truenas.com/community/threads/freenas-build-with-igpu.84347/

Simple rule: C2x6 can use an iGPU provided the board / BIOS do their part; C2x2 won’t.

 

[tupui]

I know about IPMI, but I really don’t want it. I will always be directly connected to the box and don’t want to make this accessible over wan (being behind a VPN etc or not). And if this ever change in the future: pikvm would do the job.

 

[tupui]

Do you have an online retailer (in EU) suggestion? I am almost only using Amazon as I like the peace of mind it offers (had tons of problem through years and they always got my back).

 

[Etorix]

I will be buying new and don’t want second hand stuff (except maybe for a network card).

A real pity if price is a consideration. A NAS does not need the latest and greatest, especially if it's "low capacity".

Boot drive: Kingston A400 120GB x2 in mirror (M2 or Sata to USB3 cable to spare ports and use USB3 ones?)

One drive is enough. Just keep a copy of the configuration file outside of the NAS to reinstall if the boot drive fails.

Data: 1 raidz2 vdev (4 to 6 drive)

• 1TB SSD Samsung 870 evo
• 2-4TB iron wolf

What is the SSD intended for? If it's L2ARC, you'd need at least 128 GB RAM for that!
For up to 6 drives, the Node 304 is smaller and quieter than the 804, although it limits the motherboard to mini-ITX, which is a complication on its own.

Is my network card fine? I just need 1 port.

Anything Intel is basically fine. Anything Realtek (Dragon in gamerspeak) is NOT.

Supermicro, which seem to be an obvious choice here, is not easy to find new in EU (at least I did not find these), and ECC makes the CPU choice hard.

Have you tried through geizahls.at? Supermicro boards are easy to find in Europe, only not through consumer/gamer-focused retailers.

 

[MisterE2002]

I know about IPMI, but I really don’t want it. I will always be directly connected to the box and don’t want to make this accessible over wan (being behind a VPN etc or not). And if this ever change in the future: pikvm would do the job.

Just a sidenote and i will stop bothering about it :) IPMI usually has a separate NIC so you can shield it from the normal LAN (for example using a VLAN).

Not that familiar with pikvm but seems a bad (expensive) compromis. If it works similar to latest Linus Tech Tips with wires you need to connect.

Can not help with Amazon sellers. I used normal dutch (web)shops.

 

[tupui]

@Etorix the ssd was not for the cache, it was either ssd or hdd. I detailed a bit more bellow.

I am not sure about the config to get decent speed. I think I want raidz2 with 6 drives to get the x4 speed (read that somewhere). Just not sure if hdd would be fast enough to get close to 10gb. This is also why I could consider going 1 pool mirror nvme and 1 pool raidz2 hdd in the hole to be cheaper than just a raidz2 of ssd.

For the node802, I think it’s better for the evolutivity and less constrains than the smaller case. Also thermals would be better if the case is not full.

Thanks for the website! I will look into it.

 

[joeschmuck]

Just my two cents...

I echo the fact to only use a single boot drive. Always keep a copy of your configuration file outside of the NAS. Also, mirrored boot drives does not mean automatic failover, that only comes with a true RAID card with automatic failover. And if the NAS will be used as you say, once you configure the NAS, it's doubtful you will be making any frequent changes to it. Lastly, there is a copy of the configuration file on the pool if you are using a default setup, so recovery of that data is fairly easy, but not as easy as just having a copy on another machine for quick recovery.

I would recommend a single SATA SSD, anywhere from 32GB to 120GB (whatever is cheaper when you purchase it). Do not think about wasted space on the SSD, this is a cost game. The SSD will add to the longevity of the boot drive and make software upgrades significantly faster than USB.

While there have been problems in the past (kind of distant past) with AMD CPU's, the problems do not seem to be there or at least I haven't heard of any in a while. With that said, I stick to the Intel CPU's myself for the server, personal preference.

Saturating a 10G connection will depend more on file sizes. If you are passing a lot of small files, well it does take longer to read those off the NAS, but a RAIDZ2 is likely a good compromise. Mirrors would provide better read performance in this scenario. Writing to the NAS is usually not an issue if you have enough RAM, and it sounds like you are planning on having 64GB or RAM. You would be fine with just 32GB for your needs but 64GB is good too and I defaulted to 64GB as well even though my needs are small too.

As for RAM speed, you do not need fast RAM for a NAS so the slower memory is fine, when compared to the hard drive speeds, the RAM is always faster. This too is a price comparison and buy the least expensive good quality memory.

A suggestion for the hard drives... purchase three 4TB hard drives, 5400/5600 RPM type, not 7200 RPM if you can help it. Create a 3-way mirror. test it out for transfer speeds and see if this works for you. The operative word here is TEST, do not put data on there that you cannot destroy. If you do not like it, buy one more 4TB drive and make a four drive RAIDZ2. There are a lot of ways to configure a system and it's best to set one up and test it yourself. If you do not like it you can destroy the VDEV and recreate it in another way. Why a three way mirror? Read speeds are better for random small files. This also lends itself to lowering your costs if the setup works as expected. Also the added benefit of only three hard drives producing heat and noise, and 7200 RPM drives generate more heat and use more power (not much power but that is what helps create the heat) and would not really provide you a noticeable gain in data speed. Also, there are many different ways a Vdev/Pool can be established to provide fast read/write operations, this is just one that might be fine for you.

Why 4TB drives? Because it will be about 3TB of usable space and a NAS will perform better the less full it is. Other factors: The drives should last about 5 years, and people always use more space to store crap just because the space is there. If you end up using Snapshots, that too will consume space that is otherwise hidden as being used. And remember that the rest of your parts should last 10+ years if you buy good parts.

Speaking of heat... Think about the air flow in the case to make sure you are providing proper cooling to the hard drives and across the motherboard components.

You never mentioned if you needed a quiet device. Larger solid cases with large slow turning case fans, slower hard drives, these make for a quieter system. Not that you will have a loud one, but small fans make more noise to move the same volume of air of a large slow turning fan.

And I will just put this out there, do not be fooled into adding a SSD for a Cache or ZIL, these are great for a production systems but not always a good thing for a home system.

It sounds like you have been doing a lot of research, that is great and please continue to do so. Too many people will just post something like "Will this work" and list some random components they got off of some parts place, hoping we would do all the legwork for them.

As for IPMI, if you buy a motherboard with it, you can disable IPMI if you desire. But if your network is behind a firewall then there is no need to worry about it. It does come in handy when you would like to remove the Monitor and Keyboard from the server, then you can use the IPMI to remotely control and do everything you would do from a directly connected monitor/keyboard. I didn't think I'd use mine often at all but it turns out that it allowed me the freedom to place my NAS in a place in my house that is either cooler, or just out of the way. Very nice option to have but definitely not absolutely required if you have the means to connect a monitor and keyboard to the system. But it you find a motherboard that supports IPMI, and the price difference is minimal, it's worth it in the long run. Just my opinion.

Good luck on your selection of parts and I hope your system works well.

EDIT: Just to add, I can't speak to the 10Gbit connection, I don't have experience there but I know some people have difficulty getting it to work properly and some get it to work fine. Search the threads and resources for this stuff.

 

[Torrone]

Do you have an online retailer (in EU) suggestion? I am almost only using Amazon as I like the peace of mind it offers (had tons of problem through years and they always got my back).

I bought my Supermicro motherboard here: https://www.serverparts.pl/en/
My computer case here: https://www.computeruniverse.net/de
The CPU here: https://www.senetic.net/
This retailer also ship to UE: https://direkt.jacob.de/

I use the offical Crucial shop for the RAM, it was cheaper than other retailers, and also amazon for several parts.

 

[Etorix]

Thanks I will have a closer look at the i3, as it should be equivalent to a Ryzen in some config.

ECC support was discontinued in 10th and 11th generation Core i3. So your choices here are:
Skylake/Kaby Lake i3-61xx/71xx (Xeon E3-v5/6) on C23x chipset (Supermicro X11SSx)
Coffee Lake (Refresh) i3-81xx/9100 (Xeon E-21xx/22xx) on C24x chipset (Supermicro X11SCx)
any Alder Lake/Raptor Lake Core, but only with W680 (and W780?) chipset
The first two may be hard to find and/overpriced now; the last generation boards are quite expensive and DDR5 ECC RAM may be an additional issue. In between there were Xeon W-1100/1200 and E-2300 on W480/W580 and C25x chipsets, and no way to escape the "Xeon tax".

I am not sure about the config to get decent speed. I think I want raidz2 with 6 drives to get the x4 speed (read that somewhere). Just not sure if hdd would be fast enough to get close to 10gb.

A single 6-wide raidz2 vdev will not saturate 10 GbE.
But you're planning for a lot of RAM for a small NAS. How large is the index of your Lightroom library? If it fits in ARC (RAM read cache) there's little need for a dedicated SSD pool (with replication to the HDD pool).

For the node802, I think it’s better for the evolutivity and less constrains than the smaller case. Also thermals would be better if the case is not full.

Thermals are not bad in the Node 304.
The Node 804 is well-ventilated but not quiet when loaded with drives, due to the mesh top.

If you're set on the 804 and micro-ATX, and budget is still a concern, I have a second-hand suggestion.
(FlexATX) X10SDV-TP8F from the ServeTheHome forum. 350 E + shipping from France. MB + CPU + onboard SFP+ in one go.
Add a LSI 2008/2308/3008 HBA (only 4 SATA onboard) and lots of second-hand RDIMM (you can probably have over 128 GB of second-hand/refurbished Registered DIMM for the price of 64 GB of new ECC Unbuffered DIMM, or just get 64 GB for less money).

 

[tupui]

Wooo thanks a ton for all that. I will take some time to unpack and research all you said.

@Etorix
Looking again at Supermicro, I found this easily available X11SCL-F (thanks again for the retailer site, and pushing for that!). Seems quite good paired with a i3-9100F. Would be a bit more expensive than the second hand SOC but I prefer to buy new and have a dedicated CPU I can switch along the way if wanted. (Also might be stupid, but I also want to have fun building this ). FYI on ark I see some i3 10gen which have ECC.

For Lightroom I have small needs that would definitely fit in mem. I also read a lot about all the caches and understood I don’t need it and also should look at some metrics before hand.

@joeschmuck thanks for the long write up! I will reply later, times out for me today. Still thanks for the kind words :)

 

[Etorix]

FYI on ark I see some i3 10gen which have ECC.

Yes, but these are embedded parts. Good luck finding any of them at retail!

Enjoy your new build, and do not hesitate to keep asking.

 

[joeschmuck]

Anytime. I hope some of it is helpful. Supermicro X11SCL-F is a good choice, it has a single M.2 NVMe slot that you could purchase a small SSD as your boot drive, then you have 6 SATA ports open. If you ever need more SATA ports, you could add an HBA at a later date.

As for the CPU question: The i3-9100F does support ECC and looks like a good CPU for you, and I'm certain the price is better than an older Xeon that is more aimed at the Server market. The main difference I saw was the i3-9100F has a maximum RAM capability of 64GB, while the Xeon has 128GB. Is something like that a deal breaker, and the Xeon is twice the cost? I wouldn't think so, not for the use case you have described.

If you haven't already done so, download the Motherboard User Manual, read it twice. Make an informed decision.

Also, not sure if you realize this but you can take your data hard drives out of this machine and install them into any other machine and access the data. That is something very nice about ZFS. Last thing, more of a warning... When you upgrade TrueNAS, If you get a message that says new features are available, upgrade your pool. DO NOT! Why you ask? Because if you have some problem with that version of TrueNAS, you cannot roll back to the previous version if you updated your pool. There is no reason to update your pool feature set unless there is a feature you need. Also you can update at a later date, maybe 1 month after your TrueNAS update if you really want to. This little thing has bitten more people in the butt than I can count.

Let us know what you end up ordering.

 

[Arwen]

...
Last thing, more of a warning... When you upgrade TrueNAS, If you get a message that says new features are available, upgrade your pool. DO NOT! Why you ask? Because if you have some problem with that version of TrueNAS, you cannot roll back to the previous version if you updated your pool. There is no reason to update your pool feature set unless there is a feature you need. Also you can update at a later date, maybe 1 month after your TrueNAS update if you really want to. This little thing has bitten more people in the butt than I can count.
...

Yes, I generally don't bother about upgrading my pool's feature set, unless I want a specific feature.

In fact, if you think you need a specific feature, like ZSTD compression, you can upgrade just that specific feature by it's self. But, still wait after any TrueNAS upgrade as Joe suggests. By selective upgrading you stand a better chance of backward compatibility.

It would be nice if the ZFS pool features were listed as when they were added to a TrueNAS release, in table format. This would help people who want to activate a new pool feature, know if that feature was available in a prior release of TrueNAS than they are currently running.

 

[tupui]

So, I re-read all of that carefully and went on with your suggestions (single boot, low speed RAM 32GB to start with, 4TB drives, intel CPU, supermicro board) Here is what I ended up ordering:

• Eaton Ellipse ECO 1200 750W
• Fractal Design Node 804
• be quiet! Pure Power 11 550W Gold Full modular
• Supermicro MBD-X11SCL-F-B Bulk
• Intel i3-9100F
• 2 x Kingston Server Premier 16GB 2666
• WD Blue SN570 NVMe 250GB (boot, overkill but there was an offer)
• 4 x Seagate IronWolf 4 TB (pool, I will test mirror and raidz2 and if I end up with the mirror, I have a cold spare)

I will play with this first and then assess my real needs with ethernet connection. 2.5GBe might be enough to buy in the end (also a 10GB-TB3/4 adapter to connect to a laptop costs 200 from what I've found, so I will only buy it if it does make sense).

Thanks again for all your help! I am way more confident now with these choices.

I will post when I've received and tested everything.

 

[joeschmuck]

Make sure you do some burn-in testing. MemTest86+ for at least 3 full passes, but I like to run for a few days even if that is 50 passes. Run a CPU Stress Test for several hours minimum to ensure there are no heat related issues with the CPU, motherboard, and power supply (have all the hard drives connected). I like to run it for at least 4 hours, there are others who will run it 24 hours and even longer for critical production environments.
I like to burn in my hard drives using badblocks and letting them run through one full test (5 test patterns) on each hard drive. But before running badblocks I'd run a SMART Long Test, it's faster and if it fails, then you can return the drive right away.