New Attempt at TrueNAS Hardware

[SeaWolfX]

It is ECC "capable" but error handling and reporting to the OS and no IMPI to report to. Even Ryzen plattforms with IPMI have issues to report to it in regards of ECC errors. Thus ECC is not much of use with AMD desktop line.

There was a long thread from someone here somewhere, who did took the time and tested a ton of AMD desktop / prosumer boards for Ryzen and the way ECC error handling was implemented with those. The final result was a pure catastrophe if I recall correctly. Even tested the Ryzen "server boards" from asrock rack with not much better results. That person was even in contact with the manufactors and AMD...

If you want AMD go Epyc. TR is not an option either. Or go Intel Xeon.

According to this article (a couple of year old) it's not all bad though:

https://hardwarecanucks.com/cpu-motherboard/ecc-memory-amds-ryzen-deep-dive/

 

[Etorix]

7th Gen Intel almost as expensive as the latest unobtainium from AMD? That's… unexpected!
Or there's a premium for old hardware, kept in stock for business customers who may want an exact replacement
Another common advice is to look for second-hand hardware. But maybe the Nowegian market is too small, and sourcing from the EU involves extra custom duties?

If you're fine with the four cores of a E3 v6, consider a i3-9100 and a X11SCH-F board. An 8/9th generation i3 should be dirt cheap now—and they do ECC. If that's still not attractive compared to AMD were you are, go for AMD—but rather with an ASRockRack server board than with an Aorus. IPMI and multiple LAN controllers are more appropriate to a NAS than Wi-Fi and audio codecs.

 

[Newfoundland.Republic]

consider a i3-9100

Yes, an Intel Core i3-9100 on Amazon Canada is only CDN$220. That will give you 4 cores up to 4.2 GHz, plus ECC. VT-x, VT-d, EPT will allow you to create VMs (if you are so inclined).

The X11SCH-F on Amazon Canada is only CDN$330. Not bad for CDN$550 in my humble opinion (Hmmm.... If my income tax refund is big enough maybe I'll replace my X9DR3-LN4F+ and dual Xeon E5-2630 V1 CPUs. I don't need all those cores...)

 

[jgreco]

Even the creator of the ZFS file system says there is nothing inherit about it that makes it require ECC more than any other file system.

Well, you have to be careful with that, because it is nuanced.

In one way, that's absolutely correct. Data on a non-ECC ZFS system and a non-ECC FFS system corrupt equally easily.

In another way, it fails to mention that there are no ZFS equivalents to "fsck" to fix a broken filesystem. If you spam bad data into an FFS filesystem, you can fsck and then just cope with the loss of some small fraction of the data (in most cases). However, if you spam bad data into your ZFS pool, especially metadata, you may very well be hosed, and ZFS is known to panic when there is garbage in the pool metadata. Compared to FFS, that is not quite equal in that a damaged ZFS pool often needs to be wiped out and reloaded, but, for the sake of argument, let's still concede that it is similar enough to other filesystems, because it is also possible to damage other filesystems to a point where they cannot be fsck'ed.

In my opinion, the issue comes when you find that you need to move 100TB of data off of a broken 12-drive pool that you can no longer reliably write to due to corruption. That is usually when the lack of an fsck-like tool really sucks, because the same people who do not want to pay a bit extra for ECC are the same people who usually do not have a spare 100TB server laying around to evacuate the data to.

Therefore, I submit that it is unreasonable to make the "nothing inherent" claim without acknowledging this.

@jgreco is that what this community does? I thought ECC was a good thing and here we are being schooled about how stupid we are?

It happens. At the end of the day, all you can do is try to educate. You can lead a horse to water....

 

[Herr_Merlin]

Thanks for the tip! The least expensive one I found here was $50 more than the Gigabyte X570 Aorus Pro though

well you can try to do that.. if you really want to go AMD - but that one failed as well during that ECC tests...

 

[SeaWolfX]

well you can try to do that.. if you really want to go AMD - but that one failed as well during that ECC tests...

I've tried to search some, but can't find the post/article you were referring to. Do you have a link?

I don't want to spend a bunch of money on ECC hardware if it is not really supported.

 

[Etorix]

I don't want to spend a bunch of money on ECC hardware if it is not really supported.

Well, that's exactly what you buys with Intel+Supermicro: The certified certainty that years of enginnering were spent to make sure that ECC works exactly as it should.
At this point I understand that Ryzen CPU can do ECC, vendors can implement ECC on AM4 boards (possibly as in "corrects single bit errors but does not report it"), but no-one will certify that the whole chain actually works. Of course there's an egg-and-chicken issue here: Someone has to buy into the current, imperfect, Ryzen ECC ecosystem to give vendors the incentive to improve.

 

[Herr_Merlin]

Epyc has certified ECC support as well plus vendors producing boards for servers with years of good history and experience

 

[Redcoat]

I've tried to search some, but can't find the post/article you were referring to. Do you have a link?

#8 listed thread in forum search results for "ECC testing" https://www.truenas.com/community/threads/the-usefulness-of-ecc-if-we-cant-assess-its-working.83580/

 

[Etorix]

Epyc has certified ECC support as well plus vendors producing boards for servers with years of good history and experience

Sure. But EPYC is not the same budget as Ryzen, as considered by OP.

 

[Herr_Merlin]

Sure. But EPYC is not the same budget as Ryzen, as considered by OP.

If budget is the concern go for used intel server... Overall cost about the same as AMD desktop..

 

[SeaWolfX]

#8 listed thread in forum search results for "ECC testing" https://www.truenas.com/community/threads/the-usefulness-of-ecc-if-we-cant-assess-its-working.83580/

Thanks! A lot of interesting reading :)

If budget is the concern go for used intel server... Overall cost about the same as AMD desktop..

Well, that's exactly what you buys with Intel+Supermicro: The certified certainty that years of enginnering were spent to make sure that ECC works exactly as it should.
At this point I understand that Ryzen CPU can do ECC, vendors can implement ECC on AM4 boards (possibly as in "corrects single bit errors but does not report it"), but no-one will certify that the whole chain actually works. Of course there's an egg-and-chicken issue here: Someone has to buy into the current, imperfect, Ryzen ECC ecosystem to give vendors the incentive to improve.

Many thanks for everyone's input so far.

I am beginning to understand that although ECC is "supported" it is working at different levels with different degrees if implementation for different vendors and hardware. Do I really need ECC reporting as long as I can have some confidence that ECC working at least some level in AMD desktop boards from e.g. ASRock? I will have a backup solution in place for critical files in case everything should go to s***.

There seems to be a lot of consensus that a server board (Intel/Supermicro) is the way to go. I am wondering though, aside from the verified ECC integration, what are the tradeoffs between an older model server board (like X11SSH) and a newer model desktop motherboard?

The predicament I am faced with is that even the older server motherboards and comparable Intel processors are much higher priced here in Norway than the new AMD desktop alternatives. I could look to used hardware as some has suggested, but the market here is almost non-existent with very few alternatives. Importing something (used or new) from outside the country will set me back just as mutch if not more than buying new stuff here. If I buy something new at a comparable price I am afraid that I will get much less CPU performance and older technology which will make it less future proof and more difficult to change out parts later.

I am in no way against the idea of using server grade hardware. I very much appreciate that server hardware is more robust and more thoroughly tested and have no problems understanding that most people here are swearing to it rather than desktop consumer grade hardware. I am just not convinced, from what I have read thus far, that for my use case it is worth the additional cost and/or drawbacks as I see them.

As mentioned the purpose of the home servers is mainly as a file server (Samba/Nextcloud) and media server (Plex streaming / seedbox) with very few users (<5, and even less concurrent). Although it will be running 24/7 I will in no way see heavy/constant usage. Although nice to have I do not require features like multiple NICs or IPMI.

If there still is an argument (TrueNAS compatibility aside) as to why I should go for a server grade alternative like X11SSH-F / Intel Xeon E3-1230 I would very much like the hear it and the reasoning behind. A goal in building this server is also about me learning :)

 

[Etorix]

The benefits of a server board are IPMI and knowing that the board has (hopefully) been optimised for stability while working 24/7 rather than to crank up a few more fps from overclocked hardware.
Of course, if you do not expect to ever dig in logs to diagnose a possible ECC failure, there remains little case for a fully validated solution. If at all possible or practical, I'd still go for an AsRockRack X470/X570 board over any consumer board.

You do not need the latest and greatest to run a NAS for a handful of users, so, depending on price and availability, you may still save a bit by going for a Ryzen 3000 over 5000 (and maybe not X), or X470 over X570. As for changing parts, that does not occur very often, beyond adding drives, replacing drives or adding more RAM once prices have fallen. That's why many here are happy with old Xeons on long defunct sockets. Even our X11+Intel suggestions in this thread are outdated, but, as pointed elsewhere by @Arwen, socket AM4 is likely about to be replaced. Any future upgrade or expansion would be by way of PCIe cards, and two x8 slots for an additional HBA and a faster network controller are probably enough for any foreseeable upgrade scenario over the next decade.

If buying new AMD hardware is the most practical solution where you are, go for it. It should work and you will not be alone here running TrueNAS on Ryzen.

 

[Newfoundland.Republic]

@SeaWolfX - Maybe this summary can make things clearer: The Supermicro (or other server-grade motherboards) are made for the data centre - 24x7 usage, minimal downtime for 4, 5, 6 or more years. Because server-grade motherboards (or full servers) need to meet these needs, they have built-in compensating controls like ECC memory, redundant power supplies, remote management (e.g., IPMI), etc. Of course, you have to pay for this and that is where buying a "previously loved" server comes into play. But, another caveat, they are made for the data centre and where noise (read: fans) are not all that important. (At my organization you need to wear ear protection in the data centre to comply with OH&S.) While AMD is moving up (seemingly quickly with some rather great hardware - EPYC) Intel Xeon's have been standard for years. They just work.

The other consideration is that TrueNAS (more so now with FreeNAS and TrueNAS converging) is the operating environment for Ixsystems' data centre-grade equipment. That means that the number of permutations of hardware are smaller; that is coupled with the relative base of FreeBSD v. Linux OSes deployed means there are less tested and/or support components.

So, where does that you (me/us)? It depends upon your risk tolerance. For me, while I only use TrueNAS for home use, my photos, documents, etc. are important enough to settle on "previously" loved server gear. I have my rack set up so that *most* of the fan noise is contained (e.g., I can live with it in my home office and replacing the stock Supermicro fans with quieter Nocturas isn't an option because server chassis depend on airflow - CFM - and air pressure - in Hg) and I make sure that I back everything up to different storage locations (e.g., a different NAS at home and an offsite storage pool).

If you want something that doesn't need constant care and attention (outside "normal" stuff like updates, backups, etc.) then chose components that are known to word; e.g., Supermicro motherboards, Xeon (or i3) CPUs, ECC RAM, LSI HBA (in IT mode!), etc. There are a lot of threads and resources on the forum that will help you make your choices. You will get the benefit as well of IPMI (and once you start using it you'll like it - NOTE: Try to get a motherboard that uses HTML and not Java for IMPI remote console).

If you want to experiment then, as we say here - fill your boots. But you may be alone in trying to figure out why component X won't work as you may be the only one trying it.

Good luck!
Mike

 

[ChrisRJ]

I admit that I have no idea how limited the Norwegian market is for used enterprise server gear is. But since you mentioned the X11 series from Supermicro, I wanted to mention that the X10 or even X9 is most likely just as suitable for your needs. I run an X9 series board with a 4-core Xeon (details in siganture, just purchased in October 2020) and it is more than sufficient for my purpose. Plex is not in the game, but the data stored are certainly critical to me.

 

[Newfoundland.Republic]

I think the only issue with the X9 (not sure of the X10) is the use of Java for the remote console (at least under Linux - more related to Chrome/Firefox, though). I use an X9 myself (for now )

 

[ChrisRJ]

The Java-based IPMI is a bit of a bummer indeed. Since I also have physical access, I do not use the IPMI that often. But from Windows it works, although something with HTML 5 would be much nicer.

 

[SeaWolfX]

Doing a some more digging I have found some Intel alternatives that might be worth considering as the combined price not set me back more than about $100 from my original AMD combo. None of the CPUs seem to reach up to the Ryzen 5 5600X performance though, but the E-22456G might be the closest and maybe the best suited for Plex streaming? Not sure which of X11SCH-F / X11SCA-F would be the best alternative. I might have some more flexibility with the SCA-F and I am not sure about the space for 2 x M.2 on the SCH-F.

Motherboard:
X11SCH-F / X11SCA-F

CPU:
Intel Core i7 7700K / Intel Xeon E3-1245 V6 / Intel Xeon E-2246G

 

[Etorix]

SCA is an ATX workstation motherboard (audio, video outputs… and a PCI slot); SCH is a mATX server. Unless you expect to repurpose the motherboard as a desktop later, go with SCH (it does have two M.2 slots, if that's your question).
Both boards take 8th/9th generation Core (i3 does ECC, i5/i7 do not) or Xeon E-21xx/22xx. You can't use a Xeon E3 v6 (7th gen.) in these! No match for the computing power of a 5600X… but the CPU will typically see 0-5%activity, except for the monthly scrub and transcoding duties. ZFS loves RAM; CPU power, not so much. (Hence, the many posts about old Xeons, X9, X10 boards—the X10SDV family and their ≤45 W Xeon D-15xx make lovely home NAS—, but if you can't source second-hand servers in Norway, that's pointless.)

Please double check whether, and how, an iGPU may be used for transcoding—I don't know Plex.

 

[SeaWolfX]

After more consideration and a lot of valuable input from this forum I have landed on the following which I believe will result in a solid server to be run for many years:

• Intel Xeon E-2246G
• Supermicro X11SCH-F
• Micron DDR4 2666MHz ECC 32GBG
• Seagate Exos 7E8 8TB (x4)
• Samsung 970 Pro Series 512GB (x2)

All this will be put in a full-tower cabinet and used with a power supply I already have from before. I could use some input on cooling and airflow (+ monitoring?). Any tips or recommendations? The CPU does not come with an included cooler. Is there anything in particular I should look for or consider?