Which CPU for ZFS?

[louie1961]

This is really a ZFS question more than anything. I am toying with the idea of building a new and improved NAS for home use, using something like the n5105 topton board. Leaving aside the issue of whether you like these boards or not, I am wondering am I better off with a slightly slower CPU like the J6413, which can handle more max memory (i.e., up to 32 gb according to Intel) or a slightly faster CPU that has a lower max memory limitation like the N5105 or the N6005 (both of which max out at 16gb according to intel). Amazon seems to have both a J6413 board with six SATA ports and intel 2 Intel 226-V NICs, as well as a N6005 board with six SATA ports and 4 intel 226-V NICs. I am leaning toward the J6413 board, simply because it can support more memory, and I suspect that ZFS would be happier with the extra memory.

My requirements are admittedly fuzzy. I have a NAS today based on a raspberry pi 4 and barely use a half of terabyte of space. But I want to build out my Proxmox server to include plex, radarr, sonarr, etc. so I can see my storage requirements increasing. I also don't necessarily want to leave my Proxmox server on 24x7 (its an old HP Z640) and may look to run plex on the new NAS. But as of right now I don't do any video editing or anything fancy. Most of the stuff on my raspberry Pi NAS is old family photos, home videos, some documents, and stuff for my wife's food blog. I also don't need 100% uptime, but am thinking about using ZFS to do a drive mirror and/or raid z1. Everything is backed up every 4 hours to a second disk on site using Rsync and then a third copy is backed up to the cloud daily using Rclone.

 

[jgreco]

ZFS is happier with more memory, you may be unhappy if it turns out you want to add memory and can't.

Be aware that Intel 226-V is relatively new and may not be supported. I don't recall the details of how that's progressing.

Faster CPU is also advisable; ZFS is your RAID controller and does all the parity and checksums on the CPU.

 

[louie1961]

So I guess the question is can ZFS be happy with just 16 gb of ram on a small NAS if I go for the Pentium Silver N6005, which has half way decent performance at least according to passmark https://www.cpubenchmark.net/cpu.php?cpu=Intel+Pentium+Silver+N6005+@+2.00GHz&id=4565

I mean that's a better benchmark than my old core i5 HP laptop that I currently run Linux Mint on, and I am pretty happy with how that runs.

 

[abufrejoval]

Intel is lying about RAM capacity on their Atoms, has been doing so for ages. I've run all DDR3 Atoms with 16GB when 8 was the official max and my four J5005 Goldmont Plus or Gemini Lake machines run just fine with 32GB of DDR4-2400.

I did try with once with 64GB of DDR4 and that seemed not to work. All of those were ASrock fully passive Mini-ITX boards and the BIOS may play an important role.

I tried again with a Jasper Lake 6005 Intel NUC11AT and that seemed not to work also, but just when I was ready to turn it off, I got distracted for a moment, only to see it boot and use 64GB after all: it just took an extraordinary long time to recognize it first time around, booted at normal speeds afterwards.

I haven't tried again on the N5005 to see if they could also move beyond 32GB, I'm not ruling it out entirely, but 32GB are no issue.

Using different sized modules is working for me, too, Atoms used to be single channel for a long time, went dual channel for a couple of generations and with Gracemont seem to be back to single channel.

Elkhart/Jasper lake is the better choice as the CPU is quite a bit faster than the Goldmont/Goldmont+ variants at the same clock speed, but tends to make it up with a bit more power consumption, too. It's also PCIe 3.0 all around, which is a big step from PCIe 2.0 on the previous generation, when you're thinking of using all ports.

Some Elkhart Lake SKUs even support inline ECC, which could be really, really useful if your BIOS supports that, because it allows you to use ordinary RAM and set aside some of its capacity for ECC at the cost of speed. Never managed to get a hold of one to try...

 

[Etorix]

My requirements are admittedly fuzzy.

This is the fist point to clarify.
Are you looking into this kind of embedded boards because they are small and/or low power or because they are cheap? That's not mutually exclusive, of course…
But if you can find an Atom C3000 (Supermicro A2SDi) or Xeon D-1500 (X10SDV) board at an acceptable price point, these would make for a much better NAS: Minimum 6 (X10SDV) or 8 (A2SDi) SATA ports, server-grade Intel NIC (the i225/i226 has bugs which have not yet been ironed out), can take large amounts of RAM.

 

[abufrejoval]

This is the fist point to clarify.
Are you looking into this kind of embedded boards because they are small and/or low power or because they are cheap? That's not mutually exclusive, of course…
But if you can find an Atom C3000 (Supermicro A2SDi) or Xeon D-1500 (X10SDV) board at an acceptable price point, these would make for a much better NAS: Minimum 6 (X10SDV) or 8 (A2SDi) SATA ports, server-grade Intel NIC (the i225/i226 has bugs which have not yet been ironed out), can take large amounts of RAM.

I'll second that a Xeon D is still a far better choice than a Jasper Lake, if you can get it at a reasonable price say 2nd hand. The original asking prices of both C3000 and Xeon-D were typically beyond a gamer PC equivalent so I typically gave them a pass and went with Atoms, which were around €100 just for the board, more like €400 with chassis, RAM and SSD: quite a bit less than a Xeon-D build.

I got a Xeon D-1542 with 64GB of ECC RAM out of corporate leftovers and while it's actually beaten by a Jasper Lake Atom in scalar performance, it's the much better choice in terms of reliability and expandability. It sips a bit more power, but if you add HDD storage that become less noticeable.

I've used the NVMe slot to connect a 10Gbit Aquantia NIC via an M.2 to PCIe x4 extender (mine didn't have onboard 10GBase-T) and then put a quad M.2 board (€50) into the 16x PCIe slot to turn all the leftover NVMe drives from my shelves into something useful.

Remote management can be really nice if you want that machine out of sight...

Aquantia/Marvell AQC107 support all NBase-T speeds (1/2.5/5/10 Gbit/s) and these days I'd recommend them over anything Intel for compatibility, efficiency and ease of use, with upstream support in Linux since 4.9 or on Windows: never tried with BSD. They are also rather cheap and just as compatible and easy in their Thunderbolt variant. Intel has created quite a mess over the last years and IMHO they deserve to play 2nd or 3rd fiddle, even Realtek is less trouble these days.

Now with an i3-N305 Gracemont 8-core Atom, I'm pretty sure that the Broadwell cores of the Xeon-D may struggle to keep up, but Intel has crippled those with a single DRAM channel and the only systems I've seen so far came with soldered DDR5 RAM or a single DDR4 DIMM slot, which limits RAM expansion significantly.

There are still entry level Ryzens 4000 or 5000 out there with ECC support that will do just fine and a lot faster than of these, but they are still another league than the Atoms you started with.

Inexpensive, reliable and fast is a spot that every vendor seems to avoid for some reason ;-)

 

[louie1961]

This is the fist point to clarify.
Are you looking into this kind of embedded boards because they are small and/or low power or because they are cheap? That's not mutually exclusive, of course…
But if you can find an Atom C3000 (Supermicro A2SDi) or Xeon D-1500 (X10SDV) board at an acceptable price point, these would make for a much better NAS: Minimum 6 (X10SDV) or 8 (A2SDi) SATA ports, server-grade Intel NIC (the i225/i226 has bugs which have not yet been ironed out), can take large amounts of RAM.

A little of both I guess. I just got laid off from AWS so I don't want to spend a ton of money. But mostly I am attracted by the 10 watt TDP. My proxmox server doesn't do too bad, but even at idle it draws 70 watts from the wall.

 

[louie1961]

Inexpensive, reliable and fast is a spot that every vendor seems to avoid for some reason ;-)

I would settle for inexpensive, reliable and just fast enough. I mean I am running a Raspberry Pi 4 currently, so my speed needs are not excessive. But the Pi setup will not let me do any sort of raid.

 

[Etorix]

I'll second that a Xeon D is still a far better choice than a Jasper Lake, if you can get it at a reasonable price say 2nd hand.

I just had a quick look at eBay (US). Two D-1521 boards at $250—with onboard 10 GbE.
And a complete E200-9A system at $467. The case is not suitable for a NAS, but makes for great OPNsense appliances, while the C3558 board inside could be taken out to build a NAS.

Just disgusting…

Inexpensive, reliable and fast is a spot that every vendor seems to avoid for some reason ;-)

As usual: Pick any two.

 

[Patrick M. Hausen]

I just had a quick look at eBay (US). Two D-1521 boards at $250—with onboard 10 GbE.
And a complete E200-9A system at $467. The case is not suitable for a NAS, but makes for great OPNsense appliances, while the C3558 board inside could be taken out to build a NAS.

Just disgusting…

What's disgusting about that? Great hardware at reasonable prices. Add an SC721TQ-250B enclosure and you are all set.

 

[louie1961]

Interesting, when I search for Xeon-D 1521 on ebay this came up for $330. Any reason I couldn't install TrueNAS core on that?

Datto S3B2000 4 Bay 3.5" NAS Intel Xeon D-1521 2.4GHz 32GB RAM 120GB SSD No HDD​

 

[Etorix]

Interesting, when I search for Xeon-D 1521 on ebay this came up for $330. Any reason I couldn't install TrueNAS core on that?

No. This can definitively run TrueNAS.
The case has no cut for a PCIe slot—and no place from it.
It could be that the IPMI is purely Java-based (which sucks).
And it may be that the bundled RAM is not ECC.
That's all the potential drawbacks I can think of. If you're fine with that and need no more than 4 bays, jump on it. For more flexibility, consider a home build around a Supermicro X10SDV (or an AsRockRack D1541D4U-2T8R: micro-ATX with on-board LSI3008).

What's disgusting about that?

The ease of finding such great hardware at reasonable price. On this side of the Atlantic, one pays twice as much, with less choice, and/or patiently stalks for many months, waiting for the perfect opportunity.

 

[abufrejoval]

I would settle for inexpensive, reliable and just fast enough. I mean I am running a Raspberry Pi 4 currently, so my speed needs are not excessive. But the Pi setup will not let me do any sort of raid.

While I also have a PI4 8GB just for curiosity, the reason I got a Jasper Lake N6005 NUC was just to prove that it was the far saner investment than buying a cluster board for PI4 modules: for less money you'd get a virtual cluster that is far more useful and runs rings around a QuadPI with potentially 64GB of RAM, terabytes of NVMe storage and multiple 2.5GBbaseT ports.

It is also quite a capable Windows 2D desktop machine even at 4k. It will play high-res media with ease, but gaming is remote-only. Atoms have come a long way!

While HDD RAIDs wont fit into a normal NUC chassis the M.2 connector will also hold an M.2 card with six SATA ports. That leaves a max of 5Gbit/s for USB3 NICs, but those deliver little more than 2.5Gbit USB NICs so I'd rather recommend those, multiples if you want.

A Thunderbolt NUC is a bit more expensive but will give you 10Gbit Ethernet at €100 extra for a Sabrent NIC. I have Gen 8, 9 and 11 NUCs and an Alder-Lake Mini-ITX with an i7-12700H SoC with 64GB each and Terabytes of NVMe storage in an oVirt Gluster with 10Gbit networking and these are certainly delivering a serious amount of compute for far below 100 Watts of total average power.

 

[danb35]

the Pi setup will not let me do any sort of raid.

I know I've seen a page describing setting up a ZFS-based NAS (which would certainly include the ability to do RAID) on a Raspberry Pi, and I'm pretty sure I've seen it linked in this forum, but I'm having trouble finding it right now.

Edit: Found it:

NASbeery | Alles ohne Windows

Das erste NAS welches in ein eine Bierdose passt!

aow.de

h/t @awasb

 

[abufrejoval]

That's a mirror set, not quite what most expect when the talk is about RAIDs.

But obviously USB can be connected to hubs and powered hubs can keep quite a few platters spinning.

You won't get bandwidth, but gain some IOPS and tons of capacity. Obviously it's possible to run RAIDs even on a PI, even a Petabytes if all you want is to prove a point. But it will be as useful as all these PI clusters.

I've run a 5 disk RAID5 on an Arndale A15 developer board with much less compute power but a real SATA port to connect a SATA port-replicator to. Reconfigured the Android Linux kernel to include RAID support and it just worked.... well within the boundaries of 1GBit networking.

If it's Linux, there is very little you can't do with very little effort: Unix and BSD never felt quite that easy on sysgens.

Not sure it will fit into a beer can with anything 2.5" or 3.5", aber FAXE ist ja schon mal größer: Prost!

 

[danb35]

That's a mirror set, not quite what most expect when the talk is about RAIDs.

A mirror is still RAID, though not parity RAID--it's surely a counterexample to the claim that "the Pi setup will not let me do any sort of raid." But nothing prevents you from nuking the auto-created pool and creating a RAIDZn pool instead, if that's what you want. Though, as you said, the lack of interface options on the Pi really limits the usefulness of this.

 

[louie1961]

I know I've seen a page describing setting up a ZFS-based NAS (which would certainly include the ability to do RAID) on a Raspberry Pi, and I'm pretty sure I've seen it linked in this forum, but I'm having trouble finding it right now.

Edit: Found it:

NASbeery | Alles ohne Windows

Das erste NAS welches in ein eine Bierdose passt!

aow.de

h/t @awasb[/USER
[/QUOTE]

My Pi4 is a 4gb model. Not sure how well ZFS would play with such a meager amount of RAM. OMV won't support software raid (MDADM I guess?) with drives connected via USB the way I have mine connected. I had never considered using ZFS in OMV