Boot, SLOG and special vdev on the same mirror?

[Davvo]

Perfect for when I will need to renew my system in 5-8 years.

 

[NugentS]

I have one, but driving 1h for both ways makes no economical nor environmental sense to me.

Hell - the numbers you have quoted already make economical sense. In particular as you would also need other stuff as well.

 

[Jamberry]

Hell - the numbers you have quoted already make economical sense. In particular as you would also need other stuff as well.

I mean, it depends Sure, small niche stuff like can be more expensive. Things like SATA DOM. But on the other hand, if a product is available in my small country, we have a extremely competitive IT market. Most of the time we get stuff even cheaper than the rest of the EU.

For example, a X12SAE-O will cost me 380Fr. from a not so cheap but premium service seller brack.ch.
The mentioned jacob.de shop wants 440€ which is around 440Fr.

Some is mostly true for other basic stuff like nvme drives. Like you can get a Kingston SNV2S 1TB nvme drive for 62Fr.

 

[Etorix]

But on the other hand, if a product is available in my small country, we have a extremely competitive IT market.

Indeed. I had a short look at the "server" section on ricardo.ch and it was nothing short of amazing… (Where's the drooling smiley?)
With regular visits and some patience, there should be great opportunities to assemble a nice NAS from there.

X12SDV boards will be very expensive, but second-hand X10SDV, X11SDV or M11SDV are great options. These embedded boards have very long product cycles, so I would still not be worried about Broadwell-era X10SDV.

 

[Jamberry]

X12SDV boards will be very expensive

I guess, but on the other hand, they already have 25Gbit, 10 Sata and 3 nvme included. So for me, this would work out of the box, without any add-on cards and I guess with a fairly low power consumption.

Another thing is, that my provider init7.net offers me to upgrade my 1Gbit fiber to 25Gbit for a one time fee of 333Fr. or to 10Gbit for a one time fee of 111Fr.
The yearly fee for the service is always 777Fr. regardless what speed I use.
I will have to do research, if a X12SDV SCALE system with an OPNsense VM can handle something like that.

 

[NugentS]

Bloody hell - that init7.net looks really good

 

[Jamberry]

Bloody hell - that init7.net looks really good

It is. Activate the nerd mode in the bottom left. Peering is excellent, dual stack, no CG-NAT, /48 fixed prefix, hotline workers know what IPv6 is and escalate problems they can’t solve quickly to the net eng team. CEO of the company is for open peering and net neutrality and dragged the ex state owned giant Swisscom to court three times. Won two times and the third one is pending. Yes, I am a fanboy :)

 

[Etorix]

Another thing is, that my provider init7.net offers me to upgrade my 1Gbit fiber to 25Gbit for a one time fee of 333Fr. or to 10Gbit for a one time fee of 111Fr.
The yearly fee for the service is always 777Fr. regardless what speed I use.

Love their pricing… What a change from ##9.99 offers!

I will have to do research, if a X12SDV SCALE system with an OPNsense VM can handle something like that.

Having your router/firewall in the NAS means that everything goes down if the NAS goes down. This also implies that the NAS is physically plugged into WAN. A dedicated OPNsense device and the NAS solely on LAN behind it should be more secure and easier to maintain.
As to what is required to handle 25 GbE traffic I have no idea, but I hope that, first, a 2000+ E/SFr. X12SDV is up to it and, second, the minimal requirements are less expensive than that.

 

[Jamberry]

Having your router/firewall in the NAS means that everything goes down if the NAS goes down.

I thought about that. I think I would combat this problem by using a fixed IP for the IPMI and a dedicated managment Port on the Switch with the same VLAN. But yeah, I normally also don't like putting everything into one system. I currently run TrueNAS, OPNsense and Proxmox all in different systems. But I sounds so tempting to put all into one. Mostly because it sounds like way less power consumption but also the networking part would become very easy. I mean I could use the SFP25 for WAN. The VMs on TrueNAS would also get 25Gbit if the (Linux bridge can handle that). If I continue to use separate systems, I would need network cards for all these systems and Switch that is capable doing more than 1Gbit....

a 2000+ E/SFr. X12SDV

Wow, you think it will get that expensive? I saw some x11 for 800Fr. up to 1600Fr.
Because I don't need to much CPU performance, I thought I will find one for 1200Fr. that suits my needs. Maybe that was naive :)
Because otherwise, I will be better off by buying a cheap system, a NIC card and a hba controller.

 

[Davvo]

I currently run TrueNAS, OPNsense and Proxmox all in different systems. But I sounds so tempting to put all into one.

Until something goes wrong and you have to troubleshoot the heck happened.

 

[Constantin]

...Xeon D are expensive compared to standard boards with Ryzen or similar, but again: ECC memory, lots of ports for storage, rather quiet from the get go (and you can still exchange the case fan for a Noctua), IPMI (I have four servers at home and no monitor or keyboard for them), etc.

Allow me to quibble a bit. For file-server SOHO use case, the X10SDV-2C-7TP4F ticks all of those boxes at the same price point as Atom boards yet offers a higher clock speed, more SATA ports, a real m.2 x4 PCIe for a SLOG, two built-in SATADOMs, and two bifurcatable PCIe 3.0x8 slots. All for about $500 new. Up to 128GB RDIMM, a mSATA slot for a L2ARC, 10GbE SFP+ built in, IPMI etc. etc. etc. I consider this board simply brilliant. I bought the bigger brother with more cores (and the D-1537 vs. the D-1508) and I regret that decision for my use case.

 

[Patrick M. Hausen]

@Constantin Very nice. Not to mention the X10SDV-4C-7TP4F

10G with SFP+ ... very nice indeed.

 

[Jamberry]

Until something goes wrong and you have to troubleshoot the heck happened.

Sigh, yeah I guess you are right.

If I don't put all in one, I am fine with only 4 cores and only 128GB RAM for TrueNAS. X10SDV-4C-7TP4F looks like a neat little board for TrueNAS.

 

[Constantin]

I was on that path also, thinking that I’d run zone minder, etc in jails on the server. In the end, it didn’t make sense.

Blue Iris on a dedicated NUC is a far better package than allocating cores, few remaining hairs, etc to trying to make it run in a VM on the server. (Never mind missed alerts, etc as the cores max out).

Individual packages also make sense from the troubleshooting POV, ie you only have to troubleshoot one thing at a time, vs. EVERYTHING going down when the file server goes kaploink in the night or post-update when IXsystems decides to change a jail/VM system config and doesn’t warn folk in advance.

Thus, my total aggregate rig consumes a few more watts running the ca server on one pi, two redundant DNS piholes on two more pi’s, etc. but everything is separate.

 

[Etorix]

But I sounds so tempting to put all into one.

Depending what Promox hosts, replacing it by VMs on a more powerful TrueNAS (or containers on SCALE) may be tempting. Keeping OPNsense apart, on bare metal, is probably better.
For SFP+ switches, Mikrotik should have you covered at a friendly price. I haven't looked into SFP28 yet.

Wow, you think it will get that expensive? I saw some x11 for 800Fr. up to 1600Fr.
Because I don't need to much CPU performance, I thought I will find one for 1200Fr. that suits my needs. Maybe that was naive :)

I may be wrong but 2000+ is my (totally uninformed!) guesstimate for the X12SDV range, new at retail. And maybe closer to 3000 for the 25 GbE parts. That's why I concur with @Davvo that X12SDV will be terrific boards around year 2030, when they'll eventually pop up in the second hand market.
Starting at 800 SFr, an X11SDV could make a nice (and certainly not-too-old) TrueNAS-with-VM system. Some of these boards may even make an extra NIC and/or HBA unnecessary.

Edit: typos

 

[Constantin]

To truly take advantage of a 25GbE system, you’re likely looking at all flash, multiple VDEVs, etc. at which point the cost of the motherboard becomes inconsequential. If I ever win a billion dollar lottery, I may consider such a system for lulz but in the meantime, a one-VDEV, z3 Is all I’m willing to spring for.

10GbE has minimal latency, a comparatively low cost entry point and unless your NAS can actually deliver or write content at 10Gbit/s is unlikely to be a throttle. For spinning HDDs, that’d likely be somewhere north of 4 VDEVs in a pool. I don’t have plans for 24+ drives spinning away when 8 will do.

 

[Etorix]

I also looked into the X10SDV-TLN4F. I could get that one for 950CHF. The thing that worries me is power consumption and age. This a 8 year old CPU. I know that age is not everything, and I am happy to buy a 2y old CPU, but 8y seems very old for me. According to servethehome.com the motherboard consumes 90w under load, but I was unable to find idle consumption numbers.

I found some data on STH. From the graph at the end:
Atom C3000 : 20 W idle / 40-55 W load (and that may include 10 GbE from the SoC)
D-1500: 20-30 W idle / 60-90 W load (D-1518 / D-1587)
D-2100: 55-60 W idle / 100-160 W load (D-2123 / D-2183)

The lowest Xeon D-2100 idles at the level of a top-of-the-line C3955 at full load.
Extrapolating the graphs, a X10SDV-TLN4F (D-1541) should idle around 25 W.

 

[Jamberry]

Because my system will mostly idle, the idle numbers are the most interesting to me.
Interesting, that the D-2100 use that much more power than the D-1500 at idle.

I am currently also looking into getting a cheap IPMI and ECC mainboard with two x8 PCI lanes, use a cheap i3 or pentium and combine that with a used 250$ 9400-16i and a 60$ mellanox connectx-3. Both use only around 9W. It is not nothing, but the 8 drives are probably also gonna need around 8W each.

My current TrueNAS is a X11SSL-CF I bought 3 years ago for 300$. It has 14 SATA ports, but unfortunately it is not available

 

[NugentS]

The 9400-16i is a trimode card. Are these an accepted & tested card for Core and or Scale?

 

[Etorix]

Probably not tested and accepted to the same standards as the previous 9200 and 9300 generations. But the $250 price suggests that @Jamberry found a bargain on one.