Acco: Working on a new build

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
After running a year with my first ever home-NAS server hybrid (nicknamed "Dozer"), I finally managed to bring it too its knees a little too often for my taste.
CPU is huring with big transfers (due to zstd) and the 2*3 mirrored disks are more than 80% full.

Being active within the ZSTD community I already know what was comming, so have been preparing for quite some months by now:
I knew I needed a faster CPU for zstd and wanted more disk-slots to use DRaid once it launches. It's a SCALE focused build, with enough processing power to decently run some containers/VM's as well as provide some storage
And above all, it needed to be affortable.

Meet Dozers big brother: Acco. Named after the Acco Super Bulldozer https://en.wikipedia.org/wiki/Acco_super_bulldozer
Save a 1 or 2 parts, everything is finally ordered to start the build.

Case: Inter-Tech IPC 4U-4424 (24 bay, dumb backplane)
Caseaddons: 3x pci-slot 2.5" sata bays
Motherboard: Fujitsu D3643-H
PSU: be quiet! Pure Power 11 400W
PSU: Corsair AX 860i (860W)
CPU: Intel I7 9700 (still waiting for a good deal to show up though)
CPU Cooler: Cryorig H7
RAM: 2x16gb DDR4 (non ECC)
HBA: Dell H310 (flashed to IT mode)
SAS expander: intel res2sv240
NIC: Connectx-3 dual port
OS SSD's: 2x Kingston SSD Now 60gb
Metadata SSD's: 3x Kingston SSD Now 60gb (placeholder)

Future additions (not included in current build or price estimate):
GPU: Nvidia 1660 (Future upgrade for hardware transcoding)

Some QA:
Q: Why go for a port multiplier card?
A: The case was an insanely good deal and a dumb backplane is easier to repair in the future. The expander card can just be replaced without much downtime

Q: Why no ECC
A: Personal preference, sorry I'm not a ecc "believer" for some linux-iso's and quadrouple backuped personal documents.

Q: Is the Connectx-3 stable?
A: It had some issues on 11.3 where it didn't get detected on reboot sometimes. under 12 thats solved and it's rock solid

Q: That motherboard doesn't have enough PCI-E slots!
A: It does, the NIC is going into a 1x slot, i'm happy with that throughput.

Q: But those X1 slots don't fit a connectx-3 card, they are closed
A: There is such a thing called a riser cable and/or a thing called a "file". ;)

Q: That PSU is not enough for the peak power consumption
A: I expect the max 22 harddrives to at most consume 480W at spinup (thats a worst case), which is a little above the recommended load but not too bad. PSU is sized on average expected peak load, not absolute maximum expected load.

Q: Why that CPU?
A; It's not too expensive, I already had the motherboard and it doesn't suffer much from the Intel SMT bugs.

Q: That GPU is never going to fit
A: There exist single slot 1660(ti) cards ;-)

Q: What disk setup are you planning to run?
A: At first I will use a placeholder setup (raidz2 or something), goal is to run DRAID with 3 groups of 7 disks with double parity and 1 hot-spare and a tripple mirror metadatavdev

Q: Those metadata ssd's suck
A: I know, those are going to be replaced someday. Those where just a VERY good deal.

Q: No L2ARC or SLOG?
A: I want to max ram before adding L2ARC and don't really have much need for SLOG at the moment. It's a long-term todo.

Q: You do know you are pushing the limits of your hardware?
A: Ain't that the fun of it? ;)

Q: And what did this thing cost you?
A: 1 kidney and 2 livers.
Okey just kidding, close to 1300 euro's including all cables and shipping expanses.
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Q: That PSU is not enough for the peak power consumption
A: I expect the max 22 harddrives to at most consume 480W at spinup (thats a worst case), which is a little above the recommended load but not too bad. PSU is sized on average expected peak load, not absolute maximum expected load.

That will kill your PSU. Supermicro's *SMALL* spec for a 24-bay chassis is the 920 watt dual module supply, which is not sufficient to guarantee brownout-free spinup when one of the PSU modules is removed.

We have a nice thread that explains all of this.

https://www.truenas.com/community/threads/proper-power-supply-sizing-guidance.38811/
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
That will kill your PSU.
It's an expansion project (i'm slowly going to add vdevs), In case I hit a limit i'll throw in something heavier ofc.

I'm not actually looking into discussing my current build or your numbers, but lets say I don't agree with about 50% of what you wrote.
(nor do my readings or specsheets)

For the record:
I'm also not fully completely sure what I'm going to do in terms of harddrives.
I expect someday to run draid, but thats way off still... I'm also looking into the power efficiency of certain options. I might even go 2.5" drives.

The drives are mostly TBD.

Case/build is mostly to prevent myself from annoying my wife for yet another case/build because I ran out of space/power yet again :')
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
It's an expansion project (i'm slowly going to add vdevs), In case I hit a limit i'll throw in something heavier ofc.

I'm not actually looking into discussing my current build or your numbers, but lets say I don't agree with about 50% of what you wrote.
(nor do my readings or specsheets)

That's fine, of course. There are a lot of hobbyists who make that mistake, or don't understand what the issues are. I have a background in electrical engineering and electronics, do this stuff (including build servers) professionally, I show my reasoning and my homework, etc.

PSU failures are not a "limit" type thing. If you hit a point where the PSU actually fails, you are already so far over the line that irreparable damage had probably been done at a substantially lower load. HP, for example, built their MicroServer Gen 8 with a 200W PSU for a 4-bay AMD Athlon based system. The supply is incredibly underpowered and there's actually a company that has made a good business out of 350W PSU upgrades for it when the undersized supply eventually fails. That's for a four-drive system.

Now, really, I've heard all the stuff about how I'm wrong or how I'm an idiot or how I don't know what I'm talking about, and I can serenely listen to that all day long. If you look at 24-or-more drive arrays, the *lowest* thing I have seen is an 850W nonredundant PSU on the Storinator Q30, which is only possible because they stagger spinup (this is also not their default option, which is a dual 1400W PSU). Your typical Supermicro 846 is a pair of 920W PSU's (1840W available to spin) or 1280W PSU's, HP is IIRC redundant 1460W, etc. I have a hard time thinking all these other electrical engineers are crazy too.

Now, to be sure, the numbers I derive in the PPSG are conservative and I explain why. I figure most of the crowd here can't afford to burn up a NAS that might cost $5000 or more, so spending money on a PSU that is able to safely drive their load is good insurance.

I obviously can't force you to follow my advice, nor would I care to, but I do wish everybody good luck with NAS builds and feel a responsibility to point out things like this where the "conventional PC wisdom" is incorrect.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Now, really, I've heard all the stuff about how I'm wrong or how I'm an idiot or how I don't know what I'm talking about, and I can serenely listen to that all day long.

Yeah, thats why I skipped insulting your intelligence and didn't waste both our time in endless discussions ;-)

That being said: I'm currently opting slightly more towards a more power-efficiency focussed build (with 2.5" drives), but I think 400W at least gives me some "breathing room to play with" byond just 2.5" disks (for example i also have about 6 3.5" I rather don't waste) and don't want to exceed 22 drives of spinning rust or 480W peak power consumption (whatever comes first)
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Addition:
Something I did overlook on the PSU side of things: It was a dual-rail PSU.
250W on the Disk rail is not going to work in any way, shape or form...

So I switched it for an Corsair AX 860i which can handle 860W continuesly on a single rail AND the design is verified by external sources to be able to handle almost 1KW peak loads. Guess that also statisfied @jgreco somewhat.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Guess that also statisfied @jgreco somewhat.

Only to the extent that I feel it's less likely for you to have a total disaster. I err very much on the other side of things; I do not rely on overcurrent peak availability, and in fact deliberately derate because I understand how components deteriorate over time, and expect that an 860W PSU might someday not be able to pump out even 800W. I used to work for a medical electronics manufacturer and there's a lot of healthy paranoia that can be applied to making things work reliably, especially over time.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
I err very much on the other side of things; I do not rely on overcurrent peak availability.

Imho I already do with this setup:
I account for:
- 30W per HDD
- 10W per SSD (sata based)

Which leaves me at about 150W for the system itself, which isn't stressed (much) during disk spinup.
The GPU isn't stressed in any way-shape-or-form till the disks are spinned-up.

Even if stressed, the 9700 has a peak power consumption potential of about 125W.
(although I think those tests where on a board with multicore-enhancements enabled)

How conservative these numbers are depends a lot on the disks and while I agree I wouldn't account for disk difference for other users, I do for my own builds.


expect that an 860W PSU might someday not be able to pump out even 800W
I do expect that as well. It just means I need to trash the PSU someday and replace it, which goes for about every part in this build, as most parts aren't the best grade available. It's not a professional storage server, it's a home nas.

I don't expect much issue for a year or two, as this is one of the best reviewed PSU's in this watt-range by JohnnyGuru:
 

jayecin

Explorer
Joined
Oct 12, 2020
Messages
79
I would consider a newer CPU just for longevity. I was looking at the same CPU as you were for my build last month or so. The lack of hyper threading and extra heat over the i7 10700 along with the fact that 10700 boards will support Rocket Lake S, at least gives you some room for upgrade in the future. The hyperthreading becomes way more useful when you are running VMs as well, the performance increase from a 9700 to a 10700 is nearly 30%. https://www.cpubenchmark.net/compare/Intel-i7-10700-vs-Intel-i7-9700/3747vs3477 . So 30% more performance, less heat, twice as many threads and the possibility of upgrading to Rocket Lake S by just dropping in a new CPU.

I would also consider more RAM or at least be aware that depending on the number of VMs/jails, 32GB wont be enough. I already wish I went with 64GB, I have 3 VMs and 3 Jails running.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
I would consider a newer CPU just for longevity.
I'm not going to throw away 110 euro's of motherboard. Not everything was ordered new, some parts where already in stock or bought months ago. ;)

The lack of hyper threading
Hyperthreading on ZFS has barely any significant performance enhancement and I prefered not to have it with all the specre/meltdown shizzle going around. I'm quite happy not having hyperthreading.

extra heat over the i7 10700
Both my serverroom and my cooler don't have any issue with handles a few extra watts of heat.
That being said, the 10700 has the same or higher heat output:

I also wasn't very impressed with the gain in speed either (not enough to warrant buying another motherboard)

10700 boards will support Rocket Lake S
I plan builds months to a year in advance, when I get ready to build a Rocket Lake build, you would say the same about the generation afterwards ;-)

at least gives you some room for upgrade in the future
The CPU is picked because it's already 30-40% more power than I actually need at the moment. It's not supposed to be upgraded, basically it will be replaced when the board needs replacing or visa-versa.

The hyperthreading becomes way more useful when you are running VMs as well
I don't plan to run more than 1 or 2 VMs at most, mostly just containers. But yes, containers might or might not have a gain.

So 30% more performance, less heat, twice as many threads and the possibility of upgrading to Rocket Lake S by just dropping in a new CPU.
No, thats:
- 30% more performance under ideal circumstances
- with the same heat production,
- twice as many threads (with is a repeat of argument 1)
- 100% increased pricetag (So, 200%)

I already have the motherboard and payed 235 euro's (incl. shipping) for the 9700. Thats about the same price in total as just the 10700
I've looked into the 10700 and was not impressed at all.

I would also consider more RAM
I know what I need...
Why? Because I already run everything... I just want things to go a little smoother, which a 9700 is perfectly capable of.

at least be aware that depending on the number of VMs/jails, 32GB wont be enough. I already wish I went with 64GB, I have 3 VMs and 3 Jails running.
Did you really read my post? As it's clearly labeled as a scale build it doesn't run jails and I also stated I already had the motherboard.
 
Last edited:
Top