Upcoming AMD APU Build. Thoughts?

Frikkie

Dabbler
Joined
Mar 10, 2019
Messages
41
As any self-respecting FreeNAS junkie with the need for a vast (approximately 3 :tongue:) number of PCIe slots for HBAs and a NIC, the necessity for a discrete graphics card filling up a valuable slot just for video output (no iGPU) seems like a damn waste.

Would there be anything wrong with going for the upcoming AMD 4700G APU in a FreeNAS build to give me enough room for my 2 LSI SAS 9207-8i cards and a Mellanox ConnectX-3 on a standard B550/X570 ATX motherboard (hopefully with unbuffered ECC memory support) and not some super expensive and unnecessary "enterprise" CPU and mobo combo?
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
- Intel would give you hardware transcoding on FreeBSD/TrueNAS 12.
- Intel would be more energy efficient idle
- Mellanox ConnectX-3 has some issues which require some people to reboot multiple times before it starts correctly during boot. (although I need to test if this is fixed in 12 or not)
- 1 LSI SAS 9207-8i with 1 intel expander board, would still give you enough throughput and 24 potentialsata connectors.

My current setup:
- Intel CPU with igpu
- Fujitsu motherboard (140 quid, most energy efficient out there, made to run 24/7)
- Mellanox ConnectX-3
- Dell H310 HBA
 

Frikkie

Dabbler
Joined
Mar 10, 2019
Messages
41
Hi ornias

Thanks for the response.

I am not in any way looking to use the GPU on my FreeNAS system... already have a separate "server" PC running Plex, Teamspeak and a LAN cache server (virtualised Linux Mint with a lancache docker).
I thought the much better single core performance of a newer generation CPU would be beneficial for SMB and iSCSI. There really is no need for extreme efficiency seeing as electricity is pretty cheap in Germany.

Are you having these ConnectX-3 issues? Never had anything of the sort and I occasionally do cached read tests from my FreeNAS to a RAM disk on my gaming PC by copying over the same large file (ca. 16GB) numerous times and watch it transfer over at 1.01GB/s.

I've never heard about a "storage expander board" before.... seems pretty useful for specific cases.
If this is one of the cards you are talking about, Intel RES3FV288, why would I pay around 300€ when I can get the aforementioned HBA card for 120€ and all my storage needs will be taken care of (in a Fractal Design Define 7 XL)? Or am I missing something? :smile:
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Yes i have the connectX-3 and it's a known issue some people have.
Those where my 4 cents to think about... Nothing more, nothing less.
 

Frikkie

Dabbler
Joined
Mar 10, 2019
Messages
41
Edited to remove hostile comments
Out of curiosity I'd like to hear your reasoning for a storage expander card.
 
Last edited by a moderator:

wl714

Dabbler
Joined
Jul 1, 2020
Messages
11
Lack of PCIe slots is not the main problem, short in available PCI-e lands in APU is. 4700G (Renoir) will have a total of 24 PCIe lands (and that's already quite a bit more compares to previous gen of APUs like 3400G). And you need to subtract 4 lands for onboard sata, 4 or 8 for north bridge, and 4 for each onboard m.2(NVME ones directly attached to PCIe, not hanging on the north bridge). Some board might share the PCIe lands of onboard devices (sata/NVME) with PCIe slots, in that case you get more lands on PCIe slots if you don't use the onboard devices. However, either way, there is not enough lands for your 2 HBAs and NIC to run at 8x each.
If you want to have a few decent performing device controllers and some NVME or PCIe SSDs, the server grade CPU and mobo combo is no longer unnecessary. Latest platform is pricy, but older or 2nd hand can outperform in most cases with less money spend. And the stability and durability/reliability of a system that's designed to run 24/7 for years after years is something you can't expect from a consumer grade system (at least for now and quite some time I believe).
Anyway, if you like to go with consumer grade rig and use the 3 cards listed, I suggest to pick a CPU/mobo with more PCIe lands, not an APU. You only need a graphic card for installation. The graphic card can be removed (and replaced by other card) after FreeNAS is installed, no slot wasted.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
genuinely asked for an explanation to a recommendation
I tried to explain it was just a headsup for some things to checkout for yourself and not a recomendation to anything byond just "go check out if this might suit your needs". Nothing to recommend.

But here's your explanation:
24 is more than 16.
Nothing more nothing less. Just like I stated before: Some people might want some more ports

Lack of PCIe slots is not the main problem, short in available PCI-e lands in APU is.

I think you meant "lanes" and auto correct "fixes" it?

4 or 8 for north bridge
I think you meant south, and that would be 4... which are often also used for one or two pcie or m.2 slots.

However, either way, there is not enough lands for your 2 HBAs and NIC to run at 8x each.
8 sata drives isn't going to saturate 4x pcie 3.0 much... Just because the card supports 8x, doesn't mean it really needs one.
I'm assuming he is using sata though, considering how his build seems to be "as cheap as possible" (which isn't a bad thing)...
a single 10gbe port is slightly more than 1x pcie 2.0 (connectx-3)

If you want to have a few decent performing device controllers and some NVME or PCIe SSDs
He doesn't want to use sata multipliers it seems, so a controller isn't going to push more than the disks attatched, which is 8.
He also doesn't list any NVME or PCIe SSD's, so we can assume none.


the server grade CPU and mobo combo is no longer unnecessary
Having lots of NVME or PCIe devices doesn't seem the goal of this build.
Besides "server grade", has nothing to do with the amount of PCIe lanes.
Server grade doesn't mean "higher end platform", there are a lot of server grade motherboards for lower end platforms available.

And the stability and durability/reliability of a system that's designed to run 24/7 for years after years is something you can't expect from a consumer grade system
Afaik, most devices (regardless of grade) have a peak in failures in the first few months and after a few years.
Those failure rates are often (slightly) lower on server-grade hardware... but the main argument for server grade is support (contracts), good (tested) drivers and large QVL's. All of which aren't really relevant to a cheap-home-nas build.

That being said: There are server-grade boards for lower end systems available that aren't much more expensive than a good consumer motherboard. While I don't think the difference is huge, it can be a risk and I agree buying good hardware might prevent issues along the way. (if it costs just a little bit more, it's worth it, but don't overdo (or overexpect from) it)
TLDR: If you can get one, get one...

Anyway, if you like to go with consumer grade rig and use the 3 cards listed, I suggest to pick a CPU/mobo with more PCIe lands, not an APU.
24 lanes (the consumer and low-end-server default), is enough for what he wants. But it depends on the motherboard.

I can push 8(16) to a HBA, 4 to an NVME drive, 4 to the (mostly unused) southbridge (which sends it to another 4x slot, of which barely 1x is saturated).
I'm limited by the slots in my case, not the lanes. With another slot, I couldrun 2 HBA's, 1 NVME and 1 NIC just fine.

Don't worry about putting a NIC in the southbridge controlled PCIe slot either: a it's about 1.3x per 10gbe port, if you aren't using any io/throughput heavy onboard services, you wouldn't have much issues.

To conclude my story:
The reason I did respond to @wl714 this thorough is because I do want to help/educate people. I simply do not always have the time. So I have to pick which answers I spend most time on. Attacking me for not being assisting enough is bad form, no mater how many personal attacks you add to it.
 
Last edited by a moderator:

Frikkie

Dabbler
Joined
Mar 10, 2019
Messages
41
Awesomely thorough responses... thanks! :)

Up until now I have only had great success running consumer grade hardware (disregarding current Xeon E5645):
i5-2500k, Asus P8P67 Pro → 2 years
E5645, Evga X58 FTW3 → 3 years
All without so much as a single hiccup.

I will not be limited by lanes or bandwidth, especially given the introduction of PCIe 4.0 (2GB/s per single LANE!!!), instead by the number of physical slots available to me and making sure they aren't daft x1 physical slots that are all too common on mainstream mobos.

My plan is to buy 6 2TB SATA SSDs sometime in the future and configure them in a striped mirror configuration for great random IO iSCSI performance and easily expandability.
NVMe SSDs just aren't worth it right now... maybe a single one for an L2ARC drive but there is no need for that.
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
I'm just going to skip over the entire thread and suggest that IPMI is the bee's knees, and you want it.

That'd put you into an ASRock Rack X470D4U or X570D4U, the latter with more M.2 so "yay!", and onboard IPMI-ish graphics so you are free to choose a Ryzen for compute, not iGPU. Plus ASRock take ECC about as seriously as anyone can on Ryzen, current difficulty with IPMI reporting of same notwithstanding. Still works fine with TrueNAS reporting.

So that's what I am recommending whole-heartedly: Whether AMD or Intel, go for something that a) has an IPMI (like seriously you don't know how good this is until you've experienced it) and b) takes ECC seriously. Which in our enthusiast space means ASRock Rack (AMD or Intel) or SuperMicro (Intel). AMD Epyc is yet another discussion, but I'm assuming that's not where you're going, as drool-worthy as those are.
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
NVMe SSDs just aren't worth it right now

Addendum and such ... I was of that mindset until about 6 months ago. When the economics of the whole thing flipped, and NVMe became more affordable than SATA. Alright, fair enough, while I may not need the speed of NVMe, why indeed not get more speed than SATA at a slightly lower price than SATA. Unless SATA is all that's on offer, which is true for my daily driver desktop. To keep Greta from scowling at me (anything but that), I shall endevour to hold on to it until 2022, at which point it's 10 years old and I'm allowed an upgrade. Aren't I? I'll pretend I am.
 

Frikkie

Dabbler
Joined
Mar 10, 2019
Messages
41
That'd put you into an ASRock Rack X470D4U or X570D4U, the latter with more M.2 so "yay!", and onboard IPMI-ish graphics so you are free to choose a Ryzen for compute, not iGPU. Plus ASRock take ECC about as seriously as anyone can on Ryzen, current difficulty with IPMI reporting of same notwithstanding. Still works fine with TrueNAS reporting.
Yes yes yes. This is what I wanted!
Cheers for the recommendation. I funnily enough came across that exact motherboard (was it in Gamer Nexus' Storage PC Build video with Level1Techs?) and sure as hell noticed how moist everyone was getting about IPMI and that "screenshot before a crash" feature. :grin:

Just found one for 200€ used on Ebay.de compared to 300€ on other sites. Might just drop the cash on it immediately and then wait for Ryzen 3300x and memory prices to settle.

Don't even get me started on how drool-worthy an Epyc processor with a PCIe "bifurcation" capable motherboard would be to cram multiple NVMe SSDs on one of those insane PCIe cards.
I will definitely keep on eye on NVMe prices over the coming weeks/months.

Given that I "only" have a single 10g link I wouldn't be able to fully make use of the glorious NVMe speeds over the network other than in synthetic dd file read/write tests... lets just call that "performance jerking". :tongue: But it is nice to dream.

Greta's disapproving looks and harsh words (HOW DARE YOU!) will make any man shake in his boots. But it will not keep the storage nerds away from their upgrades.
 
Last edited:

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
I will definitely keep on eye on NVMe prices over the coming weeks/months.

You're expecting around US 0.12 / GB, though I'm not sure how EU prices compare. The Intel 665p 1TB for example is at 0.11 / GB, the 2TB at 0.13 / GB. Look for the 660p as well, NewEgg had it on sale at one point for USD 83 for the 1TB, which translates to 0.083 / GB.
 
Top