TrueNas Core not working properly on Alder Lake CPU (i5-12600k)

[Kris Moore]

The i5-12600 has 6P+0E, so no hybrid architecture and no special requirements for optimal scheduling. That must help.

Yep, purposefully picked that variant since I didn't want the low-power cores. I figure it'll be a few go-arounds in kernel land before that is really optimized and working the way it should.

 

[Kris Moore]

I read many sources that Scale is "not as solid as Core" yet. However, they did not elaborate why. Is there a comprehensive decision guide on trade offs between the current Scale and Core iterations? Is Scale as reliable in terms of stability and data integrity as Core?
Thank you.

From stability / integrity side I'd say SCALE is just as reliable now as CORE. Performance will be mixed, we've just begin digging into optimizations here in prep for Bluefin BETA1. With Angelfish out of box you can expect NFS to perform much better than CORE, and SMB/iSCSI will be mixed depending on your workload. FWIW in my lab with a 10Gb network setup I didn't see any performance differences for my typical SMB use-cases.

 

[CookieMonster]

From stability / integrity side I'd say SCALE is just as reliable now as CORE. Performance will be mixed, we've just begin digging into optimizations here in prep for Bluefin BETA1. With Angelfish out of box you can expect NFS to perform much better than CORE, and SMB/iSCSI will be mixed depending on your workload. FWIW in my lab with a 10Gb network setup I didn't see any performance differences for my typical SMB use-cases.

Thank you for your answer.
Sounds like apart from some potential temporary reduced performance issues due to incomplete optimizations at the moment there is no reason for a new user who is not encumbered with legacy Jails to not use SCALE.

SCALE sounds more interesting to me as a beginner because of abundance of ready docker images and Linux support out there.

May I please ask you a follow-up question:
Some people run TrueNAS on top of Proxmox. Since SCALE has a better support for VMs, what use cases could possibly require the use of a hypervisor underneath TrueNAS? I.e., what are the virtualization limitations of SCALE? I tried researching it, but it seems like all the info I was able to find is pre-stable SCALE release era, so mostly focuses on CORE.

Also, TrueNAS moderator jgreco argues here that Proxmox is immature an unsuitable:

Proxmox W/TrueNAS vs TrueNAS SCALE?

Looking to build a single do-it-all box for a small business. Box would serve as both a file storage NAS and platform of a handful of VMs. I like the idea of being able to do it all from TrueNAS SCALE, having a single UI to handle everything. That said, Proxmox has a better UI for handling VMs...

www.truenas.com

So, to summarize:
1. Why (what use cases) could I possibly need virtualization under/below SCALE?
2. What hypervisor (preferably open-source) would you recommend (with your deep knowledge of your product's compatibility issues)?

I am trying to design my system to match my needs, and I am weary of making noob mistakes in my architectural decisions now that would severely restrict me later down the road when it's too late to change things (painlessly anyways).


Thank you so much

 

[Kris Moore]

We don't recommend running any TrueNAS virtualized in general, ZFS likes direct access to your drives for the best operating environment. If you don't have a VM heavy workload SCALE can run basic VM's via KVM, I'd recommend giving that a shot first.

 

[kongster]

I have just installed TrueNas Core on a new PC. While booting up, at some point TrueNas encounters an error and reboots, causing infinite boot loop. Last entry I saw was - "cpu shutdown..", something along those lines.
Only after I disabled the Efficiency cores in BIOS, only then it booted up without issues. So at the moment my 12600k is running only on 6 cores, 12 threads. Does TrueNas Core support the 12th gen Intel CPUs?


CPU: i5-12600k
Motherboard: Asus PRIME Z690-P D4 (newest bios 0605)
64Gb DDR4 - 3200Mt

Thank you for your post, as a result, I went ahead and install TrueNAS Scale with my 12600k PC. I am brand new to TrueNAS and so for me, it is just try and see mode at the moment. So far, my system said it recognize all 10 cores (16 threads) and has been running for 3 days now. I am excited and thanks again for the sharing.

CPU: i5-12600k
Motherboard: GIGABYTE Z690 UD AX DDR4
64 GB DDR4 - 3200MT from Corsair Vengeance LPX

 

[CookieMonster]

We don't recommend running any TrueNAS virtualized in general, ZFS likes direct access to your drives for the best operating environment. If you don't have a VM heavy workload SCALE can run basic VM's via KVM, I'd recommend giving that a shot first.

How about doing SATA controller passthrough? I've read many people doing that or passing through an HBA (is HBA passthrough better?). Is that also a no-no in your opinion?

Thanks!

 

[Ericloewe]

That is the way to make things work correctly yes.

(is HBA passthrough better?)

Not necessarily strictly better, but simpler and more straightforward. SATA controllers may be in use for the host or just not very clearly visible as PCI devices. An HBA, by necessity, is very clearly identifiable and is, of course, independent.

 

[jgreco]

How about doing SATA controller passthrough?

As long as you realize that you're likely to be passing off multiple SATA ports to the VM, yes, this works fine. Also PCH SCU ports should be fine. Or, at least, either it works or it fails very rapidly, in which case then it isn't suitable.

 

[CookieMonster]

That is the way to make things work correctly yes.

Not necessarily strictly better, but simpler and more straightforward. SATA controllers may be in use for the host or just not very clearly visible as PCI devices. An HBA, by necessity, is very clearly identifiable and is, of course, independent.

What's the best reliable HBA for this purpose?
And where is it best to buy from?

Thanks!

 

[CookieMonster]

As long as you realize that you're likely to be passing off multiple SATA ports to the VM, yes, this works fine. Also PCH SCU ports should be fine. Or, at least, either it works or it fails very rapidly, in which case then it isn't suitable.

Would you kindly elaborate?
I am new to this, and I just tried googling what you said, but still not following.
Thank you!

 

[CookieMonster]

BTW, has OP's original issue with E cores in new Intel CPUs been resolved?

 

[Etorix]

It's not going to be solved until a suitable scheduler has found its way into the underlying OS (FreeBSD/Linux Debian) AND said OS becomes the basis for a new version of TrueNAS (CORE/SCALE). Which will take time.

 

[Ericloewe]

What's the best reliable HBA for this purpose?

Same as always: https://www.truenas.com/community/resources/freenas®-quick-hardware-guide.7/

And where is it best to buy from?

Not China. That's the main criterion, once that's met it's mostly a wash and down to whatever's convenient for you.

 

[CookieMonster]

Same as always: https://www.truenas.com/community/resources/freenas®-quick-hardware-guide.7/

Not China. That's the main criterion, once that's met it's mostly a wash and down to whatever's convenient for you.

"Last update Nov 2, 2019"

:O​

Not China

Isn't everything made in China these days?

 

[Ericloewe]

"Last update Nov 2, 2019"

I mean, if you want to pay 3+ times as much for no useful new features, more power consumption and go through driver/firmware teething issues, you could conceivably spend the time to try and find stock of the LSI 95xx series. If that sounds like a bad deal, the recommendations are fine.

Isn't everything made in China these days?

Generally speaking, that's a dangerous fallacy that is something of a self-fulfilling prophecy.

When talking about server hardware, the meaning is a bit more subtle however: "Do not buy from Chinese vendors". Or, if you prefer: "Leave the importing to the manufacturer's distribution network" - the guys more likely to have put in effort to avoid knockoffs, third-shift production, rejected stock and other such undesirable items reaching their customers.
Not foolproof, but it goes a long way.

 

[jgreco]

"Last update Nov 2, 2019"

Most people here are looking for affordable hardware. This is typically gear that is being cycled out of data centers after a three or five year lease, which can sometimes be found for as little as 20% of the original price. Most "new" gear is not 5x faster or 5x better than this older gear, so, why not buy used?

The Dell PERC H200 is a 13 year old part that was originally hundreds of dollars but can often be found for as little as $30 on the used market. There's nothing wrong with this for servicing HDD's.

Used DDR3 RAM is less than half the price of DDR4 RAM, and you can find 2013's hot CPU the E5-2697v2 which sold for $2500 now on eBay used for $50.

Or you can feel free to buy the latest Supermicro X13 systems with difficult to obtain CPU's and memory, and then discover that there isn't even good FreeBSD or Linux support for P-cores and E-cores. Plus waiting six months for your system to arrive due to supply chain issues.

Isn't everything made in China these days?

No, a lot of it's made in Taiwan, Indonesia, India, etc. The warning about Asian sources that I've posted --

Hardware Buying Guide - some tips for not getting ripped off

I've been giving this information out piecemeal for years, and should maybe have it summarized in one place. There's been a lot of people who are wary of secondary market sources for hardware, such as eBay, due to bad experiences with other...

www.truenas.com

-- specifically refers to non-channel sources and the used parts market. There is a high percentage of knock-off parts sold as "used" (or sometimes amazingly cheap new) especially from China and ESPECIALLY from Shenzhen, where cheap post to the United States and the severe difficulty in returning malfunctioning or ripoff products means you ain't gettin' your money back even if you manage to ship it back. It's fine for a part to be a legitimately manufactured part that happens to be made in China. But you want to be buying it from someone who isn't an unscrupulous buyer of knock-off parts who is then taking your money and laughing at you. Buy your used parts from someplace in the US that is clearly selling used datacenter gear.

 

[CookieMonster]

Most people here are looking for affordable hardware. This is typically gear that is being cycled out of data centers after a three or five year lease, which can sometimes be found for as little as 20% of the original price. Most "new" gear is not 5x faster or 5x better than this older gear, so, why not buy used?

The Dell PERC H200 is a 13 year old part that was originally hundreds of dollars but can often be found for as little as $30 on the used market. There's nothing wrong with this for servicing HDD's.

Used DDR3 RAM is less than half the price of DDR4 RAM, and you can find 2013's hot CPU the E5-2697v2 which sold for $2500 now on eBay used for $50.

Or you can feel free to buy the latest Supermicro X13 systems with difficult to obtain CPU's and memory, and then discover that there isn't even good FreeBSD or Linux support for P-cores and E-cores. Plus waiting six months for your system to arrive due to supply chain issues.



No, a lot of it's made in Taiwan, Indonesia, India, etc. The warning about Asian sources that I've posted --

Hardware Buying Guide - some tips for not getting ripped off

I've been giving this information out piecemeal for years, and should maybe have it summarized in one place. There's been a lot of people who are wary of secondary market sources for hardware, such as eBay, due to bad experiences with other...

www.truenas.com

-- specifically refers to non-channel sources and the used parts market. There is a high percentage of knock-off parts sold as "used" (or sometimes amazingly cheap new) especially from China and ESPECIALLY from Shenzhen, where cheap post to the United States and the severe difficulty in returning malfunctioning or ripoff products means you ain't gettin' your money back even if you manage to ship it back. It's fine for a part to be a legitimately manufactured part that happens to be made in China. But you want to be buying it from someone who isn't an unscrupulous buyer of knock-off parts who is then taking your money and laughing at you. Buy your used parts from someplace in the US that is clearly selling used datacenter gear.

Thank you so much for your patience and information.
I am reading a lot on all this, but I still have massive gaps in my knowledge and understanding. I feel like no matter how much I learn about server building and administration, I still know nothing.

I was under the impression that newer is better because I watched a bunch of youtubers on the topic of building NAS/server, and they said that while old hardware can be a bargain, it's much slower and not power efficient. Power efficiency is starting to matter more due to skyrocketing fuel costs. My electricity provider recently doubled $ per kW.

Specifically, I remember reading/watching that Xeons, especially old ones, have bad power efficiency because not only they are, well, old, but server hardware is not really designed to be idle, so they lack power-saving idle states that newer power station hardware may possess.

I already scored great deals on Fractal Define 7 XL for a case and Asus ROG Strix 750W for a PSU: got the Asus PSU for only $50 -- it's A tier on cultist PSU tier list.

I was hoping to also maximize the savings and buy CPU + Motherboard + RAM to stretch my budget longer.

It won't be just a file server. It will also host multiple VMs, docker containers, media server with transcoding, maybe a build server. So, I need something performant, right?

I was also considering running TrueNAS over hypervisor, so there's performance tax there as well, correct?

Additionally, I entertained the idea of running games in a Windows VM because my server will probably be far more powerful than my laptop, but I am not sure if it's a good idea to mix fun and data preservation mission. In any case, that would require a dGPU.

For now I'd like a decent CPU (for multiple containers and VMs) with integrated graphics for power-efficient transcoding (e.g., with QuickSync, or whatever the AMD equivalent is) + Motherboard + RAM.

I appreciate the guide you linked me to.

If you have an advice outside of that guide, maybe pointer to a Black Friday deals on good TrueNAS-friendly hardware -- I would really appreciate it.

Sorry for a long-winded post, but I felt the need to have someone smart and experienced see my train of thought and maybe point out where I am being stupid.

Thank you!

 

[ChrisRJ]

Thank you so much for your patience and information.
I am reading a lot on all this, but I still have massive gaps in my knowledge and understanding. I feel like no matter how much I learn about server building and administration, I still know nothing.

It actually only gets worse ;-), but that is the nature of knowledge. Already the old Romans had the saying "scio nesciam": I know that I don't know.

I was under the impression that newer is better because I watched a bunch of youtubers on the topic of building NAS/server, and they said that while old hardware can be a bargain, it's much slower and not power efficient.

The majority of YouTube content on this subject is to be taken with a lot of caution. The problem with most of those videos in particular is that they are about building something cheap, small, power-efficient, or whatever. As a secondary aspect those are all fine things. But the primary one must always be long-term reliability and operations. And that is where most videos fail miserably.

Power efficiency is starting to matter more due to skyrocketing fuel costs. My electricity provider recently doubled $ per kW.

I am fully with you on this. For new contracts the current price per kWh here in Germany is around 50 cents.

Specifically, I remember reading/watching that Xeons, especially old ones, have bad power efficiency because not only they are, well, old, but server hardware is not really designed to be idle, so they lack power-saving idle states that newer power station hardware may possess.

My system (see signature) runs at around 90 W idle (all HDDs spinning) and CPU/board are around 10 years old. Things look indeed much worse for older Xeons.

It won't be just a file server. It will also host multiple VMs, docker containers, media server with transcoding, maybe a build server. So, I need something performant, right?

All those examples are still reasonably vague. Don't get me wrong, you are much better with this description than the average. But without a lot more details there is no way of saying what CPU/GPU performance you need for those.

I was also considering running TrueNAS over hypervisor, so there's performance tax there as well, correct?

Yes, but generally negligible. Although it depends on the hypervisor.

Additionally, I entertained the idea of running games in a Windows VM because my server will probably be far more powerful than my laptop, but I am not sure if it's a good idea to mix fun and data preservation mission. In any case, that would require a dGPU.

Mixing those two different use-cases would certainly add some complexity. Also, much less experience exists, simply because relatively few people have gone down this route. Overall, I would indeed caution here.

 

[jgreco]

I was under the impression that newer is better because I watched a bunch of youtubers on the topic of building NAS/server,

Yes, caution, there's a LOT of stupid out there. There is money to be had on YouTube generating fact-free content. Forum members are not compensated for their presence here, so you may wish to place your trust accordingly.

and they said that while old hardware can be a bargain, it's much slower and not power efficient. Power efficiency is starting to matter more due to skyrocketing fuel costs. My electricity provider recently doubled $ per kW.

Yes, old hardware is both a bargain and a power hog. But we're talking 2009-era stuff like Nehalem and Westmere boxes. These are space heaters.

Power management got a lot better with Sandy Bridge, and while efficiency has also creeped up in the 11 years since that was released, it is questionable if it will offset the issue enough to justify buying today's hardware (which also has new problems that will cause you to go rip out your hair).

For example, a typical Geekbench 5 score for a 2011-era E3-1230 (v1) is around 3000. You get 4 cores at 3.2GHz and max TDP is about 80W. A modern Intel E-2388G scores around 7800. You get 8 cores at 3.2GHz and max TDP is 95W. So in 11 years we have at best doubled CPU efficiency but mostly by doubling the core count.

But do you really need 8 cores? For basic NAS service, probably not. The E3-1230 will do just fine at that. The CPU is $15 and the mainboard (X9SCL-F) is $30 on eBay. Add a few sticks of 8GB UDIMM and call it a day at maybe $100 spent. By way of comparison, the E-2388G is $600 and the X12STL-F mainboard is about $300, so by the time the merry go round stops and you've added pricey DDR4 RAM to it, you might be at $1100-$1200.

So now you have to do a value analysis. Can you use the extra cores? Sure, the E-2388G system is going to idle at somewhat lower power, but are you going to make up $1100 in cost differential in electricity savings? It is a complicated bit of calculation. If you have the money and you WANT to build the newer system, that's fine, but you might just be throwing money into a burn pit that will not ROI. Which I heartily approve of if you just want to do it.

Incidentally, one of the differences here, if you're interested in transcoding, is that the E-2388G contains an iGPU that is probably useful for transcoding. The integrated GPU was actually THE reason I acquired this platform. Another factor in the value calculus.

 

[Etorix]

I feel like no matter how much I learn about server building and administration, I still know nothing.

You're understanding something here… Welcome to the club.

Specifically, I remember reading/watching that Xeons, especially old ones, have bad power efficiency because not only they are, well, old, but server hardware is not really designed to be idle, so they lack power-saving idle states that newer power station hardware may possess.

As said, "bad efficiency" is for really old servers. Servers do idle; what they don't do (especially Supermicro boards) is sleep.

It won't be just a file server. It will also host multiple VMs, docker containers, media server with transcoding, maybe a build server. So, I need something performant, right?

A filer would be idle most of the time, only sustaining maximal load during scrubs. With further server duties on top of that, the further load will define how much power is used.
"Performant" need not mean "last generation". The issue is whether previous generation Xeon E-2000 (more cores that Core i3… when these had ECC) and the matching motherboards are still available at good prices compared to present generation.