just a heads up, finally ASUS w680 available.

[jgreco]

I'm not sure which Linux kernel version SCALE is using as I don't use it yet, but Intel's hybrid architecture (performance + efficiency cores) is mostly supported in the scheduler in Linux 5.18, and fully supported in Linux 6.0. Some info about the patch that landed in 5.18 is here: https://www.phoronix.com/news/Intel-HFI-Linux-v2-2021.

That doesn't translate into it being properly supported in SCALE. A NAS is highly sensitive to the performance of the cores. You have the kernel (including network), ZFS filesystem, and userland all competing for high speed access and if one of your tasks lands on an efficiency core, that's not good. I'm not sure what exactly is going to be done to mitigate this, but it would be naive to assume that just because Linux integrated some "first generation" support for this, that it was going to work well. SMP UNIX systems have typically operated with matched core speeds on all cores, and signs are that this is not going to be particularly clean to integrate into Linux, much less SCALE. Making it work at all is merely a first step.

 

[Daniel15]

I'm not sure what exactly is going to be done to mitigate this

In general, the approach would be to ensure that anything performance-sensitive is pinned to particular cores (CPU affinity), and no other heavy tasks are scheduled on those cores. That also greatly reduces the amount of context switching and improves CPU cache utilization.

just because Linux integrated some "first generation" support for this

ARM has had a "big.LITTLE" design with heterogeneous multi-processing (all cores can be used at the same time) for 10+ years, so Linux already had some of the foundational work done ages ago.

 

[Etorix]

I'm not sure which Linux kernel version SCALE is using

Last time I checked it was 5.15. And TrueNAS, being an appliance OS, is NOT updated to the latest version in a whim.

 

[jgreco]

In general, the approach would be to ensure that anything performance-sensitive is pinned to particular cores (CPU affinity), and no other heavy tasks are scheduled on those cores. That also greatly reduces the amount of context switching and improves CPU cache utilization.

That doesn't work inside the kernel, where lots of important stuff happens. Network stack processing in particular.

ARM has had a "big.LITTLE" design with heterogeneous multi-processing (all cores can be used at the same time) for 10+ years, so Linux already had some of the foundational work done ages ago.

And hasn't really progressed beyond in those ten years. Plus, Linux memory management sucks, which leads us to a situation where you have an OS where its proponents like to point out that it theoretically CAN do this and that, but it doesn't ACTUALLY do it on production systems.

 

[Ericloewe]

I'd also add that there's a big difference between a phone-like device and a server, in terms of workloads and thus how any sort of asymmetric SMP handling would work.

 

[Whattteva]

I'm not sure which Linux kernel version SCALE is using as I don't use it yet, but Intel's hybrid architecture (performance + efficiency cores) is mostly supported in the scheduler in Linux 5.18, and fully supported in Linux 6.0. Some info about the patch that landed in 5.18 is here: https://www.phoronix.com/news/Intel-HFI-Linux-v2-2021.

Not even Ubuntu LTS uses 5.18 (it uses 5.15). I highly doubt SCALE would be that advanced either considering it's also targeted to Enterprise users.

 

[Daniel15]

I'd also add that there's a big difference between a phone-like device and a server, in terms of workloads and thus how any sort of asymmetric SMP handling would work.

ARM isn't just for mobile devices... There's been ARM servers for many years now too. Some Linux distros like Debian have pretty comprehensive ARM support.

Not even Ubuntu LTS uses 5.18 (it uses 5.15). I highly doubt SCALE would be that advanced either considering it's also targeted to Enterprise users.

That makes sense. I'm used to Debian using older stable versions too, but the latest stable Debian version has Linux 6.1.38 since it just came out in June 2023.

Anyways... back on topic... Has anyone built a TrueNAS server with a W680 board? I just ordered a Asus Pro WS W680M-ACE SE (the MicroATX version) and 2 x 32GB Kingston DDR5 ECC (KSM48E40BD8KM-32HM) for a server build.

 

[Whattteva]

That makes sense. I'm used to Debian using older stable versions too, but the latest stable Debian version has Linux 6.1.38 since it just came out in June 2023.

Ah yes, Debian Bookworm just came out indeed.

Anyways... back on topic... Has anyone built a TrueNAS server with a W680 board? I just ordered a Asus Pro WS W680M-ACE SE (the MicroATX version) and 2 x 32GB Kingston DDR5 ECC (KSM48E40BD8KM-32HM) for a server build.

I usually shy away from the non-traditional server boards and it's mostly all attributable to RAM availability. These "prosumer" boards typically require unbuffered ECC DIMM's which tend to be kinda' expensive and more limited in size. I typically prioritize RAM for all my servers and the cheapest way to do that is by using RDIMM or LRDIMM's which are plentiful and cheap on the second-hand market.

 

[Daniel15]

These "prosumer" boards typically require unbuffered ECC DIMM's which tend to be kinda' expensive and more limited in size.

The Kingston 32GB DDR5 ECC DIMMs I bought were $100 each with an employee discount at a supplier my workplace uses (Zones). I think they're usually $120 or so. Asus only have a few RAM models in their compatibility list and these were one of them.

A bit expensive but I wanted to buy new hardware since it should hopefully have more longevity and better power efficiency than older hardware.

 

[Ericloewe]

There's been ARM servers for many years now too.

All of which have used symmetric multiprocessing with homogeneous cores.

 

[a.dresner]

I'm coming back to this thread with an update.

I am falling back to plan b. Running Proxmox with virtualized TrueNas on this mobo wasn't working well. While I get it all working, updating Proxmox to the latest version showed me how delicate my system was and with just a single system, huge risk. It was like the cat in the hat juggling the cake in the end.

The Asus W680 is going to become my new workstation at work. I picked up another X12STH-LN4F and am building a clone of my first system. I also bailed on UNRAID. So now my 2 X12STH-LN4F based systems will run TrueNas Scale. And I picked up 2x HP 800 mini and put Proxmox on them to run containers like Plex, Homebridge, etc at my 2 locations. X12STH-LN4F based TrueNas server, HP 800 for proxmox containers, VMs, and an old 1 liter PC for Proxmox backup and lastly 2 old QNAP that will hopefully become some kind of backups for the TrueNas systems.

Trying to shoehorn my entire setup onto one machine was tenuous at best with every upgrade...might breaking this or that. Now, I can upgrade at will and it's a better setup overall.

Also, TrueNas was throwing disk errors that have now gone away since moving my virtual TNS to physical.

Can't wait, gonna have a screaming workstation in a few weeks when this migration is fully complete.

 

[Ericloewe]

X12STH-LN4F

Fancy. I remember when a board that would take high-end Xeons would cost as much as these "entry-level" server boards do now,

 

[Daniel15]

All of which have used symmetric multiprocessing with homogeneous cores.

Ah, I see! Thanks for the clarification :)

Running Proxmox with virtualized TrueNas on this mobo wasn't working well.

What issues did you encounter?

 

[a.dresner]

Ah, I see! Thanks for the clarification :)


What issues did you encounter?

I followed the tips within the forum on virtualizing truenas, but disk errors were common. The pool wasn't healthy. I had a hard time believing something was wrong with a new disk. I cannot say with 100% certainty however since converting to physical, the errors are gone. If they come back, I know the disk that was throwing them and I have a replacement waiting.

It's a lot to share here, it's hard to summarize. I always wondered about the cores as Eridloewe mentioned and also realized I wasn't going to make a virtualized windows machine as I originally planned. That CPU will be best served as a desktop, not at the heart of a server.

 

[a.dresner]

Fancy. I remember when a board that would take high-end Xeons would cost as much as these "entry-level" server boards do now,

I'm repurposing an x9 board here at my office now, looking at its specs compared to this "home server" board... we have come so far.

 

[a.dresner]

Fancy. I remember when a board that would take high-end Xeons would cost as much as these "entry-level" server boards do now,

Eric, I notice you have written some great stuff on ipmitool, fans, thresholds. Do you have anything written more recently that works with scale?

My server shuts downs gracefully if the fans are not set to full speed. Guessing the errors about breaking the thresholds scare my server and it shuts down :D Like so many before me, looking to lower those thresholds and adjust fan speeds to decrease noise.

 

[Ericloewe]

The process is the exact same on both Linux and FreeBSD.

 

[a.dresner]

The process is the exact same on both Linux and FreeBSD.

Can you do me a solid and put me onto the right path?

 

[Ericloewe]

How To: Change IPMI Sensor Thresholds using ipmitool

Many server motherboards, including the Supermicro X10 series, have their fans monitored by the BMC, which automatically attempts to recover stalling (of the motor, not in the aerodynamic sense) fans by spinning them up to maximum (PWM set to...

www.truenas.com

 

[a.dresner]

How To: Change IPMI Sensor Thresholds using ipmitool

Many server motherboards, including the Supermicro X10 series, have their fans monitored by the BMC, which automatically attempts to recover stalling (of the motor, not in the aerodynamic sense) fans by spinning them up to maximum (PWM set to...

www.truenas.com

Unfortunately I am not getting past the first step
admin@tn1[~]$ kldload ipmi.ko
zsh: command not found: kldload
I'm on Trueness scale, maybe this was written for core?