Scale memory utilization

[sretalla]

I still am curious about services memory. What is this? I have no vm or docker apps.

SMB, NFS, the middleware, and Apps/VMs (which you don't have).

 

[asap2go]

Would someone happen to know if there is an issue/ticket where the topic of ARC memory usage on SCALE is tracked?
I couldn't find anything in the GitHub repo but also did not want to open a separate post just to ask this.

 

[jgreco]

Would someone happen to know if there is an issue/ticket where the topic of ARC memory usage on SCALE is tracked?

Pretty sure there isn't. Your question implies that there's a problem with ZFS, but what's really going on here is that the memory management on Linux sucks. This isn't an "issue" that will be fixed within ZFS or within SCALE. It really requires some buy-in from the Linux development community, and there's this guy named Linus Torvalds who maintains a pretty good chokehold on Linux kernel architecture and development. Linus has repeatedly shown complete disdain for ZFS and no interest in improvements that might help ZFS.

ZFS on FreeBSD, on the other hand, integrates well with kernel memory management and a lot of work has gone into making the ARC and UNIX userland peacefully coexist to make excellent use of otherwise free memory. That's been going on for ten or fifteen years now, which is part of why CORE is so awesome.

I don't know what your exact question is, and you're certainly free to ask it, but please be aware that there is an elephant in the room that no one really likes to discuss. Many ZFS people are scared to be honest about it, for fear of further alienating the elephant, though some of us also just hate Linux and don't mind speaking truth to crappy software. No one is "tracking" this as an issue, at least that I know of. All the hard work of fixing it was done years ago in FreeBSD and similar changes could be made to Linux if the elephant would permit it.

 

[asap2go]

Thanks for the answer!
I was aware that it is an issue with linux memory management and not zfs or SCALE. Just hoped that someone would try to get a patch upstreamed or create some customised memory management for the kernel used in SCALE.
But that's probably far far easier said than done.
For the time being I guess staying on Core is the only option if one doesn't have a 50:50 storage/virtualisation workload.
Or did someone test setting max_arc tunable via init script? I was afraid that might lead to a memory overcommit.

 

[jgreco]

But that's probably far far easier said than done.

That. It's not that it is impossible to do, but iXsystems is a small team facing many challenges. From their perspective, I believe that they are fine saying "well then just double the memory". It's a short-term and probably shortsighted fix, but their commercial customers are mostly running TrueNAS Enterprise or CORE anyways. There are lots of reasons to avoid SCALE, and it may be some time before SCALE is considered production quality even if you "solve" this by doubling memory.

Or did someone test setting max_arc tunable via init script? I was afraid that might lead to a memory overcommit.

That works fine, as far as I know. The problem is that it isn't optimal. Especially on small-memory systems.

On a large system, you might find that it's sufficient to set the ARC size to 96GB of 128GB and let the userland have 32GB, even if maybe on CORE that might be fluttering between 20GB-32GB and you might see CORE "squeeze" up to 12GB more of ARC out of that. In SCALE this becomes a static configuration, something you need to keep an eye on and think about a little. So that's not catastrophic.

On a small memory system, where you only have 16GB to begin with, SCALE limits the ARC to half that, which can be really hurt-y under heavy load. If you try to squeeze more by tunable, you incur swapping, which is generally bad, and if you're foolish enough to try spinning up containers for stuff like Plex that can take lots of memory resources, it can be very painful, whereas CORE would simply nuke most of your ARC and take the RAM that it needed. That would cause poor performance and slow I/O, but that's generally still better than swapping.

So you're kinda in the correct lane if you're worrying about memory overcommit. Plan accordingly, allocate lots of resources for your NAS. It CAN be made to work out. I just personally prefer the sleek automatedness of it all under CORE, and Linus and his sucktacular attitude gives me just another reason to dislike Linux.

 

[asap2go]

I've got 128GB DDR4 3200 ECC which is the maximum for my mainboard. So doubling is not possible and I need a few gigs for my 512GB L2ARC (many files >10GB). As far as I've understood L2ARC consumes RAM which is dedicated to ARC.

"It CAN be made to work out."

Yes, that's also my impression and probably the main takeaway.
But "can" ist not something I want to commit to for my main storage server.
I just run Nextcloud and a few games servers (in a Linux VM) on Core.
But I have to little experience to judge the variability in memory consumption of those.
Also the tunable max_arc_size seems to get changed whenever a container vor VM is spun up.
At least it behaves that way on my test VM.
The only possibility seems to be to allocate a single Linux VM with fixed resources and run everything inside that.
Will test that but most likely I'll stay on Core and give up on the advantages in containerization, Plugin and better HW support and management.
Core works really predictable and nice.
It's most likely WannaHave-Syndrome with SCALE for me.

Thanks for all the info!

 

[Patrick M. Hausen]

You don't need a VM to run Nextcloud on CORE. A jail has way less overhead and most important no fixed memory consumption. Similarly Minecraft and other servers run well in jails.

 

[asap2go]

You don't need a VM to run Nextcloud on CORE. A jail has way less overhead and most important no fixed memory consumption. Similarly Minecraft and other servers run well in jails.

I know. But the jail images are less up to date or plain non existant and I need the VM anyway for my game servers.
e.g. Nextcloud AiO only exists as docker image for now.

 

[winnielinnie]

and there's this guy named Linus Torvalds who maintains a pretty good chokehold on Linux kernel architecture and development. Linus has repeatedly shown complete disdain for ZFS and no interest in improvements that might help ZFS.

Linus and his sucktacular attitude gives me just another reason to dislike Linux.

To be fair, his main gripe is not that of a technical hurdle, but rather because he fears litigation from Oracle.¹

"And honestly, there is no way I can merge any of the ZFS efforts until I get an official letter from Oracle that is signed by their main legal counsel or preferably by Larry Ellison himself that says that yes, it's OK to do so and treat the end result as GPL'd."

"Other people think it can be OK to merge ZFS code into the kernel and that the module interface makes it OK, and that's their decision. But considering Oracle's litigious nature, and the questions over licensing, there's no way I can feel safe in ever doing so."

¹ https://www.zdnet.com/article/linus...ode-on-linux-until-litigious-larry-signs-off/

 

[jgreco]

To be fair, his main gripe is not that of a technical hurdle, but rather because he fears litigation from Oracle.¹

It's pretty clear to those of us who have been watching this for some time that this is the most pathetic of fig leaves to hide behind. FreeBSD, NetBSD, iXsystems, Joyent, Nexenta, Phoronix, and others have all been using ZFS for over a decade without issue. If you bother to read a bit deeper, it's that Linus has issues with the license under which ZFS has been made available (CDDL), but Linus could also release Linux under CDDL or BSD and finally join the world of free software. Linus actually has significant issues with things that were "not invented here" and the licensing thing is just a really weak cover. If you read up more on the history of this, you'll see him treating the topic of ZFS repeatedly with disdain.

 

[asap2go]

It's pretty clear to those of us who have been watching this for some time that this is the most pathetic of fig leaves to hide behind. FreeBSD, NetBSD, iXsystems, Joyent, Nexenta, Phoronix, and others have all been using ZFS for over a decade without issue. If you bother to read a bit deeper, it's that Linus has issues with the license under which ZFS has been made available (CDDL), but Linus could also release Linux under CDDL or BSD and finally join the world of free software. Linus actually has significant issues with things that were "not invented here" and the licensing thing is just a really weak cover. If you read up more on the history of this, you'll see him treating the topic of ZFS repeatedly with disdain.

I don't think he can just release linux under the CDDL.
To release software under a different, incompatible license
he would most likely need a written confirmation from every single contributor thus far.
Which is not feasible.
#GPL problems.

 

[jgreco]

I don't think he can just release linux under the CDDL.
To release software under a different, incompatible license
he would most likely need a written confirmation from every single contributor thus far.
Which is not feasible.
#GPL problems.

That seems to be more of a him-and-his-stupid-GPL problem than anything else. If he had just used a reasonable free software license in the first place, rather than the RMS/GNU garbage. It really only takes one contributor to have died for it to become impossible to change the license on the whole thing. That's not free. That's moronic.
#BSD4e4

 

[Ericloewe]

I'll just add that "Oracle might sue!" is the most absurd legal argument floating around in the open-source world, not because they wouldn't, but because how the hell could they ever sue?! The CDDL doesn't say all code that touches it has to be CDDL, that's the GPL!

"Hey, we're going to sue you for using our software entirely in line with its license because your license for your software does not allow you to use our software!"

What kind of ridiculous logic is this? This doesn't make one iota of sense, it's absolutely ludicrous. What are the damages here, zero dollars and zero cents? It's clearly a post-hoc justification for a decision made, most likely, on the basis of Not Invented Here syndrome.

It's not even a matter of fscking principle, look at Nvidia's drivers! They get a free pass to ship their proprietary flaming pile of garbage, meanwhile the very much open-source OpenZFS has to jump through hoops to work with the Linux kernel whenever certain divalopers [sic] feel like throwing in another hoop to jump through because anything else would be a tacit admission that their precious btrfs is broken by design and terribly implemented.

I don't dislike the entirety of Linux, but I absolutely hate the "release half-broken, braindead crap now and slowly iterate until something more usable comes out". I think Docker is the poster child for that. The concept is very convenient, and it's a damn shame the FreeBSD community did not put together a combination of lobotomized shell script that serves as the blueprint, plus an open host of pre-built images - because that's why Docker became popular, the underlying technologies on Linux are hacky and the people at Docker made some truly bizarre decisions to go along with their stuff (Why on Earth are Docker images opaque? Why is there no easy way to cd into an image? It's not the layering.) in order to solve problems that had been solved better by the likes of FreeBSD and Illumos.

 

[Patrick M. Hausen]

@Ericloewe Agree. People don't want "Docker". They want an easy one-stop way to start a multi-tier application without having to learn how to configure all the moving parts.

 

[Ericloewe]

@Ericloewe Agree. People don't want "Docker". They want an easy one-stop way to start a multi-tier application without having to learn how to configure all the moving parts.

Careful, I've got a Docker Compose rant in me still...

 

[Patrick M. Hausen]

Careful, I've got a Docker Compose rant in me still...

Most users just use images or compose files that others prepared.

 

[NugentS]

@morganL responded to a thread elsewhere here saying that IX plan to do something (not that I know what something is) in Cobia

Scale 50% use of ARC

I understand that this is an upstream issue and that scale is a work in progress but there is one thing that bothers me, perhaps even irritates me (slightly). That is the Linux ZFS only uses 50% of memory (its not actually that bad when there are loads of additional services. 256GB of RAM and I...

www.truenas.com

 

[asap2go]

@morganL responded to a thread elsewhere here saying that IX plan to do something (not that I know what something is) in Cobia

Scale 50% use of ARC

I understand that this is an upstream issue and that scale is a work in progress but there is one thing that bothers me, perhaps even irritates me (slightly). That is the Linux ZFS only uses 50% of memory (its not actually that bad when there are loads of additional services. 256GB of RAM and I...

www.truenas.com

"For Linux, its the safe margin for ARC due to fragmentation on SLUB allocator.
We're treating it as a bug and plan to fix or mitigate it....
Planning to fix in Cobia."

This also sounds Luke setting max ARC size manually is a bad idea because there might be some memory fragmentation(sounds serious?). Thank you for the info!
Hope Morgan can tell what Cobia means.

 

[Etorix]

It's good to know that iXSystems is "treating it as a bug" (my emphasis).
But given where the bug resides, my (totally unqualified and uninformed) guess is for some hacky mitigation, trying to push max_arc as far as they think they can go without involving the dreaded swap, rather than an actual solution (which would require GPL-fanatics doing something for the sake of practicality).

@asap2go "Cobia" is the next SCALE release after "Bluefin". Fishes in alphabetical order… (Mischievous minds would wonder whether SCALE is a fishy business.)

Careful, I've got a Docker Compose rant in me still...

Please spit your venom. Don't let it consume your from inside.

 

[asap2go]

"guess is for some hacky mitigation, trying to push max_arc as far as they think they can go"
To me it sounds like the opposite.
max_arc seems to be fixed so low as a safety margin because of an underlying memory issue.
I would assume they intend ro fix memory allocation/fragmentation/reporting instead of just increasing max_arc. Because the latter of which may cause instability when unintentionally overcommiting memory.