Plex & ARC Integration

[tc60045]

Many of us run ZFS for myriad purposes. For those who use it in DCs or “work” purposes, I salute you, but this thread is for media enthusiasts who love ZFS for many of the same reasons, but have very different workloads.

I imagine an ARC that is told to load files in anticipation as I and my users navigate Plex:
- When a movie is selected, Plex “tells” ARC to go ahead and load it as soon as the movie is selected, so that if I hit play, the file is resident in RAM
- When an episode of TV is selected, similarly, load it to ARC
- When a TV series is selected, load the oldest unwatched into ARC
- Etc

Sure, there could be unnecessary reads, but those can be thwarted with a “lookie Lou” counter of titles-viewed-but-not-played-in-last-5-minutes, with a TTL.

How can we bridge the gap between Plex and ZFS to make this possible?

Thanks for your thoughts. TC

 

[danb35]

How can we bridge the gap between Plex and ZFS to make this possible?

By rewriting Plex to be ZFS-aware, at a bare minimum. Except even that wouldn't do it, because Plex runs in a jail, and the jail never sees ZFS.

Think about this for a second--is it really a good idea for userspace software to be directly manipulating the ARC? I'm pretty sure the answer is "no."

 

[tc60045]

By rewriting Plex to be ZFS-aware, at a bare minimum. Except even that wouldn't do it, because Plex runs in a jail, and the jail never sees ZFS.

Think about this for a second--is it really a good idea for userspace software to be directly manipulating the ARC? I'm pretty sure the answer is "no."

Doesn’t run in a jail on my box. And there is no reason why an ARC messaging bus couldn’t see suggestions. YMMV on a DC production box; this box is my own to configure.

 

[danb35]

Doesn’t run in a jail on my box.

Sorry, I figured that since you were posting on the FreeNAS forums, your suggestion had something, however remote, to do with FreeNAS.

 

[Ericloewe]

Many of us run ZFS for myriad purposes. For those who use it in DCs or “work” purposes, I salute you, but this thread is for media enthusiasts who love ZFS for many of the same reasons, but have very different workloads.

I imagine an ARC that is told to load files in anticipation as I and my users navigate Plex:
- When a movie is selected, Plex “tells” ARC to go ahead and load it as soon as the movie is selected, so that if I hit play, the file is resident in RAM
- When an episode of TV is selected, similarly, load it to ARC
- When a TV series is selected, load the oldest unwatched into ARC
- Etc

Sure, there could be unnecessary reads, but those can be thwarted with a “lookie Lou” counter of titles-viewed-but-not-played-in-last-5-minutes, with a TTL.

How can we bridge the gap between Plex and ZFS to make this possible?

Thanks for your thoughts. TC

Plex could just, you know, load the file into RAM (as much as possible). It's not rocket science, the ARC will give way to real memory usage.

Now, a better question is, "does this make my server faster or better in any way?" I can see scenarios for both "yes" and "no". Given that media is not particularly difficult to stream from disk, I'd say it's probably best to keep it out of RAM as much as possible, beyond a smallish buffer.

 

[Ericloewe]

because Plex runs in a jail, and the jail never sees ZFS.

It could, that's not a problem per se. Just another knob that has to be (carefully) exposed to the jail, similar to existing options for datasets assigned to jails.

That's one of the many wonders of ZFS. It integrates with jails in a way that just makes so much sense that you go "How is this not on every OS on the planet yet!?".

 

[tc60045]

Now, a better question is, "does this make my server faster or better in any way?" I can see scenarios for both "yes" and "no". Given that media is not particularly difficult to stream from disk, I'd say it's probably best to keep it out of RAM as much as possible, beyond a smallish buffer.

That is a better question. Try transcoding files out of RAM and then try transcoding off disk — world of difference. And with remote users, that difference matters. So if you have the RAM, why not just nudge ARC to make that possible?

 

[Ericloewe]

Try transcoding files out of RAM and then try transcoding off disk

Well, you should never be transcoding straight from disk, hence the buffer. But we're getting into very specific details that are best left available for experienced users to tune.

 

[tc60045]

Well, you should never be transcoding straight from disk, hence the buffer. But we're getting into very specific details that are best left available for experienced users to tune.

Exactly - so how might one nudge ARC to allow experienced users to tune. None of us is one who would stand in anesthetized awe of the immutable holy ARC. What say we tell ARC what we want it to cache? How?...

 

[Ericloewe]

Exactly - so how might one nudge ARC to allow experienced users to tune.

Well, you wouldn't for situations like this, because the application itself is in a much better position to manage the buffer size and it can operate directly on the buffer, instead of making repeated syscalls to get this and get that.

None of us is one who would stand in anesthetized awe of the immutable holy ARC.

Well, it's already tunable.

 

[tc60045]

Well, you wouldn't for situations like this, because the application itself is in a much better position to manage the buffer size and it can operate directly on the buffer, instead of making repeated syscalls to get this and get that.

You trust your Nest; I take mine apart at the code level. You say to trust it; I smile at the improvements I’ve long enjoyed with my own machine learning code.

As for tunable, nah; ARC needs an API like Nest has. Then we can call it tunable.