Emby Mono only running single threaded

[pclausen]

I have a pretty beefy server running FreeNAS with the Emby plugin. For some reason, mono is only running single threaded all the time, unlike ffmpeg, which will use all 28 threads of my CPU at once when transcoding.

As a result, when a user is navigating my collection from an Emby client, there are delays while mono is doing its thing. Running top shows the mono process at 100% and consuming only a single thread.

Luke over on the Emby forum suggested that it may be some sort of environment limitation. Any ideas?

Here's a video of my screen showing the delay:

http://www.cstone.net/~dk/Embybrowsespeed.flv

 

[Joshua Parker Ruehlig]

if you have a suggestion on how to fix this id be happy to encorporate them into the plugin.

 

[mattbbpl]

I wish I had more (read: any) experience with Mono as this looks like a pretty fun issue to dig into. It might be related to the issue outlined in the link below based on my limited knowledge of Mono - or it may not be. Nor do I know if the "fix" is viable in this situation, but it sounds like Joshua may be able to better deduce that.

http://stackoverflow.com/questions/17554945/mono-multiprocessing-performance-issue

 

[Joshua Parker Ruehlig]

I wish I had more (read: any) experience with Mono as this looks like a pretty fun issue to dig into. It might be related to the issue outlined in the link below based on my limited knowledge of Mono - or it may not be. Nor do I know if the "fix" is viable in this situation, but it sounds like Joshua may be able to better deduce that.

http://stackoverflow.com/questions/17554945/mono-multiprocessing-performance-issue

that seems promising. can someone test out editing /usr/local/etc/rc.d/emby-server and inserting "--gc=sgen" after procname on the command_args line

 

[Ixian]

The plugin is already using sgen garbage collection though - you can check this yourself by looking at the running process (mono-sgen).

The actual trick might be in how it's utilizing the mark and sweep collector. By default it's single threaded I believe. However changing this is a build-time option in Mono, see http://www.mono-project.com/docs/advanced/garbage-collector/sgen/working-with-sgen/

 

[Joshua Parker Ruehlig]

from checking here, there's no knobs for me to change. so maybe we should submit a patch to add this to the port

http://www.freshports.org/lang/mono

 

[Ixian]

Just double checked:

root@emby_2:/ # /usr/pbi/emby-amd64/bin/mono --version
Mono JIT compiler version 4.0.1 (tarball Thu Sep 3 05:23:50 UTC 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notification: kqueue
Architecture: amd64
Disabled: none
Misc: softdebug
LLVM: supported, not enabled.
GC: sgen
For reference.

 

[Ixian]

Just sanity checking here:

For grins, I installed the Mono FreeBSD package, which was recently updated (from the port) to 4.2 (the pbi is on 4.0). I ran it from there to see if there was any difference; there wasn't, but it was at that point I realized I don't have this problem with either version.

That is, on my FreeNAS system, Mono is definately using multiple threads. I can see this in Top by count, by wCPU and raw, and by actually displaying the threads (in FreeBSD Top, press capital H to toggle displaying all the threads a process is using).

I toggled thread display view then browsed around the Emby web client and watched the threads spawn/etc.

Also, I don't have a problem with a slow web client, in fact mine is rather snappy. I jumped in this thread because I saw PClausen's post on the Emby forums and I am working on a separate Mono project for work so this kind of troubleshooting interested me. However it doesn't appear to be a general problem.

PClausen, I'm running FreeNAS 9.3 stable on an Intel Xeon CPU E3-1230 v3 - sounds like less power than you have? I'm using the latest Emby PBI, 5781.3 which I haven't modified. As mentioned I tried running Emby under the Mono that comes with the PBI and the package version with no difference, but maybe you could try that as a test?

 

[Ixian]

Here's a screenshot of my Top output with threads displayed - what does yours look like?

 

[pclausen]

I too am running the latest Emby (5781.3). Here's my top with the H option. This first screenshot is when I go to list the movie posters. There is about a 10-15 second delay before they display. If I go back out and back in, the same delay happens every time.

If I then change the sort order, the posters slowly begin to redraw, and it takes about 30 seconds for the first 100 to complete, and then the process repeats when I go to the 2nd page of the next 100 and so on. I have about 2,400 movie titles. Here's what top looks like during the process:

So it appears to be using additional threads, but it is still very slow.

Looking at the overall server CPU usage during this, it's very low as seen here:

I should mention that I run the Cover Art plugin, in case that has something to do with it.

Also, I don't believe it is an issue with my pool being slow. Scrubs run at about 1.7G/s.

 

[kjp4756]

I have around 1300 movies on my emby server and I don't experience the issue you are seeing. There is virtually no delay in bringing up posters. I am running the regular ports version of emby in a jail. I'm also not using the cover art plugin. I bet the cover art plugin is what's causing your issue.

 

[Ixian]

I was using CoverArt a while ago and don't remember a problem this bad, but it did lead to slower loading of posters in Kodi on Android TV as well as the Android tablet client. So I disabled it. Can you try removing it and see what happens?

 

[Joshua Parker Ruehlig]

Also I'm assuming you have the slowdown issue with multiple browsers/operating systems? How about when you access emby remotely?

 

[pclausen]

Yeah, same issue with Edge as with Chrome, both on my workstation and HTPC. Also seeing the same slowness from Emby Theater on the HTPC. If anything, ET is slower that the browsers.

On my Roku, the speed is not bad, but then again there's a very limited amount displayed on the screen at once.

I did a lot of custom profiles for the Cover Art plug-in, but I suppose it wouldn't' take that long to redo. I'll see about removing it when I get a chance to see if it makes a difference.

 

[Ixian]

You're using Freenas :) If you aren't already taking automatic snapshots of your jails take a manual snapshot before you remove the plugin and if removing the plugin makes no difference roll back.

 

[pclausen]

Duh, of course. So I went ahead and removed the CoverArt plugin and that took care of it. Everything is pretty much instantaneous now.

So I guess CoverArt doesn't properly "cache" the custom covers that it generates. I mean, I know that when I first install it and browse my collection, it takes a really long time to get going, especially on Emby Theater, and then during subsequent sessions, it "only" takes 10-15 seconds to get into the main movie list. So maybe it does cache the images, but with 2,400 movie titles (and just under 10,000 TV episodes), perhaps I'm hitting some sort of limit? For example, with CoverArt running, changing the sort order, clearly causes all the movie posters to be re-created by the plugin.

I'll ping EBR over on the Emby forum to see if perhaps he can optimize the code some.

In the meantime, I think I'll leave this "eye candy" off, and it makes everything run so much smoother.

 

[Ixian]

CoverArt makes heavy use of ImageMagick these days as I recall. One thing you could try is building the latest version from the port in your jail (actually I think you can just install the pkg since it is in sync with the port, however with the port you have control over build options, one of which is openMP for multithreading - I haven't checked to see if that is enabled by default).

If I remember right, as is the case with ffmpeg, Mono, etc. you can edit the emby-server startup script in rc.local and change the path to Imagemagick from the one that is included in the PBI to the port/package, which is newer. Be useful to test, anyway.

 

[Joshua Parker Ruehlig]

CoverArt makes heavy use of ImageMagick these days as I recall. One thing you could try is building the latest version from the port in your jail (actually I think you can just install the pkg since it is in sync with the port, however with the port you have control over build options, one of which is openMP for multithreading - I haven't checked to see if that is enabled by default).

If I remember right, as is the case with ffmpeg, Mono, etc. you can edit the emby-server startup script in rc.local and change the path to Imagemagick from the one that is included in the PBI to the port/package, which is newer. Be useful to test, anyway.

looks like openmp is off. I'll enable if for the next release.
http://www.freshports.org/graphics/ImageMagick-nox11/

as for pointing to the right imagick library this is in some file in emby, not in the init script.

 

[Joshua Parker Ruehlig]

Here's a plugin build with OPENMP enabled for ImageMagick
https://www.jruehlig.com/owncloud/s/xdCI4RVE2hA1Ycs

@pclausen does this solve the issue you were hitting?

 

[pclausen]

Thanks Josh. So I should install this rather than the 5781.4 update I see in the FreeNAS GUI under plugins?

If so, what is the process for loading the .pbi directly?