[WIP] MusicBrainz mirror on FreeNAS

[cyberjock]

Yeah, I'm not able to work on this like I want. I'm away from home due to a family emergency. :(

 

[joelmusicman]

Cyber any ETA to "instructions" ??? has a lot of people anxious for it ... kkkkk :p

I found some reading that I believe it's very close to what you have done. But I found it extremely lengthy... i don't if this may help
(https://www.zufallsheld.de/2014/09/15/install-headphones-and-a-musicbrainz-server-in-your-freenas/)

Thanks for help!

Thanks for posting this guide. I am happy to report that I got it working using this method. Note that the guide mentioned RDP, but it's actually VNC that is needed to access the console (I had quite a bit of frustration until I figured this out). Also note that the guide mentioned a specific db dump; I replaced this with the latest version.

I haven't yet hooked it into Headphones, but I browsed to port 5000 and poked around a bit. I don't anticipate any problems though...

 

[Rudi Pittman]

Unfortunately, no. I'm wrapped up with 9.3 stuff right now (In fact, I'm watching my Mini throw up all over itself) while writing instructions for BIOS updates and stuff for FreeNAS Mini owners.

Hopefully this weekend I can get to this.

Did you ever get a chance to setup the musicbrainz template? Still plenty of users who would like to use it with headphones without the need to use virtualbox.

 

[Gunsmithy]

Did you ever get a chance to setup the musicbrainz template? Still plenty of users who would like to use it with headphones without the need to use virtualbox.

Agreed! Would love to see that template!

 

[Osiris]

so ... the sysctls of the jail ... should they look like this ?
allow.raw_sockets=true,allow.sysvipc=1

I had to do a lot of stuff with pkg install instead of using ports and vice versa.
And that second coffee: make that a coffee, a bath and a haircut.

Anyway, nice guide. As a base for installing musicbrainz, it gave me lots of info.
Any word on that template?

Still stuck on the carton issues

Code:

! Installing the dependencies failed: Module 'ExtUtils::MakeMaker' is not installed
! Bailing out the installation for ExtUtils-MakeMaker-7.12.

... while ...

Code:

root@musicbrain2:/home/musicbrainz/musicbrainz-server # perl -MExtUtils::MakeMaker -e 'warn $ExtUtils::MakeMaker::VERSION;'
7.12 at -e line 1.

 

[cyberjock]

So I've stopped attempting to make templates for several reasons. Right now I'm in serious question as to how much I'll be updating my main server for the forseeable future. I recently installed 9.10 again on my main server and I cannot create jails anymore. Apparently FreeNAS has made some changes and you cannot make new jails on 9.10 without an empty jails directory. This is a show stopper for me since I have about 20 jails, many running in situations I cannot easily "turn off", and at over 100 hours of work to recreate them, I'm not keen on the idea of having to redo them, especially with the possible prospect of having to redo them again when 10 comes out.

Jails are basically at a "dead end" and it appears they are going to be removed entirely. So spending time on jails seems like a waste.

https://forums.freenas.org/index.php?threads/faq-upgrading-from-9-3-to-9-10.42964/#post-281745

So for right now I'm on the last 9.3.1 build, I'm doing very little (if anything) to jails, and I'm just watching and waiting (with baited breath?)

Choosing to kill the jails function (as it sounds in that link above) is a total show stopper for me. So I may be staying on 9.3.1 forever. :/

 

[Osiris]

Cyberjock, I can only thank you to make me aware of this terrible change.
Meanwhile, I'll continue to experiment to get musicbrainz running in a 9.3 jail.

I've been checking https://github.com/elliotchance/mbzdb
It's possible to run an alternative version of musicbrainz in mysql.

 

[tonyschuite]

ok, so, apparently freenas 9.3 is actually making things A LOT better.

so far I have built it without any hiccups. right now I am downloading the dumps and it seems to work.
will post updates.

 

[Osiris]

without hickups? You didn't encounter the issues with carton that I posted above?

 

[tonyschuite]

actually... I have it running for about 10 Days... works like a charm.

Have a cron job running through the Freenas GUI that runs the replication and it works really good.

I have used the steps in the main Musicbrainz compile guide. If something didn't work or I needed the freebsd alternative I used my first post to look for solutions.
It is on my todo list to write a new guide.

no Carton problems (don't think I used it actually).

The source however to that specific carton problem is that carton sometimes leaves shadows of installed programs.
I had the same problem. try running "cpanm --uninst-shadows ExtUtils::MakeMaker"
next, run the command

cpanm --installdeps --notest .

from the dir and it should work.
don't forget
pkg install npm

as it is a new dependency

try following this: https://github.com/metabrainz/musicbrainz-server/blob/master/INSTALL.md
as much as you can. use freshports to see if there is a freebsd alternative.

 

[Osiris]

Got it running in Freenas 9.10 jail, based on this postgresql-how-to, tonyschuite's instructions and this doc.
I used only pkg & ports to install all packages.
cpanm too but I avoided using carton altogether.

pkg upgrading & other stuff tends to break the postgresql startup & service scripts.
I had to replace them with my own.

Install sudo to autorun the db as user musicbrainz.
Create the user first.
Also, create the folder /var/log/postgresql.

This is /usr/local/etc/rc.d/postgresql

Code:

#!/bin/sh
. /etc/rc.subr
name=postgresql
rcvar=postgresql_enable
start_precmd="${name}_prestart"
start_cmd="${name}_start"
stop_cmd="${name}_stop"
postgresql_prestart() {
        if ! checkyesno postgresql_enable && ! /etc/rc.d/postgresql forcestatus 1>/dev/null 2>&1; then
                force_depend postgresql || return 1
        fi
        return 0
}
postgresql_start() {
        /usr/local/bin/sudo -u musicbrainz /usr/local/bin/pg_ctl start -s -l /var/log/postgresql/log -D /usr/local/pgsql/data
}
postgresql_stop() {
        /usr/local/bin/sudo -u musicbrainz /usr/local/bin/pg_ctl stop -s -l /var/log/postgresql/log -D /usr/local/pgsql/data
}

load_rc_config $name
run_rc_command "$1"

Then I created /usr/local/etc/rc.d/musicbrainz to be able to start/stop the musicbrainz service
It's kinda shady, I know.

Code:

#!/bin/sh
. /etc/rc.subr
name=musicbrainz
rcvar=musicbrainz_enable
start_precmd="${name}_prestart"
start_cmd="${name}_start"
stop_cmd="${name}_stop"
musicbrainz_prestart() {
  if ! checkyesno musicbrainz_enable && ! /etc/rc.d/postgresql forcestatus 1>/dev/null 2>&1; then
  force_depend postgresql || return 1
  fi
  return 0
}
musicbrainz_start() {
  cd /musicbrainz-server
  /usr/local/bin/plackup --daemonize -Ilib -r &
}
musicbrainz_stop() {
  pkill -f plackup
}

load_rc_config $name
run_rc_command "$1"

Put

Code:

musicbrainz_enable="YES"

in /etc/rc.conf to autostart with the jail.

For replication, I had to patch to v-2016-05-23-schema-change-v2 using git checkout -f
After that I cronned
/musicbrainz-server/admin/replication/LoadReplicationChanges
once a day

Freenas server has the following cronjob (tasks --> cronjobs)

Code:

jexec 33 /usr/local/bin/perl /musicbrainz-server/admin/replication/LoadReplicationChanges

*UPDATE* something is broken again by patching. Autostarting no longer works & my attempt to daemonize plackup needs rewriting.
*UPDATE 20/6/2016* autostart remains a mistery. At jail start I manually launch postgress and musicbrainz from the commands above.

 

[styno]

Just a heads up as I just now noticed that the ova based install is no longer syncing new entries into the database and that there is no updated image available yet.
A jail based install seems the way to go for now on FreeNas. I hope to find some time to play with this as well.

 

[Osiris]

As an extra to that, I can tell you that the OVA took about 50 times more resources on my server than my jailed version of the musicbrainz mirror does.
I'd use the virtualized version only of you have some kind of vmware'ish monster in your basement that can carry the load.

 

[lukyjay]

Is this something we can update for FreeNAS 11 (iocage) ?

 

[drphilth]

Is this something we can update for FreeNAS 11 (iocage) ?

I'm working on this at the moment. It is almost done, I just need to figure out how to get the server started by an init script.

Most of it is scriptable, and can be run from outside the jail using "iocage exec" commands.

I'll update once I have it working reliably.

 

[drphilth]

I'm working on this at the moment. It is almost done, I just need to figure out how to get the server started by an init script.

Most of it is scriptable, and can be run from outside the jail using "iocage exec" commands.

I'll update once I have it working reliably.

Coming along nicely. Running in FastCGI mode via nginx, and using supervisor to start the server process. Next step is figuring out the search server install and then running the whole script through a deployment to find the inevitable bugs.

 

[lukyjay]

No idea what any of that means bu I'm looking forward to it

 

[drphilth]

No idea what any of that means bu I'm looking forward to it

It'll all become clear once you see the absolute pig this thing is to get installed! There are so many processes that take hours to complete. I think I'm about halfway through the search server. Then it'll be a case of tidying it all up and running through the whole thing from scratch once to see if it actually works in a repeatable way.

 

[drphilth]

OK. It is working.

I have not gone ahead with a local search server deployment. The main reason for this is that the current source is broken, and nobody at musicbrainz is going to fix it becasue they are in the middle of a move to a different search solution.

There are binaries available for download, but the index updates mechanism is broken, so you have to reindex the entire database every time there is an update from musicbrainz. On my machine, it was taking 8 hours to do an index build, which is ridiculous, when musicbrainz pushes update packets every hour.

What this means is that your local musicbrainz server will be reliant on the public search server. When they have finished their migration to their new search server I'll take a look at running it locally.

Because this is in the archived posts section, I'm going to add my solution to the FN11.1 iocage jails discussion over at https://forums.freenas.org/index.ph...ansmission-with-vpn.55502/page-14#post-436100