FN11.1 Jail Setup for musicbrainz server 2018-02-10

[drphilth]

drphilth submitted a new resource:

FN11.1 Jail Setup for musicbrainz server - Scripted setup for a local musicbrainz mirror

Based on the method here

Please make sure you have a separate dataset for your jail configuration files "/path/to/apps" which is outside your iocage dataset "/path/to/iocage"

Place the downloaded folder "musicbrainz" inside your apps folder - "/path/to/apps/musicbrainz"

Customise the jail setup script...

Read more about this resource...

 

[Ol'WhistleBritches]

I'm having no luck with this... when postgresql goes to start and initializedb during the install, I get

LOG: could not bind IPv6 socket: Can't assign requested address
HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
LOG: could not bind IPv4 socket: Can't assign requested address
HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
WARNING: could not create listen socket for "localhost"
FATAL: could not create any TCP/IP sockets
LOG: database system is shut down
pg_ctl: could not start server
Examine the log output.

I also can't find that log output. Any thoughts?

 

[drphilth]

That's a bit strange. It seems like it is failing at this stage:

Code:

# Initilaise pgsql
iocage exec $APP_NAME /usr/local/etc/rc.d/postgresql oneinitdb

What happens if you drop into an iocage console and try to run
service postgresql oneinitdb
or
/usr/local/etc/rc.d/postgresql oneinitdb

 

[Ol'WhistleBritches]

I agree with your diagnosis of when the failure occurs

service postgresql oneinitdb
returns the same response.

I'll try the second command once I'm home. Wondering if I didn't get the IP address set correctly in mb_server_setup.sh. Wasn't sure what needed to be adjusted there if its just on a local network...

Do you know where postgresql's data directory is installed to? Everything I've found on pgsql suggests the .conf file should be in that directory.

 

[Ol'WhistleBritches]

Couldn't initialize the db using /usr/local/etc/rc.d/postgresql oneinitdb because a db already existed. IE I think the db was initialized properly during the setup script, it just couldn't start because of the configuration.

So I managed to get pg started by editing postgres.conf and replacing

listen_addresses = 'localhost'
with
listen_addresses = 'X.X.X.X'

where X.X.X.X is my jail's local IP address. I have no idea if that's a kosher solution long term, but there was no way pg wanted to start as 'localhost' or '127.0.0.1'.

I'm really at a loss as to what to do now... I think I'll go ahead and run mb_server_setup.sh again from inside the jail because it was failing due to postgres being not started. Should I re-execute all the commands in setupjail.sh after the "....press enter" as well?

 

[Ol'WhistleBritches]

Well I'm one step closer.... running mb_server_setup I get stuck at:

Code:

Failed query:
		'SELECT current_setting('server_version_num')'
		()
08006 DBI connect('dbname=template1;host=127.0.0.1;port=5432','postgres',...) failed: could not connect to server: Network is unreachable
		Is the server running on host "127.0.0.1" (127.0.0.1/32) and accepting
		TCP/IP connections on port 5432?
 at /usr/local/share/musicbrainz-server/admin/../lib/Sql.pm line 460.
		Sql::catch {...} (MusicBrainz::Server::Exceptions::DatabaseError=HASH(0x8064c9930)) called at /usr/local/lib/perl5/site_perl/Try/Tiny.pm line 123
		Try::Tiny::try(CODE(0x805facc90), Try::Tiny::Catch=REF(0x805fe71b0), Try::Tiny::Finally=REF(0x802c8b498)) called at /usr/local/share/musicbrainz-server/admin/../lib/Sql.pm line 466
		Sql::_select_list(Sql=HASH(0x8054f9b88), "SELECT current_setting('server_version_num')", ARRAY(0x805ec2180), "arrayref", CODE(0x805ec2090)) called at /usr/local/share/musicbrainz-server/admin/../lib/Sql.pm line 485
		Sql::select_list_of_lists(Sql=HASH(0x8054f9b88), "SELECT current_setting('server_version_num')") called at /usr/local/share/musicbrainz-server/admin/../lib/Sql.pm line 416
		Sql::select_single_column_array(Sql=HASH(0x8054f9b88), "SELECT current_setting('server_version_num')") called at /usr/local/share/musicbrainz-server/admin/../lib/Sql.pm line 431
		Sql::select_single_value(Sql=HASH(0x8054f9b88), "SELECT current_setting('server_version_num')") called at /usr/local/share/musicbrainz-server/admin/InitDb.pl line 73
		main::RequireMinimumPostgreSQLVersion() called at /usr/local/share/musicbrainz-server/admin/InitDb.pl line 521
08006 DBI connect('dbname=template1;host=127.0.0.1;port=5432','postgres',...) failed: could not connect to server: Network is unreachable
		Is the server running on host "127.0.0.1" (127.0.0.1/32) and accepting
		TCP/IP connections on port 5432?

Since I've already modified the listening address in postgres.conf to remove 'localhost', the fact that its being referenced here leads me to believe that somewhere in mb's scripts there is another variable referencing 127.0.0.1.

Any thoughts?

 

[drphilth]

A couple of things:

in setupjail.sh, the IP address and gateway you chose are the ip address you intend the jail to have on your local network - locally mine is set to 192.168.2.something for IP and 192.168.2.1 for gateway

I haven't changed the default listen address, but if you have, you'll need to edit it in DBDefs.pm - this is why the import scripts are failing.

what does your /usr/local/share/postgresql/pg_ident.conf look like? It should have been replaced by the script to look like:

Code:

# PostgreSQL User Name Maps
# =========================
#
# Refer to the PostgreSQL documentation, chapter "Client
# Authentication" for a complete description. A short synopsis
# follows.
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
# are of the form:
#
# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
# existence of a record specifies that SYSTEM-USERNAME may connect as
# PG-USERNAME.
#
# If SYSTEM-USERNAME starts with a slash (/), it will be treated as a
# regular expression. Optionally this can contain a capture (a
# parenthesized subexpression). The substring matching the capture
# will be substituted for \1 (backslash-one) if present in
# PG-USERNAME.
#
# Multiple maps may be specified in this file and used by pg_hba.conf.
#
# No map names are defined in the default configuration. If all
# system user names and PostgreSQL user names are the same, you don't
# need anything in this file.
#
# This file is read on server startup and when the postmaster receives
# a SIGHUP signal. If you edit the file on a running system, you have
# to SIGHUP the postmaster for the changes to take effect. You can
# use "pg_ctl reload" to do that.
# Put your actual configuration here
# ----------------------------------
# MAPNAME	 SYSTEM-USERNAME		 PG-USERNAME
local all	all	trust

 

[drphilth]

Interestingly, I am now having the same problem in my musicbrainz jail which was previously working fine. I wonder if something got broken by a freenas update. I'm investigating...

 

[drphilth]

There's definitely something wrong with networking in the iocage jail. I can't ping localhost:

Code:

root@musicbrainz:~ # ping localhost
PING localhost (127.0.0.1): 56 data bytes
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable

 

[drphilth]

https://forums.freenas.org/index.php?threads/iocage-network-issues.60402/

Seems this problem was introduced in 11.1-U2. Solutions are to use a different virtual interface name than vnet0, roll back to 11.1-U1 or wait for 11.2


EDIT:
running
sysrc ifconfig_epair0_name="epair0b"
inside the jail worked to restore networking.

I'd recommend you recreate the jail at this point, but try editing this line in setupjail.sh:

Code:

# Setup iocage jail
iocage create -n "$APP_NAME" -p ${SCRIPT_DIR}/pkg.json -r 11.1-RELEASE ip4_addr="vnet0|$IP_ADDR/24" defaultrouter="$IP_ROUTER" vnet="on" allow_raw_sockets="1" boot="on"

to

Code:

# Setup iocage jail
iocage create -n "$APP_NAME" -p ${SCRIPT_DIR}/pkg.json -r 11.1-RELEASE ip4_addr="epair0b|$IP_ADDR/24" defaultrouter="$IP_ROUTER" vnet="on" allow_raw_sockets="1" boot="on"

And let me know how it goes

EDIT2: above doesn't work.

Add this in setupjail.sh

Code:

# Fix networking
iocage exec $APP_NAME sysrc ifconfig_epair0_name="epair0b"
iocage restart $APP_NAME

Above

Code:

########
# Setup musicbrainz_server
########

And see if that helps

 

[Ol'WhistleBritches]

Finally back to work on this project....

The epair0b fix works. I can ping localhost no problem.

I destroyed the jail and started fresh. Good news is that I'm past oneinitdb problem. Bad news is that the initdb.sh command in mb_server_setup.sh fails due to some other tar error "exit delayed from previous errors".

At this point, I have no idea whos script is failing. Is there a log file associated with initdb.pl?

 

[drphilth]

the mb_server_setup.sh script really just runs three commands.

The one which is failing is
/usr/local/share/musicbrainz-server/admin/InitDb.pl --createdb --import /tmp/mbdump/mbdump*.tar.bz2 --echo

The /tmp/mbdump files should have been created much much earlier in the process by the setupjail.sh script -

Code:

# Get latest mb dump ID
MB_DUMPID=$(curl -L http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/LATEST)
iocage exec $APP_NAME mkdir -p /tmp/mbdump
iocage exec $APP_NAME curl http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/$MB_DUMPID/mbdump-cdstubs.tar.bz2 -o /tmp/mbdump/mbdump-cdstubs.tar.bz2
iocage exec $APP_NAME curl http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/$MB_DUMPID/mbdump-cover-art-archive.tar.bz2 -o /tmp/mbdump/mbdump-cover-art-archive.tar.bz2
iocage exec $APP_NAME curl http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/$MB_DUMPID/mbdump-derived.tar.bz2 -o /tmp/mbdump/mbdump-derived.tar.bz2
iocage exec $APP_NAME curl http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/$MB_DUMPID/mbdump-edit.tar.bz2 -o /tmp/mbdump/mbdump-edit.tar.bz2
iocage exec $APP_NAME curl http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/$MB_DUMPID/mbdump-editor.tar.bz2 -o /tmp/mbdump/mbdump-editor.tar.bz2
iocage exec $APP_NAME curl http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/$MB_DUMPID/mbdump-stats.tar.bz2 -o /tmp/mbdump/mbdump-stats.tar.bz2
iocage exec $APP_NAME curl http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/$MB_DUMPID/mbdump.tar.bz2 -o /tmp/mbdump/mbdump.tar.bz2

(I don't know why musicbrainz have set their servers up this way. It would make much more sense for http://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/LATEST to be an alias of the directory containing the latest files, rather than a text string containing the directory name...)

I'd suggest checking that the files have downloaded correctly, and if so, then run /usr/local/share/musicbrainz-server/admin/InitDb.pl --createdb --import /tmp/mbdump/mbdump*.tar.bz2 --echo from inside an iocage console and see what happens.