New Jail - Mysql Fails to Start without any errors

[Mr. Xirtam]

FreeNAS-11.3-U3.2
Jail: 11.3-RELEASE-p10 (Tried both clone and basejail)

I have started a fresh jail and my intentions are to get a FAMP install done. Apache installs just fine and displays the test page correctly when accessed by the IP. When I get to install MariaDB 10.4, and even tried MySQL 8.0, the service fails to start. I run the following:

Code:

pkg install mariadb104-server
sysrc mysql_enable="YES"
service mysql-server start

When I attempt to start the service, it says "Starting mysql." It pauses on this for about 10 seconds, before showing an available command line again. When I check the status service myql-server status, it just says mysql is not running..

I copied the my.cnf file into the /var/db/mysql and verified it was owned by the mysql user/group. I also chown the my.cnf file in /usr/local/etc/mysql.

I have checked in /usr/local/etc/mysql for any kind of error log, ending in .err, but there is nothing listed. So I do not get any kind of error and it fails to start. I have rebooted the FreeNAS entirely, deleted and restarted a fresh, new jail multiple times, and in that process tried different options, mainly using clone or basejail. Is there some kind of file or setting I am missing specific with a jail?

 

[Alecmascot]

I have used this as a guide : Install MariaDB in a Jail
Works fine. Just extract the bits you need.

 

[Mr. Xirtam]

I have used this as a guide : Install MariaDB in a Jail
Works fine. Just extract the bits you need.

This, among many other similar guides I have found and tried. This isn't the first LAMP or FAMP server I have setup, this process is usually the easy part. It just doesn't work. Mysql says it's starting, no error, then it doesn't show it's running. I've tried multiple different versions of mysql, MariaDB 10.4, 10.3 and MySQL 8.0. All with the exact same symptoms. This just leads me to believe something is up with FreeNAS and Jails possibly.

 

[Patrick M. Hausen]

There should be a "my.fully.qualified.hostname.err" file in /var/db/mysql. The content should give you some hints.

 

[Mr. Xirtam]

There should be a "my.fully.qualified.hostname.err" file in /var/db/mysql. The content should give you some hints.

IN my searches, I have come across this as well, yet that file never generates either.

 

[Patrick M. Hausen]

Edit: before you do that - look for a directory named /var/log/mysql and look there for the .err file - the MariaDB ports *just* changed as we speak ...


OK, let's roll up our sleeves ...

1. Instead of service mysql-server start do this as root in the jail: sh -x /usr/local/etc/rc.d/mysql-server start
2. Look for the line that really tries to start your service, near the end of all the log, similar to this:
limits -C daemon su -m mysql -c 'sh -c "/usr/sbin/daemon -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/usr/local/etc/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/vpro0611.proserver.punkt.de.pid "'
3. Copy and paste and execute that entire line instead of using the start script. That should give you more detailled info about what goes wrong

HTH,
Patrick

 

[Mr. Xirtam]

Edit: before you do that - look for a directory named /var/log/mysql and look there for the .err file - the MariaDB ports *just* changed as we speak ...


OK, let's roll up our sleeves ...

1. Instead of service mysql-server start do this as root in the jail: sh -x /usr/local/etc/rc.d/mysql-server start
2. Look for the line that really tries to start your service, near the end of all the log, similar to this:
limits -C daemon su -m mysql -c 'sh -c "/usr/sbin/daemon -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/usr/local/etc/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/vpro0611.proserver.punkt.de.pid "'
3. Copy and paste and execute that entire line instead of using the start script. That should give you more detailled info about what goes wrong

HTH,
Patrick

Code:

root@test:~ # sh -x /usr/local/etc/rc.d/mysql-server start
+ . /etc/rc.subr
+ : 39349
+ export RC_PID
+ [ -n '' ]
+ _rc_subr_loaded=YES
+ SYSCTL=/sbin/sysctl
+ SYSCTL_N='/sbin/sysctl -n'
+ SYSCTL_W=/sbin/sysctl
+ PROTECT=/usr/bin/protect
+ ID=/usr/bin/id
+ IDCMD='if [ -x /usr/bin/id ]; then /usr/bin/id -un; fi'
+ PS='/bin/ps -ww'
+ JID=0
+ rc_service=/usr/local/etc/rc.d/mysql-server
+ _rc_namevarlist='program chroot chdir env flags fib nice user group groups prepend'
+ kenv -q rc.debug
+ name=mysql
+ rcvar=mysql_enable
+ load_rc_config mysql
+ local _name _rcvar_val _var _defval _v _msg _new _d
+ _name=mysql
+ false
+ [ -r /etc/defaults/rc.conf ]
+ debug 'Sourcing /etc/defaults/rc.conf'
+ . /etc/defaults/rc.conf
+ rc_info=NO
+ rc_startmsgs=YES
+ rcshutdown_timeout=90
+ early_late_divider=FILESYSTEMS
+ always_force_depends=NO
+ apm_enable=NO
+ apmd_enable=NO
+ apmd_flags=''
+ ddb_enable=NO
+ ddb_config=/etc/ddb.conf
+ devd_enable=YES
+ devd_flags=''
+ kldxref_enable=NO
+ kldxref_clobber=NO
+ kldxref_module_path=''
+ powerd_enable=NO
+ powerd_flags=''
+ tmpmfs=AUTO
+ tmpsize=20m
+ tmpmfs_flags=-S
+ varmfs=AUTO
+ varsize=32m
+ varmfs_flags=-S
+ mfs_type=auto
+ populate_var=AUTO
+ cleanvar_enable=YES
+ local_startup=/usr/local/etc/rc.d
+ script_name_sep=' '
+ rc_conf_files='/etc/rc.conf /etc/rc.conf.local'
+ zfs_enable=NO
+ zfsd_enable=NO
+ gptboot_enable=YES
+ gbde_autoattach_all=NO
+ gbde_devices=NO
+ gbde_attach_attempts=3
+ gbde_lockdir=/etc
+ geli_devices=''
+ geli_tries=''
+ geli_default_flags=''
+ geli_autodetach=YES
+ root_rw_mount=YES
+ root_hold_delay=30
+ fsck_y_enable=NO
+ fsck_y_flags='-T ffs:-R -T ufs:-R'
+ background_fsck=YES
+ background_fsck_delay=60
+ netfs_types='nfs:NFS smbfs:SMB'
+ extra_netfs_types=NO
+ hostname=''
+ hostid_enable=YES
+ hostid_file=/etc/hostid
+ nisdomainname=NO
+ dhclient_program=/sbin/dhclient
+ dhclient_flags=''
+ background_dhclient=NO
+ synchronous_dhclient=NO
+ defaultroute_delay=30
+ defaultroute_carrier_delay=5
+ netif_enable=YES
+ netif_ipexpand_max=2048
+ wpa_supplicant_program=/usr/sbin/wpa_supplicant
+ wpa_supplicant_flags=-s
+ wpa_supplicant_conf_file=/etc/wpa_supplicant.conf
+ firewall_enable=NO
+ firewall_script=/etc/rc.firewall
+ firewall_type=UNKNOWN
+ firewall_quiet=NO
+ firewall_logging=NO
+ firewall_logif=NO
+ firewall_flags=''
+ firewall_coscripts=''
+ firewall_client_net=192.0.2.0/24
+ firewall_simple_iif=ed1
+ firewall_simple_inet=192.0.2.16/28
+ firewall_simple_oif=ed0
+ firewall_simple_onet=192.0.2.0/28
+ firewall_myservices=''
+ firewall_allowservices=''
+ firewall_trusted=''
+ firewall_logdeny=NO
+ firewall_nologports='135-139,445 1026,1027 1433,1434'
+ firewall_nat_enable=NO
+ firewall_nat_interface=''
+ firewall_nat_flags=''
+ firewall_nat64_enable=NO
+ firewall_nptv6_enable=NO
+ firewall_pmod_enable=NO
+ dummynet_enable=NO
+ ipfw_netflow_enable=NO
+ ip_portrange_first=NO
+ ip_portrange_last=NO
+ ike_enable=NO
+ ike_program=/usr/local/sbin/isakmpd
+ ike_flags=''
+ ipsec_enable=NO
+ ipsec_file=/etc/ipsec.conf
+ natd_program=/sbin/natd
+ natd_enable=NO
+ natd_interface=''
+ natd_flags=''
+ ipfilter_enable=NO
+ ipfilter_program=/sbin/ipf
+ ipfilter_rules=/etc/ipf.rules
+ ipfilter_flags=''
+ ipnat_enable=NO
+ ipnat_program=/sbin/ipnat
+ ipnat_rules=/etc/ipnat.rules
+ ipnat_flags=''
+ ipmon_enable=NO
+ ipmon_program=/sbin/ipmon
+ ipmon_flags=-Ds
+ ipfs_enable=NO
+ ipfs_program=/sbin/ipfs
+ ipfs_flags=''
+ pf_enable=NO
+ pf_rules=/etc/pf.conf
+ pf_program=/sbin/pfctl
+ pf_flags=''
+ pflog_enable=NO
+ pflog_logfile=/var/log/pflog
+ pflog_program=/sbin/pflogd
+ pflog_flags=''
+ ftpproxy_enable=NO
+ ftpproxy_flags=''
+ pfsync_enable=NO
+ pfsync_syncdev=''
+ pfsync_syncpeer=''
+ pfsync_ifconfig=''
+ tcp_extensions=YES
+ log_in_vain=0
+ tcp_keepalive=YES
+ tcp_drop_synfin=NO
+ icmp_drop_redirect=NO
+ icmp_log_redirect=NO
+ network_interfaces=auto
+ cloned_interfaces=''
+ sppp_interfaces=''
+ ppp_enable=NO
+ ppp_program=/usr/sbin/ppp
+ ppp_mode=auto
+ ppp_nat=YES
+ ppp_profile=papchap
+ ppp_user=root
+ hostapd_enable=NO
+ syslogd_enable=YES
+ syslogd_program=/usr/sbin/syslogd
+ syslogd_flags=-s
+ syslogd_oomprotect=YES
+ altlog_proglist=''
+ inetd_enable=NO
+ inetd_program=/usr/sbin/inetd
+ inetd_flags='-wW -C 60'
+ iscsid_enable=NO
+ iscsictl_enable=NO
+ iscsictl_flags=-Aa
+ hastd_enable=NO
+ hastd_program=/sbin/hastd
+ hastd_flags=''
+ ctld_enable=NO
+ local_unbound_enable=NO
+ blacklistd_enable=NO
+ blacklistd_flags=''
+ kdc_enable=NO
+ kdc_program=/usr/libexec/kdc
+ kdc_flags=''
+ kadmind_enable=NO
+ kadmind_program=/usr/libexec/kadmind
+ kpasswdd_enable=NO
+ kpasswdd_program=/usr/libexec/kpasswdd
+ kfd_enable=NO
+ kfd_program=/usr/libexec/kfd
+ kfd_flags=''
+ ipropd_master_enable=NO
+ ipropd_master_program=/usr/libexec/ipropd-master
+ ipropd_master_flags=''
+ ipropd_master_keytab=/etc/krb5.keytab
+ ipropd_master_slaves=''
+ ipropd_slave_enable=NO
+ ipropd_slave_program=/usr/libexec/ipropd-slave
+ ipropd_slave_flags=''
+ ipropd_slave_keytab=/etc/krb5.keytab
+ ipropd_slave_master=''
+ gssd_enable=NO
+ gssd_program=/usr/sbin/gssd
+ gssd_flags=''
+ rwhod_enable=NO
+ rwhod_flags=''
+ rarpd_enable=NO
+ rarpd_flags=-a
+ bootparamd_enable=NO
+ bootparamd_flags=''
+ pppoed_enable=NO
+ pppoed_provider='*'
+ pppoed_flags='-P /var/run/pppoed.pid'
+ pppoed_interface=fxp0
+ sshd_enable=NO
+ sshd_program=/usr/sbin/sshd
+ sshd_flags=''
+ ftpd_enable=NO
+ ftpd_program=/usr/libexec/ftpd
+ ftpd_flags=''
+ amd_enable=NO
+ amd_program=/usr/sbin/amd
+ amd_flags='-a /.amd_mnt -l syslog /host /etc/amd.map /net /etc/amd.map'
+ amd_map_program=NO
+ autofs_enable=NO
+ automount_flags=''
+ automountd_flags=''
+ autounmountd_flags=''
+ nfs_client_enable=NO
+ nfs_access_cache=60
+ nfs_server_enable=NO
+ nfs_server_flags='-u -t'
+ nfs_server_managegids=NO
+ mountd_enable=NO
+ mountd_flags='-r -S'
+ weak_mountd_authentication=NO
+ nfs_reserved_port_only=NO
+ nfs_bufpackets=''
+ rpc_lockd_enable=NO
+ rpc_lockd_flags=''
+ rpc_statd_enable=NO
+ rpc_statd_flags=''
+ rpcbind_enable=NO
+ rpcbind_program=/usr/sbin/rpcbind
+ rpcbind_flags=''
+ rpc_ypupdated_enable=NO
+ keyserv_enable=NO
+ keyserv_flags=''
+ nfsv4_server_enable=NO
+ nfscbd_enable=NO
+ nfscbd_flags=''
+ nfsuserd_enable=NO
+ nfsuserd_flags=''
+ timed_enable=NO
+ timed_flags=''
+ ntpdate_enable=NO
+ ntpdate_program=/usr/sbin/ntpdate
+ ntpdate_flags=-b
+ ntpdate_config=/etc/ntp.conf
+ ntpdate_hosts=''
+ ntpd_enable=NO
+ ntpd_program=/usr/sbin/ntpd
+ ntpd_config=/etc/ntp.conf
+ ntpd_sync_on_start=NO
+ ntpd_flags='-p /var/run/ntpd.pid -f /var/db/ntpd.drift'
+ ntp_src_leapfile=/etc/ntp/leap-seconds
+ ntp_db_leapfile=/var/db/ntpd.leap-seconds.list
+ ntp_leapfile_sources=https://www.ietf.org/timezones/data/leap-seconds.list
+ ntp_leapfile_fetch_opts=-mq
+ ntp_leapfile_expiry_days=30
+ ntp_leapfile_fetch_verbose=NO
+ nis_client_enable=NO
+ nis_client_flags=''
+ nis_ypset_enable=NO
+ nis_ypset_flags=''
+ nis_server_enable=NO
+ nis_server_flags=''
+ nis_ypxfrd_enable=NO
+ nis_ypxfrd_flags=''
+ nis_yppasswdd_enable=NO
+ nis_yppasswdd_flags=''
+ nis_ypldap_enable=NO
+ nis_ypldap_flags=''
+ bsnmpd_enable=NO
+ bsnmpd_flags=''
+ defaultrouter=NO
+ static_arp_pairs=''
+ static_ndp_pairs=''
+ static_routes=''
+ natm_static_routes=''
+ gateway_enable=NO
+ routed_enable=NO
+ routed_program=/sbin/routed
+ routed_flags=-q
+ arpproxy_all=NO
+ forward_sourceroute=NO
+ accept_sourceroute=NO
+ atm_enable=NO
+ atm_pvcs=''
+ atm_arps=''
+ hcsecd_enable=NO
+ hcsecd_config=/etc/bluetooth/hcsecd.conf
+ sdpd_enable=NO
+ sdpd_control=/var/run/sdp
+ sdpd_groupname=nobody
+ sdpd_username=nobody
+ bthidd_enable=NO
+ bthidd_config=/etc/bluetooth/bthidd.conf
+ bthidd_hids=/var/db/bthidd.hids
+ rfcomm_pppd_server_enable=NO
+ rfcomm_pppd_server_profile='one two'
+ rfcomm_pppd_server_one_channel=1
+ rfcomm_pppd_server_two_channel=3
+ ubthidhci_enable=NO
+ netwait_enable=NO
+ netwait_timeout=60
+ netwait_if_timeout=30
+ icmp_bmcastecho=NO
+ ipv6_network_interfaces=auto
+ ipv6_activate_all_interfaces=NO
+ ipv6_defaultrouter=NO
+ ipv6_static_routes=''
+ ipv6_gateway_enable=NO
+ ipv6_cpe_wanif=NO
+ ipv6_privacy=NO
+ route6d_enable=NO
+ route6d_program=/usr/sbin/route6d
+ route6d_flags=''
+ ipv6_default_interface=NO
+ rtsol_flags=''
+ rtsold_enable=NO
+ rtsold_flags=-a
+ rtadvd_enable=NO
+ rtadvd_interfaces=''
+ mroute6d_enable=NO
+ mroute6d_program=/usr/local/sbin/pim6dd
+ mroute6d_flags=''
+ stf_interface_ipv4addr=''
+ stf_interface_ipv4plen=0
+ stf_interface_ipv6_ifid=0:0:0:1
+ stf_interface_ipv6_slaid=0000
+ ipv6_ipv4mapping=NO
+ ipv6_ipfilter_rules=/etc/ipf6.rules
+ ip6addrctl_enable=YES
+ ip6addrctl_verbose=NO
+ ip6addrctl_policy=AUTO
+ keyboard=''
+ keymap=NO
+ keyrate=NO
+ keybell=NO
+ keychange=NO
+ cursor=NO
+ scrnmap=NO
+ font8x16=NO
+ font8x14=NO
+ font8x8=NO
+ blanktime=300
+ saver=NO
+ moused_nondefault_enable=YES
+ moused_enable=NO
+ moused_type=auto
+ moused_port=/dev/psm0
+ moused_flags=''
+ mousechar_start=NO
+ allscreens_flags=''
+ allscreens_kbdflags=''
+ mta_start_script=/etc/rc.sendmail
+ sendmail_enable=NO
+ sendmail_pidfile=/var/run/sendmail.pid
+ sendmail_procname=/usr/sbin/sendmail
+ sendmail_flags='-L sm-mta -bd -q30m'
+ sendmail_cert_create=YES
+ sendmail_submit_enable=YES
+ sendmail_submit_flags='-L sm-mta -bd -q30m -ODaemonPortOptions=Addr=localhost'
+ sendmail_outbound_enable=YES
+ sendmail_outbound_flags='-L sm-queue -q30m'
+ sendmail_msp_queue_enable=YES
+ sendmail_msp_queue_flags='-L sm-msp-queue -Ac -q30m'
+ sendmail_rebuild_aliases=NO
+ auditd_enable=NO
+ auditd_program=/usr/sbin/auditd
+ auditd_flags=''
+ auditdistd_enable=NO
+ auditdistd_program=/usr/sbin/auditdistd
+ auditdistd_flags=''
+ cron_enable=YES
+ cron_program=/usr/sbin/cron
+ cron_dst=YES
+ cron_flags=''
+ lpd_enable=NO
+ lpd_program=/usr/sbin/lpd
+ lpd_flags=''
+ nscd_enable=NO
+ chkprintcap_enable=NO
+ chkprintcap_flags=-d
+ dumpdev=NO
+ dumpdir=/var/crash
+ savecore_enable=YES
+ savecore_flags='-m 10'
+ crashinfo_enable=YES
+ crashinfo_program=/usr/sbin/crashinfo
+ quota_enable=NO
+ check_quotas=YES
+ quotaon_flags=-a
+ quotaoff_flags=-a
+ quotacheck_flags=-a
+ accounting_enable=NO
+ ibcs2_enable=NO
+ ibcs2_loaders=coff
+ firstboot_sentinel=/firstboot
+ sysvipc_enable=NO
+ linux_enable=NO
+ svr4_enable=NO
+ clear_tmp_enable=NO
+ clear_tmp_X=YES
+ ldconfig_insecure=NO
+ ldconfig_paths='/usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg'
+ ldconfig32_paths='/usr/lib32 /usr/lib32/compat'
+ ldconfigsoft_paths='/usr/libsoft /usr/libsoft/compat /usr/local/libsoft'
+ ldconfig_paths_aout='/usr/lib/compat/aout /usr/local/lib/aout'
+ ldconfig_local_dirs=/usr/local/libdata/ldconfig
+ ldconfig_local32_dirs=/usr/local/libdata/ldconfig32
+ ldconfig_localsoft_dirs=/usr/local/libdata/ldconfigsoft
+ kern_securelevel_enable=NO
+ kern_securelevel=-1
+ update_motd=YES
+ entropy_boot_file=/boot/entropy
+ entropy_file=/entropy
+ entropy_dir=/var/db/entropy
+ entropy_save_sz=4096
+ entropy_save_num=8
+ harvest_mask=511
+ dmesg_enable=YES
+ watchdogd_enable=NO
+ watchdogd_flags=''
+ devfs_rulesets='/etc/defaults/devfs.rules /etc/devfs.rules'
+ devfs_system_ruleset=''
+ devfs_set_rulesets=''
+ devfs_load_rulesets=YES
+ performance_cx_lowest=C2
+ performance_cpu_freq=NONE
+ economy_cx_lowest=Cmax
+ economy_cpu_freq=NONE
+ virecover_enable=YES
+ ugidfw_enable=NO
+ bsdextended_script=/etc/rc.bsdextended
+ newsyslog_enable=YES
+ newsyslog_flags=-CN
+ mixer_enable=YES
+ opensm_enable=NO
+ rctl_enable=YES
+ rctl_rules=/etc/rctl.conf
+ iovctl_files=''
+ jail_enable=NO
+ jail_conf=/etc/jail.conf
+ jail_confwarn=YES
+ jail_parallel_start=NO
+ jail_list=''
+ jail_reverse_stop=NO
+ [ -z '' ]
+ source_rc_confs_defined=yes
+ [ -r /etc/defaults/vendor.conf ]
+ source_rc_confs
+ local i sourced_files
+ sourced_files=:/etc/rc.conf:
+ [ -r /etc/rc.conf ]
+ . /etc/rc.conf
+ cron_flags=' -J 15'
+ sendmail_enable=NO
+ sendmail_submit_enable=NO
+ sendmail_outbound_enable=NO
+ sendmail_msp_queue_enable=NO
+ syslogd_flags='-c -ss'
+ ipv6_activate_all_interfaces=YES
+ hostname=test
+ ifconfig_epair0b=SYNCDHCP
+ mysql_enable=YES
+ sourced_files=:/etc/rc.conf::/etc/rc.conf.local:
+ [ -r /etc/rc.conf.local ]
+ _rc_conf_loaded=true
+ [ -n mysql ]
+ _d=/etc
+ [ -f /etc/rc.conf.d/mysql ]
+ [ -d /etc/rc.conf.d/mysql ]
+ _d=/usr/local/etc
+ [ -f /usr/local/etc/rc.conf.d/mysql ]
+ [ -d /usr/local/etc/rc.conf.d/mysql ]
+ eval '_defval=$mysql_enable_defval'
+ _defval=''
+ [ -n '' ]
+ : YES
+ : NO
+ : mysql
+ mysql_limits_args='-e -U mysql'
+ : /var/db/mysql
+ : /var/db/mysql/my.cnf
+ command=/usr/sbin/daemon
+ procname=/usr/local/libexec/mysqld
+ start_precmd=mysql_prestart
+ start_postcmd=mysql_poststart
+ [ -n '' ]
+ [ -n '' -a -n start ]
+ /bin/hostname
+ mysql_pidfile=/var/db/mysql/test.pid
+ pidfile=/var/db/mysql/test.pid
+ mysql_install_db=/usr/local/bin/mysql_install_db
+ mysql_install_db_args='--basedir=/usr/local --datadir=/var/db/mysql --force'
+ command_args='-c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid '
+ run_rc_command start
+ _return=0
+ rc_arg=start
+ [ -z mysql ]
+ shift 1
+ rc_extra_args=''
+ _rc_prefix=''
+ eval '_override_command=$mysql_program'
+ _override_command=''
+ command=/usr/sbin/daemon
+ _keywords='start stop restart rcvar enabled describe extracommands '
+ rc_pid=''
+ _pidcmd=''
+ _procname=/usr/local/libexec/mysqld
+ [ -n /usr/local/libexec/mysqld ]
+ [ -n /var/db/mysql/test.pid ]
+ _pidcmd='rc_pid=$(check_pidfile /var/db/mysql/test.pid /usr/local/libexec/mysqld )'
+ _keywords='start stop restart rcvar enabled describe extracommands  status poll'
+ [ -z start ]
+ [ start '=' enabled ]
+ [ -n '' ]
+ eval 'rc_flags=$mysql_flags'
+ rc_flags=''
+ eval '_chdir=$mysql_chdir' '_chroot=$mysql_chroot' '_nice=$mysql_nice' '_user=$mysql_user' '_group=$mysql_group' '_groups=$mysql_groups' '_fib=$mysql_fib' '_env=$mysql_env' '_prepend=$mysql_prepend' '_login_class=${mysql_login_class:-daemon}' '_oomprotect=$mysql_oomprotect'
+ _chdir='' _chroot='' _nice='' _user=mysql _group='' _groups='' _fib='' _env='' _prepend='' _login_class=daemon _oomprotect=''
+ [ -n mysql ]
+ eval if [ -x /usr/bin/id '];' then /usr/bin/id '-un;' fi
+ [ -x /usr/bin/id ]
+ /usr/bin/id -un
+ [ mysql '=' root ]
+ [ -z '' ]
+ eval 'rc_pid=$(check_pidfile' /var/db/mysql/test.pid /usr/local/libexec/mysqld ')'
+ check_pidfile /var/db/mysql/test.pid /usr/local/libexec/mysqld
+ _pidfile=/var/db/mysql/test.pid
+ _procname=/usr/local/libexec/mysqld
+ _interpreter=''
+ [ -z /var/db/mysql/test.pid -o -z /usr/local/libexec/mysqld ]
+ [ ! -f /var/db/mysql/test.pid ]
+ debug 'pid file (/var/db/mysql/test.pid): not readable.'
+ return
+ rc_pid=''
+ [ start '!=' start ]
+ [ -n mysql_enable -a start '!=' rcvar -a start '!=' stop -a start '!=' describe ]
+ checkyesno mysql_enable
+ eval '_value=$mysql_enable'
+ _value=YES
+ debug 'checkyesno: mysql_enable is set to YES.'
+ return 0
+ [ start '=' start -a -z '' -a -n '' ]
+ eval '_cmd=$start_cmd' '_precmd=$start_precmd' '_postcmd=$start_postcmd'
+ _cmd='' _precmd=mysql_prestart _postcmd=mysql_poststart
+ [ -n '' ]
+ [ ! -x /usr/sbin/daemon ]
+ _run_rc_precmd
+ check_required_before start
+ local _f
+ return 0
+ [ -n mysql_prestart ]
+ debug 'run_rc_command: start_precmd: mysql_prestart '
+ eval 'mysql_prestart '
+ mysql_prestart
+ local dir
+ [ -f /etc/my.cnf ]
+ [ -f /etc/mysql/my.cnf ]
+ [ ! -d /var/db/mysql/mysql/. ]
+ checkyesno mysql_limits
+ eval '_value=$mysql_limits'
+ _value=NO
+ debug 'checkyesno: mysql_limits is set to NO.'
+ return 1
+ return 0
+ _return=0
+ [ 0 -ne 0 ]
+ check_required_after start
+ local _f _args
+ return 0
+ return 0
+ check_startmsgs
+ [ -n '' ]
+ return 0
+ echo 'Starting mysql.'
Starting mysql.
+ [ -n '' ]
+ _cd=''
+ _doit='/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid '
+ [ -n mysql ]
+ _doit=$'su -m mysql -c \'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "\''
+ [ -n '' ]
+ [ -n '' ]
+ _doit=$' limits -C daemon su -m mysql -c \'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "\''
+ _run_rc_doit $' limits -C daemon su -m mysql -c \'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "\''
+ debug $'run_rc_command: doit:  limits -C daemon su -m mysql -c \'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "\''
+ eval $' limits -C daemon su -m mysql -c \'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "\''
+ limits -C daemon su -m mysql -c 'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "'
+ _return=0
+ [ 0 -ne 0 ]
+ return 0
+ _run_rc_postcmd
+ [ -n mysql_poststart ]
+ debug 'run_rc_command: start_postcmd: mysql_poststart '
+ eval 'mysql_poststart '
+ mysql_poststart
+ local 'timeout=15'
+ [ ! -f /var/db/mysql/test.pid -a 15 -gt 0 ]
+ timeout=14
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 14 -gt 0 ]
+ timeout=13
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 13 -gt 0 ]
+ timeout=12
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 12 -gt 0 ]
+ timeout=11
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 11 -gt 0 ]
+ timeout=10
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 10 -gt 0 ]
+ timeout=9
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 9 -gt 0 ]
+ timeout=8
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 8 -gt 0 ]
+ timeout=7
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 7 -gt 0 ]
+ timeout=6
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 6 -gt 0 ]
+ timeout=5
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 5 -gt 0 ]
+ timeout=4
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 4 -gt 0 ]
+ timeout=3
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 3 -gt 0 ]
+ timeout=2
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 2 -gt 0 ]
+ timeout=1
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 1 -gt 0 ]
+ timeout=0
+ sleep 1
+ [ ! -f /var/db/mysql/test.pid -a 0 -gt 0 ]
+ return 0
+ _return=0
+ return 0
+ [ -n '' ]
+ return 0

Wow, this is a very detailed output. I'm not seeing anything in particular of where it gets caught up. Unless it's the reference to test.pid? I'm not seeing that particular file when I look in that folder, only a folder called test. If I create the test.pid file and assign mysql ownership to it, I run that command again and it runs through immediately, but same outcome. It just doesn't do that 15 second timeout at the end where it displays "Starting mysql."

 

[Patrick M. Hausen]

No, no, no ... nobody needs that output

I asked you to look for this particular line, then copy & paste that into the SSH session again:
limits -C daemon su -m mysql -c 'sh -c "/usr/sbin/daemon -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "'

So please do so and then report back, what that results in.

 

[Mr. Xirtam]

No, no, no ... nobody needs that output

I asked you to look for this particular line, then copy & paste that into the SSH session again:
limits -C daemon su -m mysql -c 'sh -c "/usr/sbin/daemon -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "'

So please do so and then report back, what that results in.

Oof thank goodness. Sorry about that, here's the condensed version.

Code:

Starting mysql.
+ [ -n '' ]
+ _cd=''
+ _doit='/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid '
+ [ -n mysql ]
+ _doit=$'su -m mysql -c \'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "\''
+ [ -n '' ]
+ [ -n '' ]
+ _doit=$' limits -C daemon su -m mysql -c \'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "\''
+ _run_rc_doit $' limits -C daemon su -m mysql -c \'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "\''
+ debug $'run_rc_command: doit:  limits -C daemon su -m mysql -c \'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "\''
+ eval $' limits -C daemon su -m mysql -c \'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "\''
+ limits -C daemon su -m mysql -c 'sh -c "/usr/sbin/daemon  -c -f /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mysql --datadir=/var/db/mysql --pid-file=/var/db/mysql/test.pid "'
+ _return=0

 

[Patrick M. Hausen]

No ...

Copy and paste that line - without the leading "+ " into a root shell prompt and execute it.

What we did was trace all the commands that start shellscript executed until it arrives at the command line it uses when trying to start the service. Unfortunately when the script is run, output of the commands is directed somewhere else so we don't see them.

The tracing showed us what precisely the script is doing.

Now we execute that very same command without all the script magic around it and we will hopefully see an error message.

So please (as in my last message). now execute everything beginning with limits -C daemon su ... and report back.

 

[Mr. Xirtam]

No ...

Copy and paste that line - without the leading "+ " into a root shell prompt and execute it.

What we did was trace all the commands that start shellscript executed until it arrives at the command line it uses when trying to start the service. Unfortunately when the script is run, output of the commands is directed somewhere else so we don't see them.

The tracing showed us what precisely the script is doing.

Now we execute that very same command without all the script magic around it and we will hopefully see an error message.

So please (as in my last message). ow execute everything beginning with limits -C daemon su ... and report back.

Ah sorry for the confusion, I did execute that and nothing was output:

Service is still not running after that was executed and there still isn't a test.err file in /var/db/mysql.

 

[Patrick M. Hausen]

That. Is. Weird.

What are owner/group and access rights of your /var/db/mysql directory?

 

[Mr. Xirtam]

That. Is. Weird.

What are owner/group and access rights of your /var/db/mysql directory?

Right? I did check ownership on that too:

 

[Patrick M. Hausen]

Now for even more output you could try to ktrace or truss that sucker ...

"When in doubt use brute force."
-- Ken Thompson

Let me do a quick check if that works in a jail ...

 

[Patrick M. Hausen]

truss -f service mysql-server start

Then read backwards and look for errors like "no such file or directory" or anything that might give a hint to why it is failing.

 

[Mr. Xirtam]

truss -f service mysql-server start

Then read backwards and look for errors like "no such file or directory" or anything that might give a hint to why it is failing.

This was incredibly helpful. The trace is definitely detailed! I found a few flags in there. It looks like the culprit was when it was trying to make the directory /var/run/mysql.

Code:

42511: mkdir("/var",0777)                        ERR#17 'File exists'
42511: stat("/var",{ mode=drwxr-xr-x ,inode=20,size=24,blksize=4096 }) = 0 (0x0)
42511: mkdir("/var/run",0777)                    ERR#17 'File exists'
42511: stat("/var/run",{ mode=drwxr-xr-x ,inode=46,size=16,blksize=4096 }) = 0 (0x0)
42511: umask(0x7)                                = 7 (0x7)
42511: mkdir("/var/run/mysql",0777)              ERR#13 'Permission denied'
42511: write(2,"mkdir: ",7)                      = 7 (0x7)
42511: write(2,"/var/run/mysql",14)              = 14 (0xe)
42511: write(2,": ",2)                           = 2 (0x2)

After I manually made that folder and chmod 0777 the folder, service mysql-server start executes flawlessly and I now have a running server.
Interestingly enough, looking at this trace, it would look like it should have made that test.err file:

Code:

42508: fstat(0,{ mode=p--------- ,inode=8,size=35,blksize=4096 }) = 0 (0x0)
42508: read(0,"--log-error=/var/db/mysql/test.e"...,4096) = 35 (0x23)
42508: fstat(1,{ mode=p--------- ,inode=10,size=0,blksize=4096 }) = 0 (0x0)
42508: read(0,0x800e35000,4096)                  = 0 (0x0)
42508: write(1,"--log-error=/var/db/mysql/test.e"...,35) = 35 (0x23)
42454: read(3,"--log-error=/var/db/mysql/test.e"...,128) = 35 (0x23)
42508: close(1)

So I'm not sure what was happening to that file.
One other section I saw was this:

Code:

42511: stat("/usr/share/nls/C/libc.cat",0x7fffffffe458) ERR#2 'No such file or directory'
42511: stat("/usr/share/nls/libc/C",0x7fffffffe458) ERR#2 'No such file or directory'
42511: stat("/usr/local/share/nls/C/libc.cat",0x7fffffffe458) ERR#2 'No such file or directory'
42511: stat("/usr/local/share/nls/libc/C",0x7fffffffe458) ERR#2 'No such file or directory'
42511: write(2,"Permission denied\n",18)         = 18 (0x12)

Should this be a concern?

 

[Patrick M. Hausen]

Probably not. It is looking for locale files that are not there.
Glad the BFG worked for you.