How to upgrade the ports tree and customize FreeNAS 8.0.3!

Status
Not open for further replies.

bryanburke

Dabbler
Joined
Nov 11, 2011
Messages
11
IMPORTANT: Please read the guide completely to avoid confusion and prevent errors!

Good day, everyone! I've updated the guide and the patch for FreeNAS 8.0.3. This guide will show you how to upgrade the ports tree (almost) as a whole and even add ports like transmission-daemon to the FreeNAS image. This guide requires the ability to build FreeNAS 8.0.3 from source using the FreeBSD command-line; if you have no idea how to do this, consult joeschmuck‎'s excellent guide in this same forum.

Note: This modification to the FreeNAS 8.0.3 source is NOT sponsored or supported by the development team. Do NOT send them any questions about this guide!

Note: I am NOT responsible for any damage that may occur to your data as a result of this method. If you are concerned about the safety of the following instructions, MAKE A BACKUP FIRST!

Note: I can only certify that this method works with the 8.0.3 source. Don't bug me with questions about older versions or trunk!

Note: Read all notes...

Step 1: Get the Sources

If you already have the 8.0.3 sources, skip to step 2.

We use svn just like we would for trunk, except we are pulling the 8.0.3-RELEASE tag instead:

Code:
svn co https://freenas.svn.sourceforge.net/svnroot/freenas/tags/8.0.3-RELEASE


The new folder created on your local machine (or virtual machine) is called "8.0.3-RELEASE" and is the root directory during the FreeNAS build. From here on I will simply call it FREENAS_ROOT.

Step 2: Patch the Build Script

Here is my unified diff which you can use to patch "build/do_build.sh":

Code:
--- build/do_build.sh	2012-01-11 22:07:06.000000000 -0600
+++ build/do_build.sh	2012-01-13 14:18:47.000000000 -0600
@@ -34,7 +34,8 @@
         echo "No sup host defined, please define FREEBSD_CVSUP_HOST and rerun"
         exit 1
     fi
-    echo "Checking out tree from ${FREEBSD_CVSUP_HOST}..."
+    echo "Checking out up-to-date tree from ${FREEBSD_CVSUP_HOST}..."
+# Fetch up-to-date ports tree with this file
     cat <<EOF > FreeBSD/supfile
 *default host=${FREEBSD_CVSUP_HOST}
 *default base=${root}/FreeBSD/sup
@@ -44,8 +45,39 @@
 *default compress
 
 src-all tag=RELENG_8_2
+ports-all date=.
+EOF
+# Create a new supfile to backport only the ports we need to patch
+    cat <<EOF > FreeBSD/supfile_backports
+*default host=${FREEBSD_CVSUP_HOST}
+*default base=${root}/FreeBSD/sup
+*default prefix=${root}/FreeBSD
+*default release=cvs
+*default delete use-rel-suffix
+*default compress
+
 ports-all date=2011.07.17.00.00.00
 EOF
+# Create a file containing the categories and names of all
+# ports that will be patched (and therefore must be backported)
+# 
+# Also backport open-vm-tools to prevent a packaging error in the latest version
+    cat <<EOF > FreeBSD/ports.topatch
+sysutils/ataidle
+dns/inadyn
+net/istgt
+net-mgmt/net-snmp
+net/netatalk
+sysutils/nut
+www/py-django
+www/py-dojango
+net/rsync
+net/samba36
+databases/tdb
+emulators/open-vm-tools
+emulators/open-vm-tools-nox11
+EOF
+
 	# Nuke the newly created files to avoid build errors.
 	#
 	# patch(1) will automatically append to the previously non-existent
@@ -61,6 +93,13 @@
 	# Force a repatch because csup pulls pristine sources.
 	: > $FREENAS_ROOT/FreeBSD/src-patches
 	: > $FREENAS_ROOT/FreeBSD/ports-patches
+
+# Now backport the ports that will be patched
+for port in $(cat ${root}/FreeBSD/ports.topatch); do
+	echo "Backporting $port from ${FREEBSD_CVSUP_HOST}..."
+	csup -L 1 -i "ports/$port" ${root}/FreeBSD/supfile_backports
+done
+
 fi
 
 for patch in $(cd $FREENAS_ROOT/patches && ls freebsd-*.patch); do


Copy that code and place it in a file called "do_build.sh.diff" in FREENAS_ROOT. Then run the patch command to apply it:

Code:
patch < do_build.sh.diff


Note: The diff file MUST be in FREENAS_ROOT for the above command to work as it is.

So, what does this patch actually do? Well, the original build script downloads the ports tree as it existed on July 17, 2011. The development team froze FreeNAS 8's ports tree on that date to apply custom patches to several ports. On the downside, though, all unpatched ports are frozen at older versions. So, to update the ports tree while still allowing the FreeNAS patches to be applied, the above patch to the build script does the following:

1) Download a current, up-to-date ports tree initially.
2) Regress the few ports that have custom patches for FreeNAS back to their July 17, 2011 snapshots.
3) Backport emulators/open-vm-tools as well to prevent a packaging error in the most recent version.

That's really all there is to it!

Step 3: Customize FreeNAS with More Ports (Optional)

Use your favorite command-line text editor (I use nano in my below examples) and open up the file "freenas-common" under the "nanobsd" directory.

Scroll down until you see a bunch of lines that start with "add_port" and go to the bottom of the list, which ends in "add_port sysutils/ipmitool" (line 254).

Now simply use the same syntax to add additional ports below that line with the formula:

Code:
add_port <category>/<port_name>


For example, to add transmission-daemon, the code would be like this:

Code:
add_port net-p2p/transmission-daemon


Step 4: Build FreeNAS

If you have previously built FreeNAS, you might have the "FreeBSD" directory in FREENAS_ROOT. If you do, type the following commands:

Code:
cd FreeBSD/
rm -rf supfile sup/ src-patches ports-patches
cd ../


Those commands will ensure that the patched build script updates everything.

Now from FREENAS_ROOT run the following commands (yes, dropping into sh first can prevent build errors):

Code:
sh
export FREEBSD_CVSUP_HOST=cvsup10.freebsd.org
build/do_build.sh


You can, of course substitute any CVSUP host for the one above. For a list of cvsup mirrors and their statuses, see www.mavetju.org/unix/freebsd-mirrors/cvsup-stats-global.php

When the build finishes, you should have your custom images waiting for you in the "obj.***" directory, where the *** is either i386 or amd64.

If you don't know how to install the custom image, refer to the official FreeNAS documentation.

Step 5: Make Your Changes Persist Between Reboots (Optional)

Once your FreeNAS installation has booted and you have configured it to your liking (or restored your previous configuration), you can modify "/etc/rc.conf" to start ports you added in step 3 at boot (like transmission-daemon) and pass arguments to their RC scripts (such as configuration directories, etc.). To do this, use the following commands from the FreeNAS command-line (must be logged in as root):

Code:
mount -uw /
cd /conf/base/etc/
nano -w rc.conf


Make your changes to the file as required. For example, I have the following inserted for transmission-daemon:

Code:
transmission_enable="YES"
transmission_conf_dir="/mnt/zfs_volume/config/transmission"
transmission_user="root"


Refer to each port's documentation for applicable RC options.

Now reboot your FreeNAS installation and your changes will be made!

--------------------------------------------

Thanks for reading! Let me know if there are any mistakes or if anything is missing.

Please check the ADDENDUM below before asking any questions to make sure I have not already answered them.
 

bryanburke

Dabbler
Joined
Nov 11, 2011
Messages
11
ADDENDUM

--------------------------------------------

How to fix the libSM and libICE packaging error:

If you have already built FreeNAS from source and would like to add more custom ports, you do not have to rebuild from scratch. However, if you issue the "build/do_build.sh" command again, the process will stop and complain about missing packages for libSM and libICE. This appears to be an error in either the FreeNAS or nanoBSD build system, but the fix is simple enough.

Just open up the "nanobsd/freenas-common" file, and go down to line 216 that reads "add_port devel/dbus". Since libSM and libICE are dependencies of dbus, add these two lines before line 216 (after "add_port x11/libX11"):

Code:
add_port x11/libSM
add_port x11/libICE


Now the build script should be able to update the images with your new ports.
 

dpodolak

Cadet
Joined
Jan 13, 2012
Messages
3
Thanks for the guide! One comment though...

Code:
patch < do_build.sh.diff


Note: The diff file MUST be in FREENAS_ROOT for the above command to work as it is.

When I attempted to patch the script, I received the following:

Code:
root@Sandbox:/home/d/8.0.3-RELEASE# patch < do_build.sh.diff 
can't find file to patch at input line 3
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--------------------------
|--- build/do_build.sh	2012-01-11 22:07:06.000000000 -0600
|+++ build/do_build.sh	2012-01-13 14:18:47.000000000 -0600
--------------------------
File to patch: ./build/do_build.sh
patching file ./build/do_build.sh
root@Sandbox:/home/d/8.0.3-RELEASE#


Just wanted to suggest maybe adding that the patch should be applied to the ./build/do_build.sh script for those of us that may or may not spend 30 minutes trying to figure this out :)

Also, I'm curious as to why i'm getting complaints during the build process saying:
Code:
root@Sandbox:/home/d/8.0.3-RELEASE# ./build/do_build.sh 
grep: /home/d/8.0.3-RELEASE/FreeBSD/src-patches: No such file or directory
Applying patch freebsd-ahci.patch...
patch: invalid option -- 'C'
patch: Try `patch --help' for more information.


Any ideas?

edit: dont listen to anything i said in this post. was running in ubuntu...
 

dpodolak

Cadet
Joined
Jan 13, 2012
Messages
3
What's your output of the following command?

Code:
uname -a


Code:
d@Sandbox:~$ uname -a
Linux Sandbox 3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux




edit: i think i now realize my error...i'm guessing you cant build this from a distro other than freebsd, eh?
 

bryanburke

Dabbler
Joined
Nov 11, 2011
Messages
11
There's your problem right there: Linux. FreeNAS is NOT Linux, nor is it based on Linux. FreeNAS is a derivative of FreeBSD Unix and, as such, requires FreeBSD-specific utilities to build. Please see the official FreeBSD Handbook (http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/) for instructions on installing FreeBSD 8. Installation and maintenance of a FreeBSD system are outside the scope of this how-to.
 

dpodolak

Cadet
Joined
Jan 13, 2012
Messages
3
There's your problem right there: Linux. FreeNAS is NOT Linux, nor is it based on Linux. FreeNAS is a derivative of FreeBSD Unix and, as such, requires FreeBSD-specific utilities to build. Please see the official FreeBSD Handbook (http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/) for instructions on installing FreeBSD 8. Installation and maintenance of a FreeBSD system are outside the scope of this how-to.

Gotcha...thanks for the explanation. I guess I made some incorrect assumptions :o

On another note, I successfully finished building FreeNAS with customizations to add Transmission; however when I try to load a VM with my build iso, I'm given an acpi error when the VM is trying to start up. I've attached a screenshot of the last view i see...
acpi_error.jpg

I'm fairly certain I followed the guide verbatim, but I guess I missed something. Any ideas?
 

deka

Cadet
Joined
Jan 30, 2012
Messages
4
ADDENDUM

--------------------------------------------

How to fix the libSM and libICE packaging error:

If you have already built FreeNAS from source and would like to add more custom ports, you do not have to rebuild from scratch. However, if you issue the "build/do_build.sh" command again, the process will stop and complain about missing packages for libSM and libICE. This appears to be an error in either the FreeNAS or nanoBSD build system, but the fix is simple enough.
This is not the only problem -- trying to rebuild FreeNAS again you'll got other errors: from cairo, pango and so on.

I'd learned this problem reading scripts carefully.

The logic of add_port next:
1. first, it checks <freenas-src-root>/obj.<arch>/ports/packages for existing package;
2. then, if package not exists, it builds port, makes package and places it into <freenas-src-root>/obj.<arch>/ports/packages for future use, it dramatically reduces time for rebuilds FreeNAS, because:
3. if package exists, it add package by pkg_add, not building port again!

Further, script builds port in "virtual" environment. So, when any dependency must be resolved, necessary ports also builds, but, because package builds by "make package", not "make package-recursive", only package of port itself will built, and depending packages will lost. As a result, when you try to rebuild FreeNAS, you can get "build error", if package tries to add before all required packages will also add. As an example, if you'll use code above from your message, you'll build FreeNAS, but when you try to rebuild it again you'll meet such problem when adding libSM, because it depends on libICE.

There is two solutions.

First, you can try to patch build scripts to use "make package-recursive" instead of "make package" inside <freenas-src-root>/nanobsd/common (do_add_port() shell function). This is quite bad because after updating FreeNAS sources your changes will lost. Further, script warning in comments, that "recursive build not supported", so you can meet another problems with it.

Second, you can sort add_port lines in nanobsd/freenas-common so that first build/add pre-requisite packages, then other ones. I searched script outputs (_.cust*) for string "depends.*not found", then explicitly added all unresolved dependencies, and got that any build after it completes without any error.

Here is fragment of my nanobsd/freenas-common with standard FreeNAS packages, after it was sorted and all missing dependencies was added:

Code:
add_port devel/libtool
add_port converters/libiconv
add_port converters/iconv
add_port devel/gettext
add_port devel/gmake
add_port archivers/unzip
add_port lang/perl5.12
add_port devel/autoconf-wrapper
add_port devel/automake-wrapper
add_port devel/autoconf
add_port devel/automake
add_port devel/m4
add_port devel/bison
add_port misc/help2man
add_port devel/p5-Locale-gettext
add_port devel/pkg-config
add_port devel/pcre
add_port devel/gperf
add_port textproc/p5-XML-Parser
add_port textproc/intltool
add_port lang/tcl85
add_port lang/tcl-modules
add_port lang/ruby18
add_port lang/python27 -DWITH_HUGE_STACK_SIZE
add_port security/libgpg-error
add_port security/libgcrypt
add_port x11/kbproto
add_port x11/xproto WITHOUT_X11=y
add_port x11/libXdmcp WITHOUT_X11=y
add_port x11/libXau WITHOUT_X11=y
add_port devel/libpthread-stubs WITHOUT_X11=y
add_port devel/libcheck
add_port textproc/libxml2
add_port textproc/libxslt
add_port x11/xcb-proto -DWITHOUT_X11
add_port x11/libxcb WITHOUT_X11=y
add_port devel/xorg-macros
add_port x11/libICE
add_port x11/libSM
add_port x11/bigreqsproto
add_port x11/xcmiscproto
add_port x11/xextproto
add_port x11/xtrans
add_port x11/inputproto
add_port x11-fonts/xf86bigfontproto
add_port x11/libX11
add_port x11-fonts/fontsproto WITHOUT_X11=y
add_port x11-fonts/libXfont
add_port x11-fonts/bdftopcf 
add_port devel/cdialog
add_port dns/inadyn			# -mt?
add_port editors/nano
add_port ftp/proftpd
add_port net-mgmt/bsnmp-ucd
add_port net-mgmt/bsnmptools
add_port net-mgmt/clog
add_port net-mgmt/sipcalc
add_port net/istgt
add_port databases/db46
add_port security/cyrus-sasl2
add_port net/openldap24-sasl-client
add_port net/nss_ldap
add_port net/rsync WITHOUT_POPT_PORT=true WITH_SSH=true WITHOUT_ATIMES=true \
add_port dns/py-dnspython
add_port devel/talloc
add_port devel/libexecinfo
add_port devel/popt
add_port databases/tdb
add_port sysutils/libsunacl
add_port net/samba36	WITHOUT_CUPS=y WITH_LDAP=y WITH_ADS=y WITH_WINBIND=y \
add_port security/pam_ldap
add_port security/pam_mkhomedir
add_port shells/bash WITHOUT_NLS=y
add_port shells/scponly
add_port sysutils/e2fsprogs
add_port textproc/ruby-deplate
add_port sysutils/fusefs-libs
add_port sysutils/fusefs-kmod
add_port devel/libublio
add_port sysutils/fusefs-ntfs
add_port sysutils/ntfsprogs
add_port sysutils/smartmontools
add_port devel/glib20
add_port net/libdnet
add_port emulators/open-vm-tools-nox11 WITHOUT_ICU=y
add_port databases/sqlite3
add_port databases/py-sqlite3
add_port databases/py-bsddb3
add_port www/py-django WITHOUT_MYSQL=y
add_port www/py-dojango
add_port devel/py-setuptools
add_port databases/py-south
add_port devel/py-asn1
add_port devel/py-asn1-modules
add_port www/spawn-fcgi
add_port www/lighttpd
add_port www/py-flup
add_port net-mgmt/net-snmp
add_port sysutils/nut -DWITHOUT_NEON
add_port devel/libstatgrab -DWITHOUT_X11 # run dep of collectd
add_port devel/libltdl -DWITHOUT_X11 # run dep of collectd
add_port print/freetype2 -DWITHOUT_X11 # run dep of collectd
add_port textproc/expat2 -DWITHOUT_X11 # run dep of collectd
add_port x11-fonts/fontconfig -DWITHOUT_X11 # run dep of collectd
add_port x11-fonts/libfontenc -DWITHOUT_X11 # run dep of collectd
add_port x11-fonts/mkfontscale -DWITHOUT_X11 # run dep of collectd
add_port x11-fonts/mkfontdir -DWITHOUT_X11 # run dep of collectd
add_port x11-fonts/font-bh-ttf -DWITHOUT_X11 # run dep of collectd
add_port x11-fonts/font-misc-meltho -DWITHOUT_X11 # run dep of collectd
add_port x11-fonts/font-misc-ethiopic -DWITHOUT_X11 # run dep of collectd
add_port x11-fonts/bitstream-vera -DWITHOUT_X11 # run dep of collectd
add_port x11-fonts/font-util -DWITHOUT_X11 # run dep of collectd
add_port x11-fonts/encodings -DWITHOUT_X11 # run dep of collectd
add_port x11-fonts/xorg-fonts-truetype -DWITHOUT_X11 # run dep of collectd
add_port x11/xcb-util -DWITHOUT_X11 # run dep of collectd
add_port x11/xcb-util-renderutil -DWITHOUT_X11
add_port x11/renderproto
add_port x11/libXrender
add_port devel/gamin -DWITHOUT_X11 # run dep of collectd
add_port devel/gio-fam-backend -DWITHOUT_X11 # run dep of collectd
add_port x11/pixman -DWITHOUT_X11 # run dep of collectd
add_port graphics/png -DWITHOUT_X11 # run dep of collectd
add_port graphics/cairo -DWITHOUT_X11 # run dep of collectd
add_port devel/libffi -DWITHOUT_X11 # run dep of collectd
add_port devel/gobject-introspection -DWITHOUT_X11 # run dep of collectd
add_port x11-toolkits/pango -DWITHOUT_X11 # run dep of collectd
add_port textproc/py-libxml2 -DWITHOUT_X11 # run dep of collectd
add_port databases/rrdtool -DWITHOUT_X11 -DWITHOUT_PERL_MODULE -DWITH_PYTHON_MODULE # run dep of collectd
add_port net/liboping -DWITHOUT_X11 # run dep of collectd
add_port net-mgmt/collectd -DWITHOUT_BIND -DWITHOUT_SNMP -DWITHOUT_X11
add_port devel/py-ipaddr
add_port converters/base64
add_port emulators/mtools -DWITHOUT_X11
add_port sysutils/arcconf
add_port sysutils/tw_cli
add_port net/py-ldap2 -DWITH_SASL
add_port sysutils/ataidle
add_port sysutils/gnome_subr
add_port devel/dbus
add_port devel/dbus-glib
add_port devel/libdaemon
add_port databases/gdbm
add_port net/avahi-app
add_port net/avahi-libdns
add_port textproc/py-xml
add_port sysutils/throttle
add_port sysutils/dmidecode
add_port devel/libevent
add_port sysutils/tmux
add_port benchmarks/iozone 
add_port net/netatalk WITH_PAM=y WITHOUT_APPLETALK=y
add_port dns/libidn
add_port ftp/wget
add_port benchmarks/iperf
add_port sysutils/ipmitool


NB: I did not add any build options and defines for now, maybe I'll do it some later. But keep in mind: when you decides to change build options/defines you have to remove affected package(s) from obj.<arch>/ports/packages.

So, this is applies to new added packages too. If you adds any new port, keep in mind that it depends on some other ports, which may be not built yet, so you have to add not only port itself, but also all required ports, in proper order (first required ports, and then ports which requires its).

And, last remark: size of ramdisk for /etc on production system too small. If you plans add some packages which requires config files in /etc or /usr/local/etc, you should increase size of ramdisk for /etc in build/nanobsd/nanobsd.sh (e.g., NANO_RAM_ETCSIZE=16384 -- corresponds to size of 8MB). If not, you risk to get error "cannot update /etc: file system is full" during startup.
 

deka

Cadet
Joined
Jan 30, 2012
Messages
4
IMPORTANT: Please read the guide completely to avoid confusion and prevent errors!

Step 5: Make Your Changes Persist Between Reboots (Optional)

Once your FreeNAS installation has booted and you have configured it to your liking (or restored your previous configuration), you can modify "/etc/rc.conf" to start ports you added in step 3 at boot (like transmission-daemon) and pass arguments to their RC scripts (such as configuration directories, etc.). To do this, use the following commands from the FreeNAS command-line (must be logged in as root):

Code:
mount -uw /
cd /conf/base/etc/
nano -w rc.conf


Make your changes to the file as required. For example, I have the following inserted for transmission-daemon:

Code:
transmission_enable="YES"
transmission_conf_dir="/mnt/zfs_volume/config/transmission"
transmission_user="root"


Refer to each port's documentation for applicable RC options.

Now reboot your FreeNAS installation and your changes will be made!

You'll keep configuration until you'll decide to make "firmware update", I think.

Maybe better place to make persistent configs is nanobsd/Files/etc/rc.conf, so it will be included after rebuild.
 
Status
Not open for further replies.
Top