** Updated 2022-07-26: Now for iocage jail on FreeNAS 13 **

I wanted to install a custom build of ffmpeg on my FreeNAS to do long transcoding tasks and not tie up my laptop for days at a time.  It took three days to figure it out as there were a lot of hurdles to overcome (and some peculiarities of FreeBSD), so I thought I would share the process for the one or two other people in the world who want to do this ;).  I've written it assuming knowledge of basic FreeNAS tasks like accessing by SSH, but for someone who has never used a jail.  I'm really starting to appreciate jails because, when I screwed things up (many times), I just deleted the jail and started over.  No harm to the main FreeNAS or other jails.

FFmpeg is a powerful, versatile media editing program, used in products like Plex, Kodi, and VLC.  It is often used in servers for on-the-fly transcoding to serve videos, but the use intended here is remuxing/transcoding to produce high-quality, smaller files with desired container format and codecs.  This installation process is set up to build it with few very good encoders (fdk_aac for audio and x264 and x265 for video), but you can build it with almost anything.  It builds the current master source of ffmpeg, which is needed to get support for the most part.

Newer applications and libraries seem to always add new build tools, dependencies and methods.  This is completely updated and uses only the built-in compiler clang.  To make it simpler and easier to maintain, I dropped libass as part of the build.  In all the years I have been building ffmpeg with libass, I have never once used it, but it is a royal pain with all the endless dependencies. However, I am keeping the libraries freetype (gets installed with fontconfig), fontconfig, and fribidi, because they allow full use of the drawtext filter, which is handy if you want to put timestamps or any other text on the video.


First, in SSH, make a jail without a plugin.  Jail gets its own IP address; edit for your network:
sudo iocage create --name "ffmpeg" \
--release 13.1-RELEASE \
ip4_addr="igb0|192.168.0.105/24" \
boot="on"   

Then enter the jail:
sudo iocage console ffmpeg
############################
# FIRST TIME ONLY 
############################
pkg install curl git mercurial yasm nasm gcc bash cmake gmake autoconf autotools fontconfig fribidi rsync
# Make bash the default shell: 
chsh -s /usr/local/bin/bash
# If you want to verify this works, 'exit', then reenter the jail as above.
# Then enter 'ps -p $$' and see the shell name.
# Make a symbolic link to bash in /bin so shell scripts can find it:
ln -s /usr/local/bin/bash /bin/bash
# We want to set stuff that gets read whenever you login.  Type 'ee /root/.profile' and add in these lines:
[CODE]
# This gives you an alias 'll' for a full long directory listing with human-readable sizes
alias ll='ls -lahp'
# These set clang as default compiler for most things.
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++PATH
# This lets your system find the new man pages
export PATH=$PATH:/usr/local/share
# This allows ffmpeg ./configure to find things through pkg-config
export PKG_CONFIG_PATH=/usr/local/lib:/usr/local/lib/pkgconfig:/usr/local/libdata/pkgconfig
[/CODE]
Save the file (Esc, Enter, Enter).  Read that file in so it takes effect now by typing [CMD]source .profile[/CMD] (All first time only)
############################
# End first time 
############################

JAIL OS UPDATE/UPGRADE:
# Might want to update or upgrade the jail OS to get the latest packages.
# This stuff takes a long time; wait with patience to let each finish.
# First, you should UPDATE to latest patch of the current release.  Can do this in GUI, or SSH:
# Via SSH, Not inside jail:
sudo iocage update ffmpeg

# You may be able to upgrade to a later release.
# To see releases available:
sudo iocage fetch
# You choose a release then - it wont let you choose later ones than
# the main system has.  Otherwise it will download the one you choose.  Then
# UPGRADE to it:
sudo iocage upgrade ffmpeg -r 13.1 [don't add -U1.1 or such]
# Finally, UPDATE the new release as above:
sudo iocage update ffmpeg

# UPDATE/UPGRADE PACKAGES
# Enter the jail:
sudo iocage console ffmpeg
# Instead of pkg install, just update/upgrade:
pkg update   # updates pkg list (say yes if asked)
pkg upgrade  # updates all installed pkgs


Now we're ready to download source files and start building them.

# Packages will stay unmolested in first directory (PAKS),
# will be copied to second directory for building (BUILD),
# installed in third directory (TARGET), then only ffmpeg 
# gets moved to /usr/local/bin
# You can delete BUILD and TARGET and start fresh each time if you like
# This directory variables must be made each time you enter the jail
PAKS=/usr/local/ffmpeg/packages
BUILD=/usr/local/ffmpeg/build
TARGET=/usr/local/ffmpeg/target
# NEED THIS SO SYSTEM can find new binaries like pkg-config
export PATH=$PATH:${TARGET}/bin

### First time only:

# mkdir /usr/local/ffmpeg ${PAKS} ${BUILD} ${TARGET}

### Later times if you choose to delete these directories after building:
### (or you could just leave TARGET and BUILD and write over them with updates)
# rm -R ${BUILD} ${TARGET}
# mkdir ${BUILD} ${TARGET}

# Some earlier libraries interfere (if present) so nuke them later times
rm /usr/local/lib/libavcodec* /usr/local/lib/libx2*

##############################
#  Create/Update local repositories of ffmpeg and dependencies
##############################

cd $PAKS

### fdk-aac
# Cool: This tells git to pull in the fdk-aac directory, or if there is an error (||), clone a new one.
git -C fdk-aac pull 2> /dev/null || git clone --depth 1 https://github.com/mstorsjo/fdk-aac
# Copy it (rsync) to $BUILD.  -a is archive mode (recursive, copy symlinks, preserve permissions, preserve mod. times,  preserve owner/group) --delete deletes other files on receiving side.
rsync -a --delete $PAKS/fdk-aac/ $BUILD/fdk-aac/

### pkg-config
git -C pkg-config pull 2> /dev/null || git clone https://anongit.freedesktop.org/git/pkg-config
rsync -a --delete $PAKS/pkg-config/ $BUILD/pkg-config/

### x264
git -C x264 pull 2> /dev/null || git clone http://git.videolan.org/git/x264.git
rsync -a --delete $PAKS/x264/ $BUILD/x264/

### x265
# if cd x265 2> /dev/null; then hg pull -u && cd ..; else hg clone https://bitbucket.org/multicoreware/x265; fi
# 2021-07-23 hg pull --update did not update to the latest branch.  Had to go in
# and update to it manually "hg update e7d0e439852a".  Instead, this works:
# It may give "abort" message first time but still works:
hg pull --verbose --update --repository x265 || hg clone http://hg.videolan.org/x265 x265
rsync -a --delete $PAKS/x265/ $BUILD/x265/

### ffmpeg
git -C ffmpeg pull 2> /dev/null || git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg
rsync -a --delete $PAKS/ffmpeg/ $BUILD/ffmpeg/

#######################
#  FDK-AAC
#######################

cd $BUILD/fdk-aac

# This gets built as libraries and includes only.
# Uh-oh, no configure file anymore!
# ./configure --prefix=$TARGET --disable-shared && make install clean
# Instead, this works from ffmpeg wiki/CompilationGuide:
autoreconf -fiv && \
./configure --prefix="$TARGET" --disable-shared && \
make && \
make install

#######################
#  PKG-CONFIG
#######################

# 2022 July added "--enable-iconv=no --with-libiconv=gnu" to avoid build error "gconvert.c:61:2: error: GNU libiconv not in use but included iconv.h is from libiconv"

cd $BUILD/pkg-config
./autogen.sh --with-internal-glib --enable-iconv=no --with-libiconv=gnu --prefix="$TARGET" --exec-prefix="$TARGET"
make install clean

#######################
#  X264
#######################

cd $BUILD/x264

./configure --prefix=$TARGET --enable-static --enable-pic

# De novo on 2022-02-20 got configure error, gcc not installed.
# Did "pkg install gcc" and it works again.  ??

gmake  # make suggested but doesn't work
gmake install clean

#######################
#  X265
#######################

cd $BUILD/x265/build/linux

	#######################
	#  Editing multilib.sh
	#######################
#  multilib.sh is a script that will build x265 with 8-, 10-, and 12-bit capability.
#  We have to specify the clang compiler; this process pays no attention to the
#  variables we set before.  Also set the install prefix. 
#  The only way I know how is to edit the script.
#  See https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu
#  So 'ee multilib.sh' 

#  Under 'cd 12bit', after 'cmake', insert with no line breaks:
 -D CMAKE_C_COMPILER=/usr/bin/clang -D CMAKE_CXX_COMPILER=/usr/bin/clang++ 

# Under 'cd ../8bit', after 'cmake', insert
 -DCMAKE_INSTALL_PREFIX=/usr/local/ffmpeg/target/ 

# on same line, after ../../../source, add
 -DENABLE_SHARED=OFF 

# Add '-j #' to all three make commands, where # is the number of processor cores you have.

# Near end of the script is an if/then.  It results in using libtool, but the libtool we have doesn't work as expected.  The alternative, GNU ar, does work.  So either comment out the if/then/else, or make the 'if' have an OR to recognize Linux OR FreeBSD.  That would look like this:
if [ "$uname" = "Linux" ] | [ "$uname" = "FreeBSD" ]
# Save (escape, enter, enter)
# Rename and save the edited script in case $BUILD or $BUILD/x265 are deleted
mv multilib.sh multilib_edited.sh
cp -iv multilib_edited.sh $PAKS/

	#######################
	#  Continue
	#######################
# To re-use a previously saved, edited script:
cp -iv $PAKS/multilib_edited.sh $BUILD/x265/build/linux/

# Then release the dogs.  This one takes quite a while.  Smoke 'em if you got 'em. ;-)
./multilib_edited.sh

cd 8bit  # the complete build gets put here
make install

# to remove x265 from target (if there were problems): find $TARGET -name '*x265*' -delete

#######################
#  FFMPEG
#######################

cd $BUILD/ffmpeg

export  CFLAGS="-I${TARGET}/include -I/usr/local/include -I/usr/include"
export  LDFLAGS="-L${TARGET}/lib -L/usr/local/lib -L/usr/lib"

# Have to examine ffbuild/config.log after failure.  Error message might show one thing
# not found, but log shows something else not found that causes the error.
# Font stuff is problematic (eg fontconfig, freetype, fribidi).
# freetype2 also requires zlib whose pc file is in another directory, so this finally worked
# and to help pkg config find the files it needs in our TARGET:
export PKG_CONFIG_PATH=/lib:/usr/lib:usr/local:/usr/local/lib:/usr/local/lib/pkgconfig:/usr/local/libdata/pkgconfig:/usr/libdata/pkgconfig:$TARGET/lib:$TARGET/lib/pkgconfig:$TARGET/share/pkgconfig


# Testing pkg-config problems:
# to see default pkg-config paths: pkg-config --variable pc_path pkg-config
# to see ones added with environment variable: echo $PKG_CONFIG_PATH
# to see if the pc can be found: 
pkg-config --libs --cflags x265 # OR 
pkg-config --modversion fontconfig

# removed --disable-ffserver
# removed --disable-indevs

# added --extra-libs=-lpthread # now needed for libx265.a
# added  (WHY?) --enable-libzvbi , not sure if I need to build the lib  # --enable-libzvbi \


# ./configure prefix=$TARGET --cc=/usr/bin/clang --extra-cflags="-I$TARGET/include" \
# --extra-ldflags="-L$TARGET/lib" --extra-libs=-lpthread \
# --pkg-config-flags="--static" --enable-static --disable-shared \
# --enable-libfdk-aac --enable-libx264 --enable-libx265 \
# --enable-nonfree --enable-gpl --enable-version3 --enable-hardcoded-tables \
# --enable-avfilter --enable-filters --disable-outdevs --disable-network 


./configure --cc=/usr/bin/clang \
--prefix=$TARGET \
--pkgconfigdir="$TARGET/lib/pkgconfig" \
--pkg-config-flags="--static" \
--extra-cflags="-I/usr/local/include" \
--extra-ldflags="-L/usr/local/lib" \
--extra-libs="-lpthread -ldl -lstdc++" \
--enable-static --disable-shared \
--enable-nonfree --enable-gpl --enable-version3 \
--enable-libfribidi --enable-libfreetype --enable-libfontconfig \
--enable-libfdk-aac --enable-libx264 --enable-libx265 --disable-lzma 

gmake -j 8
gmake install

# REAL INSTALL (don't need the man3 docs)
cp -v ${TARGET}/bin/* /usr/local/bin
cp -v ${TARGET}/share/man/man1/* /usr/local/share/man/man1

# Also so pkg-config can easily find the library in future if you rebuild ffmpeg but x265 hasn't changed.
# cp -v ${TARGET}/lib/libx265* /usr/local/lib

# Take TARGET out of PATH to test
PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin:/usr/local/share"

[/CODE]
You can establish storage for ffmpeg (a folder for ffmpeg to do it's work).  I call my storage folder FFstor outside the jail, ffstor inside.  First create the folder outside the jail the usual way, in whatever dataset you want.
Then, in WebGUI > Jails, select 'Mountpoints' in the ffmpeg options menu; 
Source: /mnt/<Pool>/<Dataset>/FFstor.  (for me: /mnt/Ark/Media/FFstor)
Destination: /media/ffstor (inside the jail).  
Submit.

Actually using ffmpeg is a much bigger learning process, and involves learning a lot about video, but here's a command to get you started.  It takes a big old movie file called "in.wmv" using crappy old codecs.  It converts the video to x265 (the latest and most efficient codec, though not in terms of encoding time or decoding horsepower required) at a bit rate of 2750 kB/second (enough for 1080p), converts the audio to fdk-aac (the best and most versatile aac encoder) at 256 kB/second (plenty for stereo; you might want to double it for surround sound), and outputs it in a mp4 container.  It uses 2-pass encoding for the video, giving better quality than a single pass, and uses the default medium (speed) preset.  For a 2-hour 1080p movie, this takes about a day and a half on my low-powered processor.  Your mileage will probably be better.

[CODE]
cd /media/ffstor
ffmpeg -i in.wmv -an -sn -c:v libx265 -preset slow -b:v 3000k -x265-params pass=1 -f m4v -y /dev/null && \
ffmpeg -i in.wmv -c:a copy -c:s copy -c:v libx265 -preset slow -b:v 3000k -x265-params pass=2 -y out2.m4v [/CODE]