IPFW Not Logging

Brad1976

Dabbler
Joined
Mar 31, 2015
Messages
40
Problem: IPFW seams to not be logging anything within my Transmission jail. I'm currently using FreeNAS 11.2-U6 and my iocage jail has been created with the 11.2-RELEASE. I'm trying to use the logs to troubleshoot a seperate issue I'm having with Transmission Remote GUI (installed on my laptop) not connecting through a PIA openvpn connection... But, it appears that this is a roadblocking issue just getting the ipfw logs to work in itself! I'll probably give you too much info, but here's what I have...

Code:
# $FreeBSD: releng/11.2/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0

#Emable ipfw logging
net.inet.ip.fw.verbose=1
#Limit to 5 lines of activity from ipfw
net.inet.ip.fw.verbose_limit=5
Code:
##########################################################################################
#   IPFW RULES
##########################################################################################
#!/bin/bash

# Flush out the list before we begin
ipfw -q -f flush

# Set  variables
CMD="ipfw -q add"                            # too lazy to type this out every time
PIF="epair0b"                                 # interface name of NIC attached to Internet
VPN="tun0"                                    # interface name for openvpn PIA connection

# Retrieve the openvpn port number created by transmission-port-forward.sh
VPNPORT=$(cat "/myscriptlogs/ipfw-openvpn.port")
#echo $VPNPORT            #just to test

#sample command
#add 01000 allow log udp from 192.168.1.0/24 to 184.75.220.106 dst-port 53 keep-state

# allow all local traffic on the loopback interface
$CMD 00001 allow log tcp from any to any via lo0

# allow any connection to/from VPN interface
$CMD 00010 allow log tcp from any to any via $VPN

# allow anything incoming on the vpn with the port number
$CMD 00011 allow log tcp from any to any via $VPN

# allow connection to/from LAN by Transmission
$CMD 00101 allow log tcp from me to 192.168.2.1/24 uid transmission
$CMD 00102 allow log tcp from 192.168.2.1/24 to me uid transmission

# deny any Transmission connection outside LAN that does not use VPN
$CMD 00103 allow log tcp from any to any uid transmission
Code:
root@Transmission:/ # ipfw list
00001 allow log logamount 5 tcp from any to any via lo0
00010 allow log logamount 5 tcp from any to any via tun0
00011 allow log logamount 5 tcp from any to any via tun0
00101 allow log logamount 5 tcp from me to 192.168.2.0/24 uid transmission
00102 allow log logamount 5 tcp from 192.168.2.0/24 to me uid transmission
00103 allow log logamount 5 tcp from any to any uid transmission
65535 allow ip from any to any
Code:
Dec 11 02:00:48 Transmission newsyslog[77158]: logfile first created
Code:
#!/usr/bin/env bash
#
# Enable port forwarding when using Private Internet Access
# install the following in order to run
#  - pkg install curl
#  - pkg install bash
#  - pkg install jq
#  - pkg install -y p5-Digest-SHA
#
# Usage:
#  ./usr/local/etc/openvpn/transmission-port-forward.sh

#Declare the variables for the script
TRANSUSER=transuser
TRANSPASS=transpass
TRANSHOST=192.168.2.xxx

error( )
{
  echo "$@" 1>&2
  exit 1
}

error_and_usage( )
{
  echo "$@" 1>&2
  usage_and_exit 1
}

usage( )
{
  echo "Usage: `dirname $0`/$PROGRAM"
}

usage_and_exit( )
{
  usage
  exit $1
}

version( )
{
  echo "$PROGRAM version $VERSION"
}


port_forward_assignment( )
{
  #Loading port forward assignment information
  if [ "$(uname)" == "Linux" ]; then
    client_id=`head -n 100 /dev/urandom | sha256sum | tr -d " -"`
  fi
  if [ "$(uname)" == "FreeBSD" ]; then
       client_id=`head -n 100 /dev/urandom | shasum -a 256 | tr -d " -"`
  fi

  #Retreive port information
  json=`curl "http://209.222.18.222:2000/?client_id=$client_id" 2>/dev/null`
 
  #What to do if port forward is already active or if connection has failed
  if [ "$json" == "" ]; then
    
    #Write to the log file as follows
    #Uncomment this line for troubleshooting while working on stuff as needed.....
    #echo As of $(date) port forwarding is already activated on the connection or has expired. Restart the jail... >> /usr/local/etc/openvpn/transmission-port-forward.log 2>&1
    
    exit 0
  fi

  #Show the returned port number
  echo server returned "$json"
 
   #trim VPN forwarded port from JSON
    PORT=$(echo $json | awk 'BEGIN{r=1;FS="[{}\":]+"} /port/{r=0; print $3} END{exit r}')
    echo if successful, trimmed port is:"$PORT"

    #Get the openvpn WAN IP Address
    OPENVPNIP=`curl "ifconfig.me" 2>/dev/null`
    #json=`curl "http://209.222.18.222:2000/?client_id=$client_id" 2>/dev/null`

    #write the port number to a log file to me to quickly keep tabs on
    echo As of $(date) the transmission-remote access is "$OPENVPNIP:$PORT" >> /myscriptlogs/transmission-port-forward.log 2>&1
    
    #write the port number only to a file for use by ipfw then restart ipfw to load the new port number
    echo $PORT > /myscriptlogs/ipfw-openvpn.port 2>&1
    
    #change transmission port on the fly
    transmission-remote $TRANSHOST --auth $TRANSUSER:$TRANSPASS -p "$PORT"
    echo "Transmission-remote updated successfully" >> /myscriptlogs/transmission-port-forward.log 2>&1
    #echo remember to run no longer than 2 mins after reconnecting/connecting to vpn server.
}

EXITCODE=0
PROGRAM=`basename $0`
VERSION=2.1

while test $# -gt 0
do
  case $1 in
  --usage | --help | -h )
    usage_and_exit 0
    ;;
  --version | -v )
    version
    exit 0
    ;;
  *)
    error_and_usage "Unrecognized option: $1"
    ;;
  esac
  shift
done

port_forward_assignment

exit 0

I have read the FreeBSD HandbookIPFW page and I everything seams right. I have also read through the ipfw(8) man page and still can't seam to figure out why the logs are not working... The results of the ipfw logs are supposed to be put into the /var/log/security file, but as per what I shared above, that's the only contents that has ever appeared in that file... Any help on what I've posted, or even pointing me int he right direction would be GREATLY appreciated!!! (I can't believe that this late into the game there could still possibly be a bug here...) Or, if there's any questions, I should be able to get online and answer anything every evening sometime (EST).

Thanks again all!!!
 

PnoT

Dabbler
Joined
Apr 12, 2017
Messages
41
Did you ever figure this out as I'm running into the same issue currently? I don't think VNET in the iocage jail system supports some of the features that you see described in the FreeBSD manual.
 
Last edited:

Brad1976

Dabbler
Joined
Mar 31, 2015
Messages
40
Unfortunately, I got absolutely nowhere with this... As such, because I have zero logs to diagnose I have also not been able to figure out why I am not able to get the port forwarding via openvpn and PIA to work and I'm not able to access transmission using the remote gui at all... My only work around is to have a separate Windoze computer on the local network where my server is located and then remote into that using a Windoze remote desktop called Radmin. It's not preferred, but it gets the job done...

I have recently moved on 28 March. My better half misplaced my special bag with a bunch of ziplocks with all my small parts for a few pieces of furniture... Tomorrow I will be picking up some replacement M5 nuts and bolts so I can mount everything back into the server rack, then get the rest of my office furniture put together... Other then starting a 7 day stretch on duty with work on Saturday, I am fully hoping to try and get back to this issue towards the end of the month unless anyone else if able to chime in with something to consider... <enter sound of crickets here...>

I'm also considering upgrading to 11.3-U1 as I'm currently on 11.2-U8, but I'm not quite holding my breath... (I'm also having a separate unrelated issue where the FreeNAS graph on the main gui is flat lined for the cpu load, yet the cup load from within my PlexMediaServer is working fine! Hoping 11.3 will fix that as well as the logging issue this thread is about, but only time will tell! lol
 

PnoT

Dabbler
Joined
Apr 12, 2017
Messages
41
In looking over the iocage documentation there isn't anything listed in it, concerning ipfw, that talks about logging. Only how to enable it.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
You did enable logging with firewall_logging="YES" in /etc/rc.conf in your jail? And then add explicit log statements to the rules you want logged?
 

PnoT

Dabbler
Joined
Apr 12, 2017
Messages
41
You did enable logging with firewall_logging="YES" in /etc/rc.conf in your jail? And then add explicit log statements to the rules you want logged?

Definitely have that in my config and have multiple entries in my ipfw_rules that have "log" in them.

firewall_enable="YES"
firewall_logging="YES"
firewall_script="/home/testuser/ipfw_rules"

add_fw deny log ip from any to any

Do you have it working?
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
No, but we do have a couple of hundred jails with IPFW so I can try some things tomorrow. Just wanted to check the obvious first.

Kind regards,
Patrick
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Are you running syslogd in your jail? It's syslogd that listens to the kernel interface and writes to the userland file.
 

PnoT

Dabbler
Joined
Apr 12, 2017
Messages
41
Are you running syslogd in your jail? It's syslogd that listens to the kernel interface and writes to the userland file.

I don't have that installed. In looking at a quick pkg search there's more than a few syslogd replacements but not one that is specifically syslogd.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
It's in the base system, so already in every jail. And it is started by default.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
It's in the base system, so already in every jail. And it is started by default.

syslogd being in the base system isn't equivalent to it being in every jail. It is important to remember that a jail isn't a fully virtualized FreeBSD instance. It is simply several namespace abstractions within the base system's kernel. Things such as ipfw which are handled within the system kernel don't always work exactly as expected. It's easy to lose sight of this as things like vnet allow you to push stuff at the kernel from within a jail.

The work I do with jails isn't within the FreeNAS framework, so I don't have much clarity on how this might be working on a FreeNAS host. I've been building highly secure jails on FreeBSD basically since phk introduced the feature, though, so I'll talk about general syslogd stuff for a few moments to see if it gives any insight or an aha moment.

So the thing is, the kernel interface exists in the base system, so when you set "log" on an ipfw rule, the output of that is fed to the host system's syslogd via /dev/klog alongside other host system kernel messages.

/dev/klog is available within jails. I'm 99.9% sure positive this didn't used to be the case, so I suspect it was added to silence errors from utilities like syslogd which have it hardwired. If that's the case, I expect it isn't getting the kernel message stream, which would have to be duplicated within the kernel to each instance, and that could get problematic on a large host with many jails. Plus the expected privacy/security issues. Perhaps someone cleverly addressed this within vnet, or through jail tagging on the ipfw rule, I wouldn't be shocked if so, but I expect it's probably just a dead stump of a device.

So the first thing to do is to see if the messages you're wishing for are showing up in the base system's /var/log/security file. Once you find that the messages you seek are showing up there, you can determine whether you are fine with them being there, or if you need to find a trick of some sort to forward them back into the jail.

If the jail is already running its own syslogd, which is a totally reasonable thing to do in many cases, you could forward the security.* stream to the jail syslogd using the @ directive in the base's syslogd.conf. You need to remove the -s from the jailed instance's syslogd flags.

If the jail isn't running its own syslogd, you could arrange for the base system to log security.* into /your/jail/path/var/log/security but I think this probably wouldn't persist well in FreeNAS.

Anyways, @Patrick M. Hausen if you discover more specific capability within jails to handle this, I'm interested in hearing about it. I've not really had a use for any of the "iocage" or "vnet" add-ons that try to fix perceived deficiencies in jails. I create jails from the ground up with just the packages needed (not FreeBSD packages or ports - from source) so that there's nothing rattling around inside a jail like /bin/sh that could easily be exploited. This is highly compatible with phk's original vision for jails. But I do like to understand what's evolving in the base system...
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
@jgreco You are of course technically correct - and I was not very precise. So let me rephrase:

Every jail created by the method described in the jail(8) manpage and every jail created by the FreeNAS UI contains a complete FreeBSD installation minus kernel.

Which is precisely what I use jails for - virtual FreeBSD hosts. But there are other use cases as you said. I'll look into this IPFW logging thing later, but I will use (and assume) a complete jail with all the userland installed, of course.

Kind regards,
Patrick
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
@jgreco You are of course technically correct - and I was not very precise. So let me rephrase:

Ok so yes then we're close to the same page, your previous post just interpreted poorly. :smile:

If we're looking at a syslogd in a jail, I am not aware of anything that would feed it the particular kernel output relevant to that jail, which means no ipfw log output.

I could be wrong on this, if there's some special magic done at the vnet layer, and I'd be happy to be wrong, because I know FreeBSD scales very well when hosted in jails this way, and klogging would make the illusion more complete. I know that's the general direction they were driving in with vnet.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
OK, first try - partly successful:

1. In the jail

/etc/rc.conf
Code:
firewall_enable="YES"
firewall_type="open"
firewall_logging="YES"


/etc/rc.firewall
change the "open" template entry from
${fwcmd} add 65000 pass all from any to any
to
${fwcmd} add 65000 pass log all from any to any

Restart the jail.

N.B. my jail that I used does not carry production traffic unless I actively produce some. It might not be wise to log every single packet in a production system. This was the fastest way to get a working test environment (I think).

2. On the FreeNAS host

/var/log/security
Code:
[...]
Apr 11 16:01:56 freenas-pmh ipfw: 65000 Accept TCP 192.168.93.50:62751 192.168.93.51:3000 in via epair0b
Apr 11 16:01:57 freenas-pmh ipfw: 65000 Accept TCP 192.168.93.51:3000 192.168.93.50:64683 out via epair0b
Apr 11 16:01:57 freenas-pmh ipfw: 65000 Accept TCP 192.168.93.50:64683 192.168.93.51:3000 in via epair0b
[...]


So fundamentally it works but the logs are written to the host system, probably because (see @jgreco above) there is no /dev/klog for the jail's syslogd to listen to. If this already solves your problem - fine. If you must have the logs in the jail in question, you would now need to toy with devfs rules to get the /dev/klog device inside the jail. Caveat: the logs will probably end up in the jail (if you get it to work) and on the host system. So make sure your system dataset can cope with the amount you are logging ...

HTH,
Patrick
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Well there IS a /dev/klog inside the jail (at least if you mount devfs), but I think it's probably just a null device because syslogd would complain if it couldn't open it. devfs rules typically control access to a device so I don't think you can devfs-rules your way into it, because the device is already there but not providing any data. Hm.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
This is my /dev inside the jail:
Code:
root@gitea:~ # ls -l /dev/klog
ls: /dev/klog: No such file or directory
root@gitea:~ # ls -l /dev/
total 1
crw-------  1 root  wheel     0x13 Apr  7 16:15 bpf
lrwxr-xr-x  1 root  wheel        3 Apr 11 16:20 bpf0 -> bpf
crw-rw-rw-  1 root  wheel     0x64 Apr  7 16:15 crypto
dr-xr-xr-x  2 root  wheel      512 Apr  7 16:15 fd
crw-rw-rw-  1 root  wheel      0xf Apr 11 16:47 null
crw-rw-rw-  1 root  wheel     0x12 Apr  7 16:15 ptmx
dr-xr-xr-x  2 root  wheel      512 Apr 11 16:20 pts
crw-r--r--  1 root  wheel      0x7 Apr  7 16:17 random
lrwxr-xr-x  1 root  wheel        4 Apr 11 16:20 stderr -> fd/2
lrwxr-xr-x  1 root  wheel        4 Apr 11 16:20 stdin -> fd/0
lrwxr-xr-x  1 root  wheel        4 Apr 11 16:20 stdout -> fd/1
lrwxr-xr-x  1 root  wheel        6 Apr 11 16:20 urandom -> random
crw-rw-rw-  1 root  wheel     0x10 Apr  7 16:15 zero
crw-rw-rw-  1 root  operator  0x62 Apr  7 16:15 zfs


And in my experience devfs rules control which devices are present inside the jail ... then just regular Unix access controls apply. E.g. if you map bpf into the jail (like I do) you can tcpdump.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Okay, but are you actually applying a devfs ruleset to that? Because default FreeBSD with devfs mounted and ruleset 0 generates a klog.

Code:
Script started on Sat Apr 11 10:03:41 2020
root@freebsd-121r-i386-lab1:~ # uname -a
FreeBSD freebsd-121r-i386-lab1 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  i386
root@freebsd-121r-i386-lab1:~ # mount -t devfs foo /jail/dev
root@freebsd-121r-i386-lab1:~ # jail /jail jail 127.0.0.1 /bin/tcsh
root@jail:/ # ls -l /dev/klog
crw-------  1 root  wheel  0x28 Apr 11 15:03 /dev/klog
root@jail:/ # exit
root@freebsd-121r-i386-lab1:~ # jls -n
devfs_ruleset=0 nodying enforce_statfs=2 host=new ip4=disable ip6=disable jid=1 name=1 osreldate=1201000 osrelease=12.1-RELEASE parent=0 path=/jail persist securelevel=-1 sysvmsg=disable sysvsem=disable sysvshm=disable vnet=inherit allow.nochflags allow.nomlock allow.nomount allow.mount.nodevfs allow.mount.noprocfs allow.noquotas allow.noraw_sockets allow.noread_msgbuf allow.reserved_ports allow.set_hostname allow.nosocket_af allow.nosysvipc children.cur=0 children.max=0 cpuset.id=2 host.domainname=0"" host.hostid=0 host.hostname=jail host.hostuuid=00000000-0000-0000-0000-000000000000 ip4.addr=127.0.0.1 ip4.saddrsel ip6.addr= ip6.saddrsel
root@freebsd-121r-i386-lab1:~ # exit

Script done on Sat Apr 11 10:03:57 2020


Check using "jls -n" to see if maybe you have a different ruleset being applied. Ruleset 0 is magic, it can't be modified and is always empty.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Of course - iocage applies the special "jail" ruleset to all jails. If we had the /dev/k* devices inside the jails, it would be trivial to break out ...
And don't use jls but iocage get ;)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
iocage is an add-on of some sort, and I really don't care to learn yet another idiotic jail management system that'll be unsupported in a few years. The jailed apps I write do their own jail management and are easily understood. I don't understand the strategy of loading craptons of scripting languages and their modules onto a system in order to do simple things, only to have them break when the next version of some idiot module comes along and your stuff goes T-U. Wrote it for Python 2.7? Oh! Look! Doesn't work on 3.5.

My stuff? Written in sh? (and not bash?) Working fine 20 years later.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
I am strictly speaking about the FreeNAS environment and the jail framework it provides ...
 
Top