Very hard time getting krb5p to work with NFS shares

[Joshua Schaeffer]

Greetings all,

I'm having a very difficult time getting my NFS shares to work with krb5p. Here is what my system looks like:

• Build: FreeNAS-9.10.1-U2 (f045a8b)
• CPU: Intel Xeon E3-1231 v3
• RAM: 32GB DDR3

I've setup a RAID-Z2 on 4x Western Digital 6TB SATA drives for an effective 10.2TB of available space. I have a dataset called "home" and under that I have a child dataset for my username. I then created an NFS share that points to that user's dataset (/mnt/harm/home/jschaeffer). On the share I have krb5p and sys security principals allowed (krb5p first). My entire network has Kerberos setup and working successfully. I have both a host and nfs principal in my /etc/krb5.keytab file on the NFS server, I can connect with kinit without any issues. However, when I try to actually mount my share it always gives me an access denied error. Here are all the actual configuration and commands I'm running:

[root@ultralisk] /var/log# ktutil -k /etc/krb5.keytab list
/etc/krb5.keytab:

Vno Type Principal Aliases
4 aes256-cts-hmac-sha1-96 host/ultralisk.harmonywave.com@HARMONYWAVE.COM
4 arcfour-hmac-md5 host/ultralisk.harmonywave.com@HARMONYWAVE.COM
4 des3-cbc-sha1 host/ultralisk.harmonywave.com@HARMONYWAVE.COM
4 des-cbc-crc host/ultralisk.harmonywave.com@HARMONYWAVE.COM
2 aes256-cts-hmac-sha1-96 nfs/ultralisk.harmonywave.com@HARMONYWAVE.COM
2 arcfour-hmac-md5 nfs/ultralisk.harmonywave.com@HARMONYWAVE.COM
2 des3-cbc-sha1 nfs/ultralisk.harmonywave.com@HARMONYWAVE.COM
2 des-cbc-crc nfs/ultralisk.harmonywave.com@HARMONYWAVE.COM

[root@ultralisk] /var/log# cat /etc/exports
V4: / -sec=sys:krb5:krb5i:krb5p
/mnt/harm/home/jschaeffer -sec=krb5p:sys -network 10.1.0.0/16
/mnt/harm/videos -maproot=sysadmin:nogroup -sec=krb5p:sys -network 10.1.0.0/16
/mnt/harm/unix_backups -maproot=sysadmin:nas-backup -sec=sys:krb5p -network 10.1.0.0/16

[root@ultralisk] /var/log# cat /etc/krb5.conf
[appdefaults]
pam = {
forwardable = true
ticket_lifetime = 86400
renew_lifetime = 86400
}

[libdefaults]
dns_lookup_realm = true
dns_lookup_kdc = true
ticket_lifetime = 24h
clockskew = 300
forwardable = yes
default_realm = HARMONYWAVE.COM

[domain_realm]
harmonywave.com = HARMONYWAVE.COM
.harmonywave.com = HARMONYWAVE.COM

[realms]
HARMONYWAVE.COM = {
kdc = immortal.harmonywave.com
admin_server = immortal.harmonywave.com
kpasswd_server = immortal.harmonywave.com
}

[logging]
default = SYSLOG:INFO:LOCAL7

This is what I get when I run a mount command on a client:

jschaeffer@zipmaster07 ~ $ sudo mount -vvv -o rw,sec=krb5p,proto=tcp,port=2049 ultralisk:/mnt/harm/home/jschaeffer /mnt/tmp/home/jschaeffer/
[sudo] password for jschaeffer:
mount: fstab path: "/etc/fstab"
mount: mtab path: "/etc/mtab"
mount: lock path: "/etc/mtab~"
mount: temp path: "/etc/mtab.tmp"
mount: UID: 0
mount: eUID: 0
mount: no type was given - I'll assume nfs because of the colon
mount: spec: "ultralisk:/mnt/harm/home/jschaeffer"
mount: node: "/mnt/tmp/home/jschaeffer/"
mount: types: "nfs"
mount: opts: "rw,sec=krb5p,proto=tcp,port=2049"
mount: external mount: argv[0] = "/sbin/mount.nfs"
mount: external mount: argv[1] = "ultralisk:/mnt/harm/home/jschaeffer"
mount: external mount: argv[2] = "/mnt/tmp/home/jschaeffer/"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,sec=krb5p,proto=tcp,port=2049"
mount.nfs: timeout set for Tue Nov 8 09:50:38 2016
mount.nfs: trying text-based options 'sec=krb5p,proto=tcp,port=2049,vers=4,addr=10.1.33.3,clientaddr=10.1.32.2'
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting ultralisk:/mnt/harm/home/jschaeffer

At one point I was getting a message in /var/log/messages that said something to the effect of export request succeeded from the client IP address, now however I don't get anything in the messages folder. All I see is this, when I run the mount command no new entries appear in the file:

[root@ultralisk] /var/log# tail -f /var/log/messages
Nov 8 09:22:29 ultralisk ntpd[34346]: ntpd 4.2.8p8-a (1): Starting
Nov 8 09:25:17 ultralisk nfsd: can't register svc name
Nov 8 09:42:01 ultralisk manage.py: [common.pipesubr:66] Popen()ing: klist
Nov 8 09:42:01 ultralisk manage.py: [common.pipesubr:66] Popen()ing: klist
Nov 8 09:42:01 ultralisk manage.py: [common.pipesubr:66] Popen()ing: klist
Nov 8 09:42:01 ultralisk manage.py: [common.pipesubr:66] Popen()ing: klist
Nov 8 09:42:14 ultralisk manage.py: [common.pipesubr:66] Popen()ing: klist
Nov 8 09:42:14 ultralisk manage.py: [common.pipesubr:66] Popen()ing: klist
Nov 8 09:42:14 ultralisk manage.py: [common.pipesubr:66] Popen()ing: klist
Nov 8 09:42:15 ultralisk manage.py: [common.pipesubr:66] Popen()ing: klist

In the WebGUI under System | General | Syslog level I have "Debug" set. If I run my mount command with sec=sys then it succeeds. I've done a bit of research and found several links to related problems.

• https://bugs.pcbsd.org/issues/8316
• https://bugs.freenas.org/issues/8876
• https://bugs.pcbsd.org/issues/7775

Bug #8316 (first link) is very similar to my problem but ultimately ended without a resolution. It was on 9.3 while I'm on 9.10 however. I've gone through several of the suggestions posted in that bug such as changing my /etc/hosts file. This what is currently set:

[root@ultralisk] /var/log# cat /etc/hosts | grep -v '^#'

127.0.0.1 ultralisk.harmonywave.com ultralisk
::1 ultralisk.harmonywave.com ultralisk

I've restarted nfsd, gssd, and mountd in the order suggested, but I still get an error when mounting. I do get the nfsd: can't register svc name issue as show above, but I've not found a solution to this problem.

I'm really stuck trying to get krb5p to work successfully and would appreciate any help. If you need any additional info from my server or client please let me know.

Thanks,
Joshua Schaeffer

 

[dlavigne]

Were you able to resolve this?

 

[SweetAndLow]

Are you using LDAP or local users?

Sent from my Nexus 5X using Tapatalk

 

[Joshua Schaeffer]

Unfortunately no, I was not able to resolve this, but I did get some more information. I sniffed the wire on my client and FreeNAS server and I got back a
NFS4ERR_WRONGSEC error from the server. I started to read the NFSv4 RFC [3530] to try and understand NFSv4 better (didn't get all the way through it yet). I understand what the error means, I just don't understand why the client can't use the rpc_gss_svc_privacy service as (as far as I can tell) everything is setup on the client. I also tried on multiple clients to eliminate a client-side issue. I was originally testing on Linux Mint 17.2, but I also recently tried to mount the NFS share on a Debian 8 container and ran into the exact same issue.

I am using LDAP for authorization and Kerberos for authentication in my entire network. Kerberos uses LDAP as its backend as well.

The next avenue I was pursing was adding just the nfs/* service principal to the /etc/krb5.keytab file. I read somewhere that having multiple principals in the keytab can cause issues.

Thanks,
Joshua Schaeffer