Kerberos Keytabs That Never Worked

The lab runs nfs-kernel-server with sec=krb5i exports on both chonk (10.10.8.60) and stargate, yet Kerberos keytabs were never deployed to any client host. Every krb5-mounted NFS share silently falls back to insecure anonymous access or fails entirely — the security posture is a paper wall.

This post traces the full gap between theoretical Kerberos-authenticated NFS and the actual auth stack as it exists today, then lays out a concrete remediation playbook.

The Export Landscape

chonk exports three directories to lab subnets:

/mnt/storage1   *(rw,sync,sec=krb5i,crossmount,no_root_squash)
/mnt/storage2   *(rw,sync,sec=krb5i,crossmount,no_root_squash)
/mnt/storage3   *(rw,sync,sec=krb5i,crossmount,no_root_squash)

stargate exports its own cluster filesystem:

/mnt/clusterfs2 10.10.14.0/24(rw,sync,sec=krb5i,crossmount,no_root_squash)

All six exports declare sec=krb5i (Kerberos with integrity checking). None have functional client keytabs deployed.

Why the Gap Persisted

The Kerberos realm for lab.bitsmasher.net exists on odroid-c1 (kdc1) with ports 88 and 749 open. kinit to admin@lab.bitsmasher.net works from any reachable host. The gap is specifically in keytab generation and distribution:

1. Keytabs were never generated for NFS service principals. Without entries like nfs/chonk.lab.bitsmasher.net@lab.bitsmasher.net in the KDC, there is nothing to extract.
2. No automation existed for keytab distribution. Each host needs its own keytab placed at /etc/krb5.keytab with correct permissions (0600).
3. The clusterfs2 ownership question remained unresolved. Even if keytabs were deployed, it is unclear which host owns /mnt/clusterfs2 on stargate post-provisioning.
ol>

Remediation Playbook

The fix requires three coordinated steps:

Step 1: Generate NFS Service Principals on KDC

kadmin -q "addprinc -randkey nfs/chonk.lab.bitsmasher.net@lab.bitsmasher.net"
kadmin -q "addprinc -randkey nfs/stargate.lab.bitsmasher.net@lab.bitsmasher.net"
# Repeat for every host that mounts NFS exports

Step 2: Extract and Distribute Keytabs

kadmin -q "ktadd -k /tmp/nfs.keytab nfs/chonk.lab.bitsmasher.net@lab.bitsmasher.net"
# Then deploy to each host:
scp /tmp/nfs.keytab franklin@hostname:/tmp/
ssh hostname "sudo cp /tmp/nfs.keytab /etc/krb5.keytab && sudo chmod 600 /etc/krb5.keytab"

Step 3: Verify Mount Integrity

mount -t nfs4 -o sec=krb5i chonk.lab.bitsmasher.net:/mnt/storage1 /mnt/nfs1
id  # Should show a Kerberos-mapped UID, not nobody:nogroup
cat /proc/mounts | grep nfs1  # Verify sec=krb5i is active

A full Ansible role for this remediation should be written as part of the openclaw user placement strategy — it belongs alongside the existing SSH roles on stargate.

The clusterfs2 Ownership Question

A parallel finding from the lab infrastructure assessment: /mnt/clusterfs2 on stargate has unresolved ownership. The NFS exports declare it, but no host management role claims responsibility for its provisioning. Before deploying keytabs to stargate, this ownership question must be resolved to prevent double-ownership conflicts post-migration.

Action required: Assign clusterfs2 ownership to a named role before running the keytab distribution playbook. The NFS + Kerberos remediation and the clusterfs2 ownership assignment are interdependent.

Taking Stock

The Kerberos/NFS auth gap is one of many "paper security" issues across the lab — visible in configuration files but not enforced at runtime. It is low-risk for the current isolated lab environment (no external attackers), but it sets a precedent that should be broken: if sec=krb5i doesn't actually require Kerberos, what else in the exports list is equally cosmetic?

This post documents the gap and the fix. Next step: write the Ansible role and deploy keytabs across all reachable hosts.