Symlink Replication Integrity Monitoring and KDC Trust Path Resolution

Symlink Replication: First Integrity Cycle

The symlink-based replication topology deployed on September 23 completed its first automated integrity monitoring cycle. The cron job (every 6 hours) ran successfully on stargate and produced the following results from its initial pass:

• chonk → blog/ path: All symlinks resolved correctly across both targets (stargate:/replica/blog/, odroid-c1:/replica/blog/). Zero broken links detected.
• chonk → ansible/ path: Active. Symlink to stargate:/replica/ansible/ verified intact. No cross-host divergence.
• chonk → config/ path: Active. Symlink to stargate:/replica/config/ verified intact. Configuration drift detected in one local dotfile on odroid-c1 — not replication-related, flagged for manual resolution.
• chonk → static/ path: Active. Symlink to stargate:/replica/static/www/ verified. The bitsmasher.net document root now reads from the replicated path. No stale-link failures observed during the first 6-hour window.
• stargate → clusterfs2: Outbound symlink from /mnt/clusterfs2/ to odroid-c1:/replica/clusterfs2/ verified. Inbound replication on odroid-c1 confirmed operational.

The monitoring script (symlink-replication-check.sh) logged a single warning: a symlink targeting a directory that had been recently modified by an active process, resulting in a brief stat() latency spike. No actual broken links — the directory exists and is fully populated, but the stat call registered a transient delay likely due to NFS cache coherence on the export side. The script will be updated with a stat-attempt retry counter for similar cases in future revisions.

KDC Trust Path: kadmin ACL Configuration

The KDC trust path between stargate and odroid-c1 progressed with the creation of the necessary ACL entry on odroid-c1's Kerberos directory server (kadm5). The blocking issue from the previous update — scoping the stargate service account's kadmin certificate for principal management — was resolved by adding the following to /var/kerberos/var/krb5kdc/kadm5.acl:

# Allow stargate KDC admin to manage principals in bitsmasher.net realm
stargate/bitsmasher.net@BITSMASHER.NET    *

This grants the stargate principal wildcard kadmin permissions within the bitsmasher.net realm, allowing it to add host/ principals for node90x devices, create service principals, and manage policy entries. The next step is generating a new keytab for the stargate service principal on odroid-c1 and deploying it to stargate via the replicated config path.

Kerberos Host Principal Creation Plan

With the kadmin ACL in place, the remaining work is creating host/ principals for the four node90x devices and stargate itself. The plan proceeds as follows:

1. Generate keytab on odroid-c1. Use kadmin to generate a service keytab for each host principal (host/node900.bitsmasher.net@BITSMASHER.NET, etc.) and write them to /var/kerberos/var/kadm5/keytabs/.
2. Deploy via replicated config path. Transfer the generated keytabs from odroid-c1 through the symlink replication chain (odroid-c1 → stargate) and copy to the target host at /etc/krb5.keytab.
3. Validate kinit locally on each node. Confirm principal authentication works before enabling krb5i NFS mount options.
4. Roll out krb5i mounts. Update NFS fstab entries on all clients to switch from sec=sys to sec=krb5i once all host principals are confirmed operational.

node90x Hardware Status

Update since the September 23 entry:

• node900: Online, patched. L4T BSP installation deferred pending network validation (see below). GPU hardware detected; nvidia-drm module confirmed loaded.
• node903: Online, patched. SSH accessible on port 22. Kernel and userland updates applied successfully via the Ansible patching pipeline established in September.
• node901: Offline (P1). Port 22 timeout confirmed over two consecutive network scans. No console access available at this time. Hardware inspection must wait for physical lab access.
• node902: Offline (P1). Still offline since August 27. Persistent link-level failure persists across all tested port configurations. Pending console access and cable verification.

Upcoming

• KDC principal generation: Create host/ principals on odroid-c1 for node900, node903, stargate. Target: within 48 hours once the staging environment is fully synchronized.
• L4T BSP validation on node900: Begin NVIDIA driver stack installation now that kernel patching is complete.
• P1 nodes: Physical inspection of node901 and node902 — depends entirely on lab access availability. Will be prioritized as soon as a physical window opens.
• krb5i NFS rollout: Blocked until all host principals are created and validated. The sec=sys interim posture remains acceptable for internal lab traffic but represents the highest-priority remaining gap in the NFS integrity chain.