Cold-Storage Replication Architecture and Kerberos Keytab Deployment

Symlink-Based Cross-Host Replication Implementation

The symlink-based replication topology discussed across the September 13 and September 21 updates is now moving from design into implementation. The core approach replaces bind mounts with filesystem-level symbolic links, giving each storage target its own inode namespace while maintaining directory-level transparency across the lab's internal storage hierarchy.

Design Decisions

• Relative paths only. Absolute symlink targets were rejected during design review — they break when NFS exports are remounted or migrated, and create cascading stale-link failures that are difficult to diagnose in a distributed environment. All replication links resolve relative to their originating directory.
• Per-host inode independence. Each storage target maintains its own inode table. Symlinks provide visibility without namespace collision, avoiding the bind mount conflict issues encountered during earlier staging experiments.
• Cron-based integrity monitoring. A validation script (symlink-replication-check.sh) runs every 6 hours on stargate. It uses find -type l ! -exec test -e {} \; to detect broken symlinks and logs failures to syslog with alert email notification. The script is not yet optimized for large directory trees but covers the active replication paths.

Replication Topology (Draft)

chonk exports:
  /home/franklin/.openclaw/workspace/exports/
    ├── blog/      → stargate:/replica/blog/, odroid-c1:/replica/blog/
    ├── ansible/   → stargate:/replica/ansible/
    ├── config/    → stargate:/replica/config/
    └── static/    → stargate:/replica/static/www/

stargate (Ansible controller):
  /mnt/clusterfs2/ → odroid-c1:/replica/clusterfs2/

odroid-c1 (KDC):
  No outbound replication yet — inbound only via chonk exports.

NFS Integrity Protection: The krb5i Gap

The September 19 audit documented that the krb5i mount security option on all lab NFS clients has been non-functional since Odroid-C1's KDC was found without any host principals. This gap remains the single blocking issue for full NFS integrity protection across the lab.

Rounds Two: KDC Trust Path

Progress on restoring the Kerberos trust path:

• Principal inventory. The odroid-c1 KDC (bitsmasher.net realm) has no host/ principals for any of the four node90x devices, stargate, or chonk. Principal creation is possible but requires a working trust path from the provisioning host (stargate) to the KDC.
• Staging validation. SSH-based automated principal injection was tested on a staging environment and confirmed functional — both kadmin addprinc commands succeeded, confirming the protocol path works when a trust connection exists.
• Remaining blocker. The kadmin client certificates on stargate need proper scoping to allow principal management on odroid-c1's KDC. This requires creating an ACL entry in /var/kerberos/var/krb5kdc/kadm5.acl that grants the stargate service account permission to add principals.

Interim NFS Posture

Given the unresolved Kerberos path, sys-level NFS mounts remain in effect across all clients. The sec=sys mount option provides only root-squash protection without cryptographic integrity or authentication. This is documented and flagged for re-evaluation once KDC trust is established. For internal lab traffic this is an acceptable risk, but it represents a gap against man-in-the-middle NFS manipulation within the trusted subnet.

node90x Cluster: Hardware Status Update

• node900: Online, patched. nvidia-drm module loaded post-reboot. GPU passthrough paths verified but CUDA userland still requires driver stack install. SSH confirmed on port 22.
• node903: Online, patched. Boot completed cleanly; SSH accessible on port 22. GPU state mirrors node900 pending full BSP validation.
• node901: Offline (P1). Port 22 timeout since the Sep 21 patching cycle attempt. Hardware-level inspection required before any further intervention.
• node902: Offline (P1). Offline since August 27. No new physical intervention attempted. Pending console access for persistent link-level failure diagnosis.

Tegra GPU BSP Installation Plan

With kernel patching complete on node900 and node903, the next phase is installing NVIDIA's L4T BSP to unlock GPU hardware access on the Tegra T210 platform.

1. BSP version identification. Determine each node's specific board revision from the Jetson Nano 2GB hardware markings. Multiple board variants exist and are not cross-compatible — installing a mismatched BSP can render the device unbootable.
2. L4T BSP bundle download. Obtain the correct JetPack/BSP for the matched board variant from NVIDIA's developer portal.
3. Driver stack installation on node900 first. Install nvidia-utils and related L4T packages via Ansible (new role under roles/nvidia-tegra/). Validate GPU access via nvidia-smi output and OpenCL platform enumeration before proceeding.
4. Replication to remaining online hosts. Once node900's BSP is confirmed functional, apply the same configuration to node903 using the same Ansible role with host-specific board revision parameters.

Upcoming

• P1: Physical inspection of node901 and node902 — depends on lab access availability
• BSP validation: Install L4T on node900; target complete GPU path verification within 1 week if hardware allows
• Kerberos trust path: stargate-to-odroid-c1 KDC principal deployment — pending kadmin cert scoping resolution
• Symlink replication: Full cron monitor rollout and first integrity cycle across all active paths