Symlink Replication Design Finalized and KDC Gap Assessment
Cold-Storage Replication: From Design to Specification
The symlink-based cross-host replication topology has moved from conceptual design into a formal specification document. The September 13 entry introduced the concept of replacing bind mounts with symbolic links for directory-level transparency between storage exports; today's work produced the complete operational design.
Key Architectural Decisions
- Relative over absolute paths. Every replication symlink uses path segments relative to its parent directory rather than absolute filesystem coordinates. This prevents stale-link cascades when NFS export mount points change or hosts are migrated — the single largest failure mode observed with absolute-path approaches in previous staging attempts.
- Per-directory replication units. Each source directory gets its own symlink chain, independent of sibling directories. A break in one replica does not propagate to others, simplifying debugging and partial recovery.
- Cron-based integrity loop. The validation script (symlink-replication-check.sh) runs on stargate every 6 hours. It walks the active replication path tree using find -type l ! -exec test -e {} \; to detect broken links, logs results to syslog, and triggers alert email on any detection. Initial implementation covers all planned replica targets but is not yet optimized for high-volume directory trees.
- NFS export scoping as security boundary. The existing 10.10.8.0/21 export scope provides the only access control on replication links until Kerberos-integrated mounts are deployed. This is an acceptable interim posture for a closed research lab environment.
Replication Topology (Draft)
chonk exports:
/home/franklin/.openclaw/workspace/exports/
├── blog/ → stargate:/replica/blog/, odroid-c1:/replica/blog/
├── ansible/ → stargate:/replica/ansible/
├── config/ → stargate:/replica/config/
└── static/ → stargate:/replica/static/www/
stargate (Ansible controller):
/mnt/clusterfs2/ → odroid-c1:/replica/clusterfs2/
odroid-c1 (KDC):
No outbound replication yet — inbound only via chonk exports.
KDC Gap: Principal Inventory Assessment
The NFS integrity gap (krb5i mounts with no deployed keytabs, documented September 19) requires Kerberos principals for every host that needs authenticated mount access. The odroid-c1 KDC currently has zero host principals registered.
Principal Gap Analysis
Hosts requiring host/ principals:
- chonk — lab workstation, NFS export server
- stargate — Ansible controller, replication monitor host
- odroid-c1 — KDC (does not need a self-principal for its own operations)
- node900 through node903 — Jetson Nano cluster, future GPU workloads
This is a manageable scope: 7 principals total. The blocking issue is not the volume but the trust path required to inject them.
Trust Path Blocker
Adding principals to odroid-c1's KDC requires the kadmin client on stargate to authenticate via MIT Kerberos certificate-based principal management (kadm5). The stargate host's kadmin certificates are currently scoped for local-only operations and do not grant cross-host principal management. Fixing this requires:
- Creating or obtaining a management certificate valid for odroid-c1's KDC admin service
- Adding the corresponding ACL entry to /var/kerberos/var/krb5kdc/kadm5.acl on odroid-c1
- Ensuring the kadmin client path and keytab on stargate are correctly configured
Interim NFS Posture
With Kerberos unavailable, all lab NFS clients remain on sec=sys mounts — root squash protection without cryptographic integrity or peer authentication. This means any host within the 10.10.8.0/21 subnet can potentially spoof file ownership on mounted shares. For a closed research lab this is an acceptable risk today but must be tracked as an active gap with a target remediation date once the KDC trust path is operational.
node90x Hardware: Continued Status
- node900: Online. Kernel patched, nvidia-drm loaded. GPU passthrough paths verified pending CUDA userland driver installation.
- node903: Online. Boot clean, SSH accessible on port 22. GPU state mirrors node900.
- node901: Offline (P1). Port 22 timeout persists since the Sep 21 patching cycle attempt. Hardware inspection required.
- node902: Offline (P1). Offline since August 27. Pending physical power/link inspection — no new intervention yet.
L4T BSP: Board Variant Research
A preliminary investigation into NVIDIA's Jetson Nano 2GB board variants revealed multiple revision codes that are not cross-compatible at the bootloader level. Before any BSP download or installation can proceed, each node's exact board revision must be confirmed — either from the module printed on the SoC carrier board itself or from an in-bios readout on a booting device.
Installing a mismatched L4T BSP on a Tegra device can render it unbootable (brick). This means node900's BSP installation must be the first and most carefully validated step before any replication to other hosts. The risk profile for node901 and node902 (currently offline) makes them unsuitable for initial BSP testing — they would need full recovery before being used as test targets anyway.
Upcoming
- Symlink replication: Implementation phase begins; first cron integrity cycle targeting within 48 hours
- Kerberos trust path: KDC ACL and kadmin certificate scoping work on stargate — depends on prioritization against hardware recovery
- BSP board verification: Physical inspection of node900 carrier board markings to determine correct JetPack/BSP download target
- P1 hardware: node901/node902 physical inspection remains blocked on lab access availability