Disaster Recovery

Critical Dependencies Chain

The lab infrastructure has a cascading dependency chain. A failure in one layer propagates:

1. <strong>Time (NTP)</strong> $\rightarrow$ if time drift exceeds 128 seconds, Kerberos tickets expire and TLS handshakes fail
2. <strong>DNS (ns1/BIND)</strong> $\rightarrow$ without DNS resolution, all hostname-based services (Kerberos SRV records, NTP upstream, LDAP) break
3. <strong>Kerberos (odroid-c1/KDC)</strong> $\rightarrow$ without the KDC, authentication fails for all Kerberos-authenticated services
4. <strong>LDAP (bbb1/slapd)</strong> $\rightarrow$ without LDAP, directory lookups fail and user/service account information is unavailable

Recovery priority should follow this chain in reverse: fix time first, then DNS, then KDC, then LDAP.

Failure Scenarios and Recovery

NTP Failure -- All Hosts Lose Time Sync

Symptoms: timedatectl} shows "System clock synchronized: no", Kerberos tickets rejected.

Recovery:

1. Verify GPS unit is locked on odroid-c1 (time host)
2. Check ntpd/ntpsec service: \texttt{systemctl status ntpsec}
3. If GPS is offline, enable holdover mode (internal oscillator continues at reduced accuracy)
4. On affected clients, force resync: \texttt{ntpdate -s time.lab.bitsmasher.net} or wait for ntpd's step-sync to kick in
5. Verify with \texttt{ntpq -p} on clients and \texttt{ntptime} on the server

DNS (ns1) Offline -- Cascade Failure

Symptoms: hostnames don't resolve, Kerberos SRV discovery fails.

Recovery:

1. Check ns1 reachability: ping from any known-good host
2. If ns1 is up but not answering: check BIND (named) service status
3. If BIND has crashed due to config error: restore from last known-good named.conf
4. As a workaround, add manual entries to /etc/hosts on affected hosts until DNS recovers
5. Update krb5.conf with explicit \texttt{kdc = odroid-c1.lab.bitsmasher.net}

LDAP (bbb1) Down -- No Directory Service

Symptoms: slapd failed, status.sh reports both slapd and ldapsearch as failing.

Recovery:

1. SSH to bbb1/lab.bitsmasher.net as root
2. Check TLS certificate validity in the configured cert path
3. Regenerate or replace expired certificates
4. Restart slapd: \texttt{systemctl restart slapd}
5. Verify with ldapsearch for franklin and sly DNs

KDC (odroid-c1) Offline -- No Authentication

Symptoms: kinit fails for all realms, service authentication denied.

Recovery:

1. SSH to odroid-c1 as root (franklin user's key is not authorized)
2. Check kdc process: \texttt{kadmind/krb5kdc status}
3. Verify KDC database integrity: \texttt{kdb5_util list}
4. Restart kerberos services if needed
5. Regenerate host keytabs for affected services via \texttt{kadmin.local}

Backup and Restoration

• Ansible playbooks (lab-franklin collection) are the single source of truth -- they can reconstruct any configured state
• Bare git repos with GPG-encrypted pass entries handle credential backup
• KDC database (\texttt{\$KRB5\_KDB\_FILE}) should be backed up regularly on odroid-c1
• DNS zone files should have local copies (in the DNS role's files/ directory)

Prevention: Monitoring

The current monitoring approach is manual via status.sh. For improved resilience, consider:

• Automated ping/SSH checks on critical hosts in HEARTBEAT.md
• Nagios/Zabbix-style monitoring on a dedicated host
• Email/slack alerts for service down conditions

The lab-franklin Ansible collection should eventually include a monitoring role to automate this.