Troubleshooting

SSH Connection Failures

Host Key Mismatch

Symptom: @@@@@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @@@@@}

Fix:

\begin{lstlisting}[style=mystyle]

ssh-keygen -R <hostname>

    \end{lstlisting}

This occurs when a host was re-imaged with the same IP. Check /etc/ssh/ssh\_config.d/} for stale entries.

Auth Denied (Port 22 Open, Key Rejected)

Symptom: TCP port 22 is reachable but SSH returns "Permission denied".

Common causes:

• No matching key in the remote host's authorized\_keys
• The key file permissions are too open on the target (must be mode 600 for .pub and 700 for ~/.ssh)
• Wrong user context (e.g., trying franklin@ when only root is configured)

Fix: Add the openclaw ed25519 key to the target host's authorized\_keys file. The pubkey is:

\begin{lstlisting}[style=mystyle]

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJb9mV02PpxD8VpzYCnu7192dTwMnGWUc3qh55BIeElN openclaw@chonk

    \end{lstlisting}

No Route to Host

Symptom: SSH hangs or returns "No route to host". The host is likely powered off. Common in Jetson nodes (node902, etc.) that are manually managed.

GSSAPI Interference

Symptom: SSH times out waiting for GSSAPI authentication before falling back to key auth.

Fix: Add to ~/.ssh/config}:

\begin{lstlisting}[style=mystyle]

Host *

GSSAPIAuthentication no

IdentitiesOnly yes

IdentityFile ~/.ssh/id_ed25519_openclaw

    \end{lstlisting}

Service-Specific Diagnostics

Kerberos Authentication Failure

tabularx}{}{l X}

<strong>Symptom</strong> & <strong>Likely Cause</strong> \\

---

kinit fails with "clock skew" & NTP not synced -- check time.host first\\

kinit fails with "KDC has no support for encryption type" & Wrong krb5.conf realm or KDC unavailable\\

kadmin.local works but kadmin (remote) fails & Port 749 firewall rule on odroid-c1\\

Ticket expired after reboot & Time drift during host downtime

tabularx}

Verify: klist}, kinit username@LAB.BITSMASHER.NET}, ktutil} for keytabs.

NTP Issues

tabularx}{}{l X}

<strong>Symptom</strong> & <strong>Likely Cause</strong> \\

---

ntpd not running & ntpsec service not enabled\\

"no server suitable for synchronization" & No upstream reachable (time host offline)\\

Clock gradually drifting & Missing driftfile or Jetson oscillator drift\\

Time jumps by several seconds & Large correction applied; step-sync mode engaged

tabularx}

Verify: ntpq -p}, timedatectl}, check /var/log/ntpstats/.

NFS Mount Issues

tabularx}{}{l X}

<strong>Symptom</strong> & <strong>Likely Cause</strong> \\

---

mount hangs & NFS server unreachable or export doesn't list client\\

"Stale file handle" & Server-side data was deleted/recreated\\

Permission denied on mount & /etc/exports restricts the client IP

tabularx}

Verify: showmount -e <server>}, check NFS server's exports, verify client IP matches the export rule.

Common Ansible Troubleshooting

Module Not Found / Missing Collection

\begin{lstlisting}[style=mystyle]

ansible-galaxy collection install lab.franklin

# or for individual modules:

ansible-galaxy collection install community.general

    \end{lstlisting}

Molecule Test Failures on New Hosts

If molecule tests fail on a new host, the most common causes are:

1. Podman not installed or permissions wrong (openclaw user needs podman group)
2. Docker driver configured instead of podman -- edit \texttt{molecule.yml} to set driver
3. Python version mismatch (molecule requires $\ge$3.9)
4. Role hostname gates preventing test execution (e.g., the DNS role's hostname check for the real server vs. molecule container name)

General Debugging Checklist

When a service is down, work through this checklist:

1. Can you ping the host? ($\rightarrow$ network up?)
2. Is SSH responding? ($\rightarrow$ host OS running?)
3. Does the service process exist? (\texttt{systemctl status or ps aux})
4. Are logs showing errors? (\texttt{journalctl -u <service> --since today})
5. Can you connect to the service port from another lab host? (\texttt{nc -zv <host> <port>})
6. Is DNS resolving correctly? (\texttt{dig or getent hosts})
7. Is time synchronized? (\texttt{timedatectl})

Quick Reference Commands

\begin{lstlisting}[style=mystyle]

# Check if a host is reachable from chonk

ping -c 1 <hostname>

ssh -o ConnectTimeout=5 openclaw@<host> "echo alive"

# Check service status remotely

ssh openclaw@<host> "systemctl status <service>"

# Verify time sync across hosts

for host in stargate skynet chonk; do

ssh openclaw@$host "timedatectl | grep synchronized"

done

# Verify DNS resolution from a client

dig @time.lab.bitsmasher.net <hostname>

# Check Kerberos auth

kinit username@LAB.BITSMASHER.NET

klist

    \end{lstlisting}