infrastructure automation ansible security

This entry covers the latest round of hardening and restructuring work on the lab-franklin Ansible collection hosted on stargate. The collection currently manages 44 roles across the bitsmasher.net lab topology -- from Jetson Nano edge nodes to Kubernetes clusters, Kerberos KDCs, and Samba file shares. Today's changes fall into three categories: config hygiene, role hardening, and operational planning.

The 695-Line Config File That Was Nobody's Business

The most impactful change was not a role at all -- it was the ansible.cfg at the repository root. It contained 695 lines of heavily commented boilerplate, every default option Ansible ships with, each one documented and disabled via semicolon. Nobody reads that file. Nobody edits it. It just sits there taking up terminal real estate and git blame history.

Before: 695 lines of commented-out Ansible defaults
After: 10 lines -- collections_paths, roles_path, inventory, remote_tmp, plus minimal privilege escalation config

The trimmed config keeps only what's actually non-default for this lab: custom collection paths across two locations (local ~/.ansible and /mnt/backup1/workspace), the collection-based roles directory, local inventory path, remote temp directory, and basic become configuration.

This also shipped new linting files at the collection root level: .yamllignores (51 entries for legitimate static YAML in k8s manifests and Kubernetes CRDs that don't follow role conventions) and .yamllint with relaxed rules for document-start, truthy literals, line length, and spacing -- because the lab's manifests have style conventions different from standard role files.

DNS Role: From Static Copies to Jinja2 Templates

The lab.franklin.dns role's master path got the most structural rewrite. Previously it used static copy operations for named.conf.options and zone files -- fine until you need different forwarders per deployment or want to avoid config drift across environments.

The new tasks use Jinja2 templates:

# roles/dns/templates/named.conf.options.j2
options {
 directory "/var/cache/bind";
 recursion yes;
 allow-recursion {
  127.0.0.1;
  {{ ansible_default_ipv4.address | default('10.10.12.3') }};
 };
 forwarders {
  10.10.16.1;
  1.1.1.1;
  8.8.8.8;
 };
 dnssec-validation auto;
 listen-on { 127.0.0.1; {{ ansible_default_ipv4.address }}; };
};

# roles/dns/templates/named.conf.local.j2 (dynamic zones)
{% for zone in dns_master_zones | default(['lab.bitsmasher.net', 'research.bitsmasher.net']) %}
zone "{{ zone }}" {
 type master;
 file "{{ dns_zone_dir | default('/var/lib/bind') }}/db.{{ zone }}";
 allow-transfer { none; };
};
{% endfor %}

The task list was also restructured to match the modern pattern: explicit package installation with apt, directory ownership by bind user, config validation via named-checkconf before applying changes, and proper systemd service management. Handler names were updated from the old restart bind9 service to restart named and reload named zones.

dns The zone directory was moved from the default /etc/bind to a configurable /var/lib/bind path -- this is important because some deployments mount that directory separately for backup purposes.

Jetson-Nano Role: Post-Power-Restoration Sequence

The Jetson nodes (node900-903) present a unique challenge: they're low-power ARM edge devices that power-cycle unexpectedly. When they come back up, they need to bootstrap themselves -- SSH keys, NTP sync, interface verification -- before any automation can reach them.

The expanded role adds:

# Configurable vars for the role
nvidia_driver_version: "535"
cuda_version: "12.2"
jetson_overclock_mode: "max perf"
post_power_restore: true

# Post-restoration tasks (conditional)
- name: Post power restore: SSH key injection
  ansible.builtin.copy:
    content: "{{ post_restore_ssh_public_key }}"
    dest: "/home/franklin/.ssh/authorized_keys"
    mode: "0600"
  when: post_power_restore | default(false)

- name: Post power restore: NTP sync via chrony
  ansible.builtin.command: chronyc makestep
  when: post_power_restore_ntp_sync | default(true)

Meta file was also updated from the galaxy scaffold template to proper Galaxy metadata format with author, license (MIT), minimum Ansible version (2.14), and platform targets (Ubuntu jammy/noble). The role now depends on lab.franklin.common.

SSH Role: From GPG Generators to Focused Key Management

The SSH role's defaults had been bloated with GPG key generation parameters -- user names, passphrases, fingerprint paths, export locations. The role is called ssh, not gpg-and-ssh. That config was removed and replaced with a focused variable:

# defaults/main.yml
ssh_pubkeys_franklin:
  - "{{ lookup('file', '/home/franklin/.ssh/id_ed25519_openclaw.pub') }}"
  - "{{ lookup('file', '/home/franklin/workspace/lab-franklin/ansible/collections/ansible_collections/lab/franklin/roles/ssh/files/chonk-key.pub') | default('') }}"

A new sshd_config.j2 template was created alongside the existing hardened drop-in. The playbook playbooks/ssh.yml now explicitly excludes certain hosts from the standardize run:

hosts: all:!chonk.lab.bitsmasher.net:!node_access_pending:!console_required

This prevents the role from overwriting chonk's existing SSH hardening or touching nodes with physical console requirements that need default configuration.

Planning Documents Created

Samba Active Directory Domain Controller Plan

A full 7-phase plan was created for deploying Samba AD on chonk (10.10.8.60) as the lab's primary Kerberos/LDAP backend and file share host. This replaces/supplements odroid-c1 (KDC) and ldap.bbb1 (OpenLDAP, degraded since December 2025).

The plan:
Phase 1: Install samba + krb5-user + ldb-tools on chonk
Phase 2: Provision AD domain with samba-tool (BIND9_TDB backend)
Phase 3: DNS integration -- Samba DNS for lab zones only, forward external queries to time server
Phase 4: Update /etc/krb5.conf across all lab hosts for LAB.BITSMASHER.NET realm
Phase 5: Deploy file shares via existing lab.franklin.samba role
Phase 6: ACL permissions and NT4-style security model
Phase 7: Testing and migration -- parallel run with odroid-c1, decommission on confidence

Risk mitigation includes keeping odroid-c1 active during the transition period, using Samba DNS only for lab bitsmasher.net zones to avoid conflicts with BIND on the time server, and monitoring /var/lib/samba disk usage quarterly.

Lab Manual Restructuring: From 6 Lines to 18 Chapters

The lab manual at docs/manual/ was only 16 lines of actual content with a missing _header.tex file. A restructuring proposal maps all existing sections to a comprehensive 18-chapter layout:

PartChaptersDescription
I1-3Infrastructure Foundations: History, Network Arch, Host Inventory
II4-8Core Services: NTP, DNS, Kerberos/LDAP, Certificates, Network Security
III9-11Compute & Storage: Kubernetes, Containers, File Services (Samba + NFS)
IV12-15Development & Automation: Ansible Collection, Terraform, CI/CD, Monitoring
V16-18User Docs: Developer Onboarding, Hardware Inventory, Appendix

The Terraform chapter was already created in this session. The remaining 17 chapters map to existing markdown documents and infrastructure state -- most content already exists as scattered .md files under docs/markdown/, just never compiled into a single manual.

Test Harness Audit

A comprehensive audit of the ansible-test integration harness documented four active targets:

TargetStatusNotes
ntp_serverPASSntpsec + gpsd, validates /etc/ntpsec/ntp.conf
ntp_clientFALSE FAILShared env conflict -- ntpsec from previous test pollutes state
dns_masterPASSbind9 forwarder, validates named.conf.options
dns_slavePASSbind9 slave, validates allow-recursion ACL

The hourly test runner script exists but has no cron job attached. The deprecated test/test_collection.sh script outputs "This script is deprecated" and tries to call a nonexistent flag -- it should be deleted.

Collection Inventory: What's Working, What Isn't

A full audit of all 44 roles in the collection produced this breakdown:

CategoryCountExamples
Complete (tasks + handlers + docs)30common, k3s_server, kerberos, dns, nfs, ssh, security
Minimal/Functional9samba, tls, shell, media, paloalto
Stubs (zero tasks)4edge, extensions, latex, odroid
Partially Complete1k8s (159 static YAML files, zero Ansible glue)
Recommendations:
1. Delete or implement the 4 stub roles (edge, extensions, latex, odroid)
2. Audit k8s role: promote manifest files to actual tasks or move as reference content
3. Clarify media role scope -- installs single fbi package but README claims "audio and video packages"
4. Rename HOMELAB_MOLECULE_TEST variable in dns tests -- molecule is gone, the env var name is misleading

Status

Branch: 20260820-01 (stargate)
Latest commit: cac6694e "even more ansible work"
Uncommitted today: ansible/playbooks/ssh.yml (host exclusion list update)
New planning docs: chonk-samba-plan-20260802.md, manual-layout-proposal-20260802.md

Production site: wonderland (www.bitsmasher.net/research)