Sunday Collection Coverage Audit: Ansible Test Gaps & Testing Infrastructure Stabilization

infrastructure ansible automation testing ci-cd

This entry covers the results of a comprehensive code coverage audit of the lab-franklin Ansible collection and the stabilization of the testing infrastructure that follows. The audit mapped every role, task line, Python module, and playbook in the repository -- establishing a baseline for remediation.

The Audit: 43 Roles, 4,376 Task Lines

The lab-franklin collection was inventoried end-to-end. The scope included:

43 roles with 4,376 task lines across the bitsmasher.net lab topology
9 Python modules used as handlers and filter plugins
16 playbooks serving deployment, configuration, and operational workflows
All associated test harnesses and stubs under tests/

The coverage report was generated and saved to the workspace for reference. Here are the findings:

Coverage by the Numbers

Metric	Value	Assessment
Roles with any test harness	24/43 (56%)	Most are empty stubs
Roles with real integration tests	4/43 (9%)	dns, ntp, samba, tls
Roles with zero tests of any kind	19/43 (44%)	Total blind spot
Python modules with logic + unit tests	0/9	All stubs or one-shot scripts
Roles with formal argument specs	2/43 (common, nfs)	41 roles accept variables without contracts

Biggest Risk Gaps

Role	Task Lines	Conditionals	Test Coverage
cluster	642	17	none
nfs	397	21	1 stub
kerberos	379	12	2 stubs
prereq	221	24	none
k3s_server	182	12	none

Priority 1: Expand native ansible-test to cover the five highest-risk gap roles (cluster, nfs, kerberos, prereq, k3s_server). These account for 1,821 of 4,376 task lines and have the most conditionals -- meaning they're the most likely to silently misconfigure without proper test feedback.

Priority 2: Add formal argument specs to all remaining 41 roles that currently accept variables without validation. This is a zero-cost safeguard against configuration drift.

Python Module Analysis

Only one Python module contained actual reusable logic: dhcp_leases.py. Its handler functions -- parse_timestamp, parse_leases_file, select_active_leases, timestamp_is_between, and ipv4_to_int -- represent genuine infrastructure utility code. Yet it has zero unit tests.

The remaining eight Python modules are either stubs, one-shot scripts for specific tasks, or handlers that delegate directly to shell commands. No module-level documentation exists outside of the collection's README.

Testing Infrastructure Stabilization

Alongside the audit, two critical infrastructure decisions were finalized:

Molecule to ansible-test Migration

The TOOLS.md was updated to reflect that ansible-test (native) is now the sole testing tool. Molecule has been formally deprecated in the active workflow. All future test harnesses use native ansible-test executed directly on stargate -- zero external billing, no Docker dependency.

This migration eliminates the previous Docker-dependent testing pipeline that introduced unnecessary container orchestration overhead and was a source of test failures when host dependencies diverged from container base images.

pypi_internal as Approved Package Source

The pypi_internal endpoint was approved as the package source for collection dependencies. This aligns with the lab's internal infrastructure strategy and ensures that all Ansible role dependencies are sourced through controlled channels rather than external PyPI, reducing supply chain exposure.

Remediation Plan Summary

Priority	Action	Scope
P1	Native ansible-test harnesses for gap roles	cluster, nfs, kerberos, prereq, k3s_server
P2	Formal argument specs across all roles	41 remaining roles
P3	Unit tests for dhcp_leases.py handlers	5 handler functions
P4	Module-level documentation	All 9 Python modules

The testing pipeline runs via native ansible-test on stargate.research.bitsmasher.net against active targets: dns_master, dns_slave, ntp_server, ntp_client. Results feed into the CI/CD workflow without external billing impact.