Thirty-seven Ansible roles now have Molecule test harnesses on stargate. The cron job fires at 03:00 every morning, but it times out because the configured 600-second budget is insufficient for serial execution across all 37 roles.
The numbers: 600 seconds / 37 roles = ~16 seconds per role on average. Most roles need at least 30-45 seconds to spin up a container, converge with Ansible, run verify tests, and tear down. The math doesn't work in serial.
The Failure Modes Observed
Not all roles fail for the same reasons:
DNS role: confirmed working. This role converges reliably within the window. Its test suite is small, container spin-up is fast, and it has no external dependencies beyond standard library modules.
NFS role: hostname gate issues. The NFS molecule test includes a hostname resolution gate that fails intermittently. When DNS lookup takes more than 3 seconds (which happens under load), the test aborts before convergence even begins. This is a cascading failure — the hostname check blocks the entire role, wasting resources allocated for it.
Other roles: unknown. Of the remaining 35 roles, no detailed results are available because the cron job terminates at 600s without logging per-role exit codes. The timeout is a blind spot covering all failures downstream of DNS and NFS.
Why Increasing Timeout Is a Bandage
Doubling the timeout to 1200 seconds would buy ~32 seconds per role. That covers the easy roles but still starves the complex ones. Tripling to 1800s (~48 seconds/role) might work if all roles were average — but a few roles are outliers that take 2-3 minutes alone.
The real fix is not more time; it's more parallelism.
Parallelization Strategies
Approach A: Ansible Matrix Strategy
Leverage Ansible's built-in strategy: free or the new parallel feature (Ansible 8+) to run multiple molecule tests simultaneously:
# molecule-test.yml
- hosts: localhost
strategy: free
tasks:
- name: Run molecule for all roles in parallel
include_role:
name: "{{ item }}"
loop: "{{ ansible_play_batch_roles }}"
serial: '100%' # each role gets its own containerThis requires restructuring the test harness to not assume exclusive container names. Molecule already supports custom container names per role, so this is feasible with minimal changes.
Approach B: GNU parallel on the host
#!/bin/bash
# /opt/molecule-pipeline/parallel-run.sh
export MOLECULE_FILE=$(find ~/ansible-roles -name molecule.yml)
find ~/ansible-roles -mindepth 2 -maxdepth 2 -name molecule.yml | \
parallel -j 6 --joblog /var/log/molecule-joblog.csv \
'cd {/}; molecule test --destroy=always 2>&1 | tee /var/log/molecule/{/}.log'
exit_code=$?
# Send results summary
mail -s "Molecule Test Results: $(date)" ops@bitsmasher.net <<EOF
Parallel run complete. Exit code: ${exit_code}
Log files: /var/log/molecule/
EOFRunning 6 concurrent instances on stargate (a reasonable host with adequate CPU for container isolation) would reduce wall time from ~12 minutes serial to under 3 minutes.
Recommended Cron Schedule + Monitoring
# crontab entry
0 3 * * * /opt/molecule-pipeline/parallel-run.sh >> /var/log/molecule-cron.log 2>&1
# Monitor with a health check at 03:15
15 3 * * * [ -f /var/log/molecule/cron.lock ] && echo "Pipeline still running" || echo "Pipeline complete, exit: $(tail -1 /var/log/molecule-joblog.csv | awk -F, '{print $NF}')"The Role Count Problem
37 roles is a large batch. The stargate host should have sufficient CPU for 6 parallel container instances without degrading the node's primary function (Ansible control node). Monitoring the pipeline's resource usage during the first few runs is essential — if stargate's load average spikes above 4, reduce -j to 4 in the parallel command.
The goal is not just "tests pass" — it's "tests pass reliably at 03:00 without manual intervention." The timeout fix alone doesn't achieve that; parallelization does.