eBPF Socket Tracing and Raw Packet Telemetry Pipeline
eBPF Socket Tracing Architecture
Implemented a network-level telemetry pipeline on skynet using the BCC toolkit's tcplife and sslsniff eBPF programs to trace all TCP socket lifecycle events — connect, accept, send, recv, close — with per-connection latency measurement. This provides visibility into every connection in the lab.franklin topology without requiring instrumentation on individual applications.
tcplifecaptures full TCP flow lifetimes, computing duration, bytes transferred, and retransmission counts for each connection.sslsniffextracts TLS handshake metadata (client/server version, cipher suite, certificate CN) from all encrypted traffic on port 443/8443.
Raw Packet Audit via Ring Buffer
A separate eBPF program was deployed to capture raw packets destined for or sourced from management ports (SSH:22, DNS:53/514, Vault:8200) using the eBPF ring buffer map (BPF_MAP_TYPE_RINGBUF). This replaces a previous tcpdump-based approach with a zero-copy kernel-side pipeline:
- Ring buffer size: 32MB per-CPU, dynamically scalable under load.
- Egress path: Ring buffer consumer writes captured packets to a shared-memory queue consumed by the telemetry pipeline daemon, which formats and ships to node3 (logging.lab.bitsmasher.net).
- Performance: At 10k pps sustained traffic, eBPF ring buffer consumes ~2.1% CPU on skynet's quad-core, compared to tcpdump's 14.7% on the same workload.
Benchmark: eBPF vs. tcpdump
| Metric | eBPF Ring Buffer | tcpdump |
|---|---|---|
| CPU overhead (10k pps) | 2.1% | 14.7% |
| Packet loss rate | 0% | 0.3% at sustained 15k+ pps |
| Latency (capture-to-queue) | ~12µs median | N/A (userspace write delay) |
The eBPF pipeline is now running in production on skynet with a planned rollout to stargate and chonk. node3's syslog receiver was updated to accept raw packet data via UDP port 9060 (custom protocol, separate from standard syslog).