Conformance-monitor hunt — finding silicon-class defects with the verification moat

This guide is the adversarial companion to CONTRACT_FIRST_WALKTHROUGH. That one is for the IP author closing the loop on one IP. This one is for the verification engineer (or agent) running a campaign: bolt RouteRTL's validated passive monitors onto every real bus port in the codebase and hunt for protocol-conformance defects that pass the IP's own sim stimulus but violate the spec against a legal partner.

It is written from a real campaign (EPIC RTL-P2.719 / ticket RTL-P3.851) that surfaced four silicon-class defects across four independent submodules in one session — DDR-T2.9 (data loss under backpressure, both DDR backends), VHD-T2.7 (short Ethernet preamble — found and fixed), and DS-T2.47 (VALID-into-reset) — plus a tooling bug (RTL-T3.14). Every command below is one that campaign actually ran.

0. Why this works — the architecture you're leveraging

The hunt is only cheap because four pieces of RouteRTL compound:

1. The verification moat — rr_cocotb_tb/. A registry of validated passive protocol monitors (rr_cocotb_tb.monitors.*) and adversarial BFMs (rr_cocotb_tb.drivers.*). A monitor attaches to a bus and raises ProtocolViolation whenever a spec invariant breaks — you write zero checker code. Each monitor is itself proven by a Known-Answer-Test (its matching BFM EMITS the corner-case, the KAT asserts the monitor flags it), so when a monitor fires on your DUT, the monitor is the oracle, not the suspect.

2. pip install -e distribution. RouteRTL is installed editable, so every submodule — h264-dec, rr-ddr-stream, vhdl-ethernet, deepskopion — can from rr_cocotb_tb.monitors.axi_stream_monitor import AxiStreamMonitor with no vendoring. One registry, every repo. The same recipe is portable across the whole super-repo because the verification IP is shared, not copied.

3. rr sim run — one sanctioned executor. The same command runs in every submodule; it owns sys.path / SDK_ROOT / build_env.tcl / log capture / result collection, and autodiscovers HDL from project.yml so you usually don't even list sources. You learn it once and hunt anywhere.

4. The conformance graph + PM loop. rr_sim_conformance_graph and friends tell you where the holes are (which monitor clauses no BFM drives); tlog / tcap / tctx / tdecision track what you find. The graph is the moat's self-knowledge; the PM loop is its memory.

1. Find the holes — what's unchecked

Before hunting, ask the moat what it already knows and where it's blind:

# Every passive monitor shipped + the clauses each enforces
rr_sim_monitors                 # (MCP tool; or: rr sim monitors)

# Every BFM + the spec corner-cases it can DRIVE (its EMITS tuple)
rr_sim_drivers                  # filter: rr_sim_drivers axi-stream

# The contract-first graph: spec-anchor -> BFM.EMITS -> monitor.observable,
# with the TYPED GAPS (asset gap = a monitor checks it but no BFM drives it)
rr_sim_conformance_graph

An asset gap in that graph is the input-side hole that hid the original RMII-RX-EOF miss: a check that can never fire because no stimulus reaches it. Closing those (RTL-P3.852) is prerequisite hygiene; what's left is the hunt — pointing the now-complete monitors at real DUT ports.

2. The recipe — one port, end to end

For each real bus port on a DUT:

# tests/test_demo_<port>_bfm_conformance.py
from rr_cocotb_tb.monitors.axi_stream_monitor import AxiStreamMonitor
from rr_cocotb_tb.drivers.axi_stream import AxiStreamBus

# 1. Wrap the DUT's real port (the DUT drives valid/data; you drive ready).
bus = AxiStreamBus(tvalid=dut.m_axis_tvalid_o, tready=dut.m_axis_tready_i,
                   tdata=dut.m_axis_tdata_o, tlast=dut.m_axis_tlast_o)

# 2. Attach the passive monitor BEFORE reset deasserts (so reset clauses fire).
mon = AxiStreamMonitor(bus, dut.clk_i, dut.rst_i, log=dut._log,
                       rst_active_low=False, channel="dut_m_axis")
cocotb.start_soon(mon.run())

# 3. Drive realistic stimulus — and apply BACKPRESSURE (toggle the consumer
#    ready). This is where the defects hide; happy-path tests pin ready=1.
# 4. Verdict: conformance + NON-VACUOUS coverage.
assert mon.violation_count == 0, mon.report()
assert mon.observed("AXIS_TLAST") >= 1          # the check had real traffic

Run it with the one executor (set the mem cap on a shared bench):

export RR_MEM_LIMIT_GB=60
rr sim run test_demo_<port>_bfm_conformance

A trailing ⚠️ Could not determine the simulation outcome … (RTL-P1.75) line is a benign parser quirk when TESTS=N PASS=N FAIL=0 appears above it.

3. Where the defects cluster (hunt heuristic)

From this campaign, findings concentrate on two axes — target them first:

• A DUT's MASTER OUTPUT under BACKPRESSURE. Existing tests pin the consumer ready/tready=1, so the AMBA hold rules (*_VALID_STICKY, *_DATA_HOLD) are never exercised. Toggle ready low for multi-cycle runs and a master that drives valid/data straight off a FWFT FIFO violates the instant a sink stalls → DDR-T2.9 (head word advances with no handshake → data lost), found on both the AVMM and AXI DDR backends from the same shared FIFO core.
• VALID-into-reset. An output strobe driven combinationally off an FSM state that only clears on the synchronous reset edge leaves VALID high for the first reset cycle → AXIS_RST violation. Hit three times (RTL-T2.26 regbank, DS-T2.47 framer). Start the monitor before reset deasserts and drive a reset mid-offer to catch it. Fix is a one-line reset mask.
• The TX / emit side of a protocol block (the RX side usually gets the attention) → VHD-T2.7, a 6-vs-7 Ethernet preamble off-by-one on the MAC's GMII TX wire, caught by EthIntegrityMonitor.

4. Rigor — a violation is a WIN, but earn it

The success metric is inverted: a green hunt that finds nothing is a yellow flag, not victory. But every claimed defect must survive a hostile reviewer:

• Diagnose before claiming. Real DUT defect vs test artifact (two drivers on one bus, wrong reset polarity, illegal stimulus, wrong-cycle sample)? Waveform-confirm. State confidence as proven vs hypothesis.
• Drive consumer *_ready with DEFERRED writes, never Immediate(). This is the trap that produced a false DDR-T2.9 ("M_AXIS loses the head word under backpressure"). The monitor samples post-edge (RisingEdge+ReadOnly); a cocotb Immediate() write to ready lands in that same timestep, so against a combinational-FWFT source (whose data advances at the accept edge) the monitor reads next-cycle ready against this-cycle already-advanced data → a bogus AXIS_DATA_HOLD + off-by-one drain. The bus was compliant all along. Use a small synchronous driver coroutine with sig.value = … (the AxiStreamMonitor KAT's own convention). Before claiming any backpressure DATA_HOLD/VALID_STICKY defect, confirm it survives two cross-checks: (1) re-drive ready deferred → does the real monitor still fire? (2) sample the bus at FallingEdge (mid-cycle steady state = what real synchronous hardware latches) → are all words present and in order? If both come back clean, it was the Immediate phase-skew, not the RTL — and do not "fix" it by bolting on an output register (that can reintroduce downstream bugs the bare pass-through was masking). See Haz.73 / RTL-P3.853.
• Prove non-vacuous. Assert the backpressure actually coincided with VALID (e.g. assert stalled_offers > 0) so the hold rules had something to check.
• Never weaken the assertion, the DUT, or the BFM to go green — no pinning ready=1, no loosening asserts, no skip/xfail without a paired T-ticket, no settle-sleeps to dodge a race (the race is the bug).
• Leave the oracle frozen. File the defect, then commit the reproducer as @cocotb.test(expect_fail=True) with the spec-correct assertion intact. It reports "failed as expected" while the bug lives and auto-flips to a real PASS the moment the RTL is fixed — that frozen reproducer is the finder/fixer separation, regardless of who fixes it.

5. File and track — the PM loop

# File the defect in the owning repo (T-tier = defect found by verification)
python tools/tlog.py add --repo <repo> --tier T2 --cat conformance \
  --title "..." --desc "<root cause + reproducer path>"

# Register the campaign artifact as a capability
python tools/tcap.py add --repo routertl --cat Simulation \
  --name "Conformance sweep ..." --ref P3.851 --status partial

# Record where the sweep stands so the next session (or agent) resumes cleanly
python tools/tctx.py set --repo routertl --status "..." --next "..."

# When a fix lands, close it and let the reproducer flip green
python tools/tlog.py close <REPO>-T2.N --repo <repo> --reason "fixed in <sha>; reproducer now PASS"

6. Scale it — parallel agents, uniform interface

Because the monitors, the executor, and the PM loop are identical across submodules, the hunt fans out trivially: launch one agent per DUT, each running the same from rr_cocotb_tb… import, the same rr sim run, the same tools/tlog.py. They work in different submodules (separate git indexes, no collision), and the orchestrator audits each reproducer and batches the super-repo pointer bumps. This campaign hunted five ports across four submodules this way.

The compounding is the point: every monitor, BFM, and graph node makes the next find cheaper. DDR-T2.8 cost a bespoke demo; DDR-T2.9, VHD-T2.7, and DS-T2.47 fell out of the proven recipe in parallel — and a fix made once at a generator (the RTL-T2.26 reset mask in generate_axilite.py) protects every IP that generator emits.

7. The gate — correctness is enforced, not hoped for

The pre-commit hook (sdk/infra/hooks/pre-commit, plus per-repo hooks) runs ruff, the structural guards (test_bfm_emits_kat_guard, test_conformance_graph_p2_722), and — when HDL is touched — the full cocotb sim-regression against the working tree. In this campaign that gate caught a concurrent agent's half-finished clause and a downstream test that had codified the VHD-T2.7 bug (its expected preamble was the buggy 6 octets) — aborting the commit until both were genuinely green. The gate is why "it passed" means it passed.

See also

• CONTRACT_FIRST_WALKTHROUGH — close the loop on one IP
• DRIVER_COOKBOOK — author a new BFM with EMITS + KAT
• ASSERT_QUALITY — non-vacuous, hard-coded-expected assertions
• COCOTB_QUICKSTART — the rr sim run basics