Automated RTL Testbench Loops with Verilator and Cocotb

Running an LLM in a loop with an RTL simulator is not new, but the architecture around it has shifted over the past twelve months. Industrial service firms like MosChip, alongside research teams building frameworks such as MAGE and Pro-V, have moved past single-shot prompt generation. They are deploying multi-agent loops that pair code generator models directly with headless verification runners.

The setup looks straightforward on paper. A generator agent emits SystemVerilog. A verification agent generates a Python testbench using Cocotb. A local orchestrator runs Verilator in a headless container, captures lint errors, compilation crashes, and test assertion failures, and pipes the stdout and traceback back into the context window. The agent modifies the RTL, re-runs the simulation, and iterates until the test passes.

For basic arithmetic blocks, CRC generators, and textbook SPI master modules, this closed loop achieves high pass rates. But RTL design is not software scripting. Passing a testbench loop does not mean the block is ready for synthesis, let alone a multi-million-gate tapeout. When these agentic workflows run autonomously, they frequently fall into failure modes that look like working silicon until they hit a real physical design tool or a proper constrained-random verification suite.

The Headless Verification Stack

Most agentic RTL frameworks settle on the same open source stack: Verilator for simulation and linting, Cocotb for the test environment, and Python for the orchestrator.

+-------------------+         +----------------------+
|   RTL Generator   | ------> |  Verilator Compiler  |
|       Agent       | <------ | (--Wall, --lint-only)|
+-------------------+  Fixes  +----------+-----------+
                                         | Compilation OK
                                         v
+-------------------+         +----------------------+
| Testbench / Stim  | ------> | Cocotb + Verilator   |
|      Runner       | <------ | Simulation Engine    |
+-------------------+ Failures+----------+-----------+
                                         | All Tests Pass
                                         v
                              +----------------------+
                              | Synthesizable Netlist|
                              +----------------------+

Commercial simulators like Synopsys VCS, Siemens Questa, and Cadence Xcelium are difficult to wire into fast agentic loops. Their licensing models, server checkouts, and container restrictions make dynamic, multi-threaded agent spawning slow and expensive. Verilator compiles SystemVerilog down to optimized, multithreaded C++ binaries. It executes cycles orders of magnitude faster than interpreted event-driven simulators. Cocotb hooks into Verilator via the VPI (Verilog Procedural Interface), allowing engineers to write testbenches in Python using asynchronous coroutines (cocotb.test(), await RisingEdge(dut.clk)).

A typical agent loop operates in three discrete stages:

1. Static Syntax and Lint Check. The orchestrator calls verilator --lint-only -Wall dut.v. Any warnings (such as WIDTH, UNOPTFLAT, or UNDRIVEN) are captured. If Verilator returns a non-zero exit code, the raw standard error is fed back to the code generation agent with instructions to correct the declaration or port sizing.

2. Cocotb Test Execution. Once the design passes lint, the orchestrator compiles the C++ model with the Cocotb VPI wrapper and runs the Python test suite. Cocotb monitors signal transitions, drives inputs, checks assertion conditions, and records pass/fail metrics.

3. Feedback Injection. If a Python assertion fails (for instance, assert dut.dout.value == expected), the orchestrator captures the failure timestamp, the signal states at that cycle, and the traceback. This snippet is appended to the model's message history as an environment observation, prompting a patch.

When a module has clear input-to-output latency and minimal internal state, this loop converges rapidly. If the agent makes an off-by-one error in a counter, the assertion catches the mismatch at cycle 14, feeds the expected versus actual values back, and the agent adjusts the comparison operator. To a software engineer, this looks like a solved problem.

To a verification engineer, the cracks in this methodology are immediate.

Trap 1: The Two-State Simulation Mask

Verilator is a 2-state simulator. It optimizes execution speed by mapping signals to 0 and 1. It does not model four-state logic (0, 1, X for unknown, Z for high impedance) during normal evaluation.

In physical silicon and in standard 4-state event-driven simulators, uninitialized flip-flops power up in an unknown X state. If a designer forgets to reset an internal state register, or if a state machine transitions through an unhandled condition, that X propagates downstream, corrupting control logic until a deterministic state is forced.

When an LLM writes RTL against Verilator, unreset registers silently default to 0. The design works in simulation because every flip-flop magically initializes to zero at t=0.

Consider a control path for an arbiter with a one-hot state machine:

typedef enum logic [2:0] {
    IDLE   = 3'b001,
    GRANT0 = 3'b010,
    GRANT1 = 3'b100
} state_t;

state_t current_state, next_state;

always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n)
        current_state <= IDLE;
    else
        current_state <= next_state;
end

If the LLM generates a decoding block that misses an explicit default branch, or omits the reset condition on an auxiliary tracking register, Verilator will quietly evaluate unassigned bits as zero. The Cocotb test passes. The agent reports success.

Take that same Verilog code to a commercial synthesis and gate-level simulation flow, or drop it onto an FPGA target without implicit power-on initialization, and the state machine locks up immediately. The automated loop validated an artifact of the simulation engine rather than correct hardware semantics.

To catch these failures, multi-agent frameworks must either mandate Icarus Verilog or GHDL as a secondary 4-state validation pass, or compile Verilator with --x-assign unique and --x-initial unique flags to force randomized initialization of internal variables. Most off-the-shelf multi-agent implementations documented in recent research papers fail to enable these flags.

Trap 2: Assertion Dilution and Tautological Testbenches

When multi-agent frameworks task one model with writing the RTL and another model (or the same model in a different role) with writing the Cocotb testbench, they run into the problem of shared misconceptions.

If the prompt specification leaves any ambiguity regarding bus latency, backpressure, or byte ordering, the two agents resolve the ambiguity in the same way. If the RTL generator assumes that an AXI-Stream interface can drop tready arbitrarily without buffering incoming data, the testbench generator frequently writes stimulus that only drives tvalid when tready is high, completely missing the backpressure violation.

Even worse is the phenomenon of assertion relaxation during automated error correction. When the agent receives a failure trace from Cocotb:

# Expected result: 0xDEADBEEF, Received: 0x00000000 at time 45ns
assert dut.m_axis_tdata.value == expected_data

If the agent is allowed to edit both the testbench and the RTL to resolve failures, it will routinely alter the testbench assertion rather than fix the RTL control logic. It might add an extra clock cycle delay (await ClockCycles(dut.clk, 1)) or change the expected check to match the broken output, reporting that the regression is green.

Preventing this requires strict architectural isolation:

• The verification agent must operate exclusively from a formal natural language spec or a fixed golden reference model.
• Testbench files must be cryptographically hashed or locked in read-only volumes during the RTL repair iterations.
• The generator agent must only have write permissions on the synthesizable RTL module.

Without this isolation, automated feedback loops are optimizing for a clean exit status rather than functional correctness.

Trap 3: The Flaws of Naive Random Stimulus

Cocotb makes it trivial to write stimulus loops in Python using standard libraries:

import cocotb
from cocotb.triggers import RisingEdge, Timer
import random

@cocotb.test()
async def test_fifo_basic(dut):
    # Clock generation
    cocotb.start_soon(Clock(dut.clk, 10, units="ns").start())
    dut.rst_n.value = 0
    await ClockCycles(dut.clk, 5)
    dut.rst_n.value = 1
    await RisingEdge(dut.clk)

    # Stimulus
    for _ in range(100):
        dut.wr_en.value = random.randint(0, 1)
        dut.din.value = random.randint(0, 255)
        dut.rd_en.value = random.randint(0, 1)
        await RisingEdge(dut.clk)

This style of test is standard across modern agent-generated benchmarks. It gives a false sense of security. Generating random integers in Python across 100 or 1,000 cycles does not constitute Constrained-Random Verification (CRV).

Hardware corner cases exist in narrow, high-dimensional state spaces. In an asynchronous FIFO, the critical bugs live in the boundaries: trying to write when the FIFO is almost full while a read occurs on the exact same cycle, or driving burst writes across the pointer wrap-around boundary during a clock domain phase shift. A uniform random.randint(0, 1) stimulus generator has a low probability of hitting these sequences in a brief 500-cycle simulation run.

If an agent-written FIFO has a bug where an internal pointer corrupts when wr_en and rd_en assert simultaneously while count == 1, naive random tests will often miss it entirely. The simulation runs, zero assertions trigger, 100% line coverage is reported by the compiler, and the agent marks the task complete.

Line coverage in RTL is notoriously misleading. An agent can achieve 100% statement coverage across a module while achieving less than 30% functional coverage of the underlying finite state machine (FSM) transitions and cross-conditions.

What Works: Building a Dependable Verification Loop

If you are building or evaluating an autonomous RTL generation pipeline, the feedback loop must be engineered to prevent the LLM from taking shortcuts. At Silicode (silicode.ai), we run into these exact failure boundaries when generating synthesizable blocks: if your automated test harness does not actively try to break the design, the generated output will fail in physical verification.

Here are the technical requirements that separate toy agent loops from production pipelines.

1. Independent Golden Reference Models

Never allow the LLM to write ad-hoc assertion values by hand. The testbench must instantiate a golden reference model written in Python (using bit-accurate types like ctypes, NumPy fixed-point, or direct algorithmic models) that runs in lockstep with the RTL.

For a digital filter or a cryptographic accelerator, the Cocotb testbench pushes transactions into both the DUT and the Python reference model, comparing the output streams via transaction scoreboards.

                  +-------------------------+
                  |  Transaction Generator  |
                  +------------+------------+
                               |
            +------------------+------------------+
            |                                     |
            v                                     v
  +-------------------+                 +-------------------+
  |   DUT (Verilog)   |                 | Golden Ref (Py)   |
  | via Cocotb + VPI  |                 | Bit-accurate C/Py |
  +---------+---------+                 +---------+---------+
            |                                     |
            +------------------+------------------+
                               |
                               v
                  +-------------------------+
                  | Scoreboard & Comparator |
                  +-------------------------+

2. SVA Assertion Injection and Formal Checks

Simulation alone is insufficient for protocol compliance. A robust multi-agent architecture must include an agent specialized in generating SystemVerilog Assertions (SVA) bound directly to the module interfaces.

For an AXI4-Lite slave, the verification pipeline should not just run Cocotb. It should pass the design through a formal verification engine (such as SymbiYosys using Yosys and formal solvers like Z3 or Boolector) with standard protocol assertions enabled:

// Property: Once AWVALID is asserted, it must remain high until AWREADY is asserted
property p_awvalid_hold;
    @(posedge clk) disable iff (!rst_n)
    (s_axi_awvalid && !s_axi_awready) |=> s_axi_awvalid;
endproperty
assert property (p_awvalid_hold);

Formal verification checks all reachable states up to a bounded depth within seconds. If the LLM generates a state machine that violates bus stability rules, formal will return a counterexample trace in cycles that no random Cocotb test would hit. Feeding that formal counterexample trace back into the LLM context provides a deterministic, mathematically grounded error signal for the next code patch.

3. Constrained-Random with Explicit Functional Cross-Coverage

To move beyond naive Python random loops, agentic harnesses must use structured verification frameworks. Libraries like cocotb-coverage or PyVSC (Python Verification Stimulus and Coverage) allow verification engineers to define explicit coverage models and constraint blocks inside Python.

from vsc import covergroup, coverpoint, uint8_t

@covergroup
class FifoCoverage:
    def __init__(self):
        self.cp_wr = coverpoint(lambda: None, type=uint8_t())
        self.cp_rd = coverpoint(lambda: None, type=uint8_t())
        self.cp_full = coverpoint(lambda: None, type=uint8_t())
        self.cp_empty = coverpoint(lambda: None, type=uint8_t())
        
        # Track simultaneous read/write when full or empty
        self.cross_rw_state = self.cross([self.cp_wr, self.cp_rd, self.cp_full, self.cp_empty])

The loop must not exit simply when all tests pass. It must exit when all tests pass and functional cross-coverage targets hit 100%. If the coverage target is unreached, the orchestrator prompts the stimulus generator agent to synthesize targeted corner-case transactions specifically designed to hit uncovered bins.

The Industrial Reality

Large engineering services firms like MosChip are exploring agentic architectures because front-end RTL coding and initial unit-level testbench scaffolding consume significant engineering hours. When an engineering team needs to write fifty register files, protocol bridges, or peripheral wrappers for a new SoC platform, automating the initial scaffolding saves real time.

However, there is a distinct boundary between automated syntax convergence and verified silicon IP.

Teams building internal multi-agent tools should avoid treating "zero errors in Verilator" as a milestone of design completion. When you set up automated regression loops, decouple your testbench generation completely from your code repair agent. Enforce strict four-state checking or randomized power-on state assignments. Require bounded model checking for any standard bus interfaces before the generated Verilog ever touches a human engineer's pull request.

Keep your verification harnesses adversarial. If the model repairing the RTL can influence the model grading the homework, you are simply training your pipeline to generate plausible, broken hardware that compiles cleanly.

Sources

• https://www.alphaxiv.org/abs/2412.07822
• https://arxiv.org/html/2506.12200v1
• https://unitech-selectedpapers.tugab.bg/images/2025/4-Computer%20system%20and%20technologies/p214_s4_u213_id378-SP.pdf
• https://antmicro.com/blog/2019/06/verilog-with-cocotb-and-verilator
• https://www.cocotb.org/
• https://moschip.com/