Nemotron 3 Ultra at 97 Percent on Agentic RTL

NVIDIA claims 97.1% pass rates with Nemotron 3 Ultra on the CVDP benchmark. We break down the token economics, CDC risks, and agent loop realities.

NVIDIA recently published results showing its Nemotron 3 Ultra model achieving a 97.1% pass rate on the Comprehensive Verilog Design Problems (CVDP) benchmark. The workflow does not rely on single-shot code completion. It runs an iterative agentic loop where the model writes RTL, parses compiler and simulator output, and rewrites the module until tests pass, consuming an average of 6,629 tokens per iteration.

To make this agent loop computationally viable, NVIDIA paired a Mixture-of-Experts (MoE) routing layer with a hybrid Mamba-Attention backbone. They claim up to five times higher inference throughput and 30% lower generation cost compared to dense, pure-transformer open models.

For design and verification engineers who have spent the last three years testing LLM-generated Verilog, high benchmark numbers provoke immediate scepticism. We have seen models score well on VerilogEval by writing self-contained 4-bit counters, shift registers, and basic arithmetic logic units, only to completely fall apart when dropped into an existing IP block with complex reset schemes, multi-cycle paths, and strict synthesis guidelines.

Evaluating Nemotron 3 Ultra requires looking past the 97.1% headline. We need to evaluate the token cost per resolved issue, how the agent loop handles non-trivial state spaces, and whether the resulting Verilog survives static linting, clock domain crossing (CDC) analysis, and physical synthesis.

Moving Past Toy Benchmarks to CVDP

Early LLM hardware evaluations suffered from trivial test suites. VerilogEval and similar early datasets tested whether a model could emit syntax-compliant Verilog for textbook problems. A standard GPT-4 or Claude 3.5 Sonnet instance can write a parameterized FIFO or a round-robin arbiter without breaking a sweat. The training data for those modules is abundant on GitHub.

The CVDP benchmark, co-developed through Si2's LLM Benchmarking Coalition, represents a necessary shift. Instead of isolated functions, CVDP measures nine distinct categories of RTL work:

Leaf-module RTL generation from natural language specifications
Bug localization and functional debugging from simulation trace logs
Code modification and feature extension on existing codebases
Generation of directed and constrained-random testbenches
Assertion-based verification (SVA) writing
Synthesis warning and lint error resolution
Refactoring for timing closure and critical path reduction
Multi-module hierarchy integration
Protocol translation and bus wrapper generation (AXI, APB, Wishbone)

Scoring 97.1% across these categories using an iterative agent is technically impressive. But the core operational question is what happens inside those 6,629 tokens per iteration, and how many iterations a real engineering team must pay for when a module fails verification.

The Hybrid Architecture: Why Mamba Matters for EDA Loops

In a standard Transformer, attention scales quadratically with sequence length. When an agentic tool runs an RTL design loop, the context window fills rapidly. The prompt does not just contain the spec and the current Verilog module. It accumulates compiler error logs, Verilator warning traces, waveform value change dump (VCD) excerpts, and previous failed diffs.

By the fourth iteration, the context history easily exceeds 32,000 tokens. In a standard multi-head attention model, calculating the key-value (KV) cache for every regeneration step becomes a major bottleneck. The throughput drops, and the cost per generated line of RTL spikes.

Nemotron 3 Ultra replaces most attention layers with State Space Model (SSM) blocks based on the Mamba architecture, retaining attention layers only at periodic intervals for global token recall. Mamba processes sequential inputs with linear time complexity and constant memory footprint during inference.

Agent Iteration Context Growth
-------------------------------------------------------------------------
Pass 1: Spec + Interface Def + Initial RTL Gen             (~4,000 tokens)
Pass 2: Above + Verilator Lint Log + SVA Failures           (~9,500 tokens)
Pass 3: Above + ModelSim Waveform Trace + Reg Diff         (~16,000 tokens)
Pass 4: Above + Formal Counterexample Trace                (~24,000 tokens)
-------------------------------------------------------------------------
Standard Transformer: O(N^2) compute, exploding KV cache memory
Nemotron 3 Mamba-MoE: O(N) compute, selective state updates

For iterative hardware generation, this structural shift is practical. The model can digest a 5,000-line simulation log without blowing up inference latencies. The MoE routing further ensures that only a fraction of parameters are active per token, keeping generation speeds high enough for near-interactive terminal use.

Throughput alone does not guarantee hardware correctness. Cheap tokens are useless if the generated logic contains subtle race conditions.

The Functional Bug Density Problem

When Nemotron 3 Ultra operates in an agent loop, it relies heavily on simulator feedback to fix its mistakes. If you pair the model with Verilator or Icarus Verilog, it will loop until $fatal is not triggered and the testbench reports success.

This creates a classic optimization trap: the model learns to satisfy the testbench rather than the specification.

We frequently see models resolve a simulation mismatch by hardcoding cycle-accurate delays or adding priority muxes that mask the root cause. If a testbench only exercises eight transactions, the agent will happily patch the RTL with an ad-hoc counter that matches the testbench sequence, passing the benchmark while leaving the underlying state machine completely broken for corner cases.

Finite State Machine Pathologies

In CVDP testing, multi-state sequential controllers remain the most failure-prone area. LLMs struggle with three specific FSM structures:

Deeply nested hierarchical state machines with asynchronous abort conditions.
Mealy outputs that create combinational paths between inputs and outputs across module boundaries.
One-hot state encodings where default branches fail to properly recover from illegal states.

Consider an arbiter designed to manage credit-based flow control across four channels. When Nemotron generates the RTL, it easily constructs the IDLE, GRANT, and WAIT states. However, when asked to implement credit starvation timeouts during an active transaction, the agent often inserts synchronous reset branches that inadvertently clear credit counters for idle channels.

// Typical agent-generated bug: blind counter reset on timeout
always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
        state           <= IDLE;
        credit_count    <= '0;
        timeout_counter <= '0;
    end else begin
        case (state)
            ACTIVE: begin
                if (timeout_triggered) begin
                    state        <= IDLE;
                    // Bug: clears all credits instead of preserving current balance
                    credit_count <= '0; 
                end else if (tx_valid && tx_ready) begin
                    credit_count <= credit_count - 1'b1;
                end
            end
            // ... other states
        endcase
    end
end

If the CVDP verification testbench does not specifically test credit retention following an abort, the agent records a 100% pass rate. The bug only surfaces during system-level regression or after synthesis.

The Invisible RTL Failures: CDC, Lint, and Inferred Latches

Functional correctness in simulation is only half the battle. Synthesizable RTL must obey electrical and structural rules that simulation testbenches routinely ignore.

Clock Domain Crossing (CDC)

LLMs do not have an internal graph model of clock trees. When prompted to pass a multi-bit control vector from clk_a (100 MHz) to clk_b (250 MHz), Nemotron 3 Ultra frequently generates a standard two-stage flip-flop synchronizer directly on the multi-bit bus.

// Dangerously generated by LLMs for multi-bit CDC:
always_ff @(posedge clk_b or negedge rst_n) begin
    if (!rst_n) begin
        sync_stage1 <= '0;
        sync_stage2 <= '0;
    end else begin
        sync_stage1 <= async_data_bus; // Data coherency violation!
        sync_stage2 <= sync_stage1;
    end
end

Any digital designer knows that passing a multi-bit bus through a 2-FF synchronizer without Gray coding, a handshake protocol, or an asynchronous FIFO causes bus skew and data coherency failures. Yet, because a standard testbench runs in an idealized zero-delay simulation environment with synchronized clock sources, this code simulates perfectly. It will pass CVDP leaf-level tests every single time.

To make agentic RTL generation trustworthy, the agent's inner verification loop cannot just run a logic simulator. It must run static CDC checkers like Synopsys SpyGlass CDC or Questa CDC, and feed those structural rule violations back into the LLM context window.

Inferred Latches and Case Completeness

Another routine failure mode in open-model generation is the accidental inference of transparent latches in combinational always blocks. Nemotron 3 Ultra performs better than older models like VeriGen on this metric, but complex combinational decoding logic still exhibits incomplete assignments.

// Incomplete case causing inferred latch
always_comb begin
    next_state = current_state;
    alu_out    = 32'h0;
    case (opcode)
        OP_ADD: alu_out = reg_a + reg_b;
        OP_SUB: alu_out = reg_a - reg_b;
        OP_BRANCH: begin
            if (condition_met) 
                next_state = TARGET_STATE;
            // Missing else branch leaves next_state latched under certain conditions
        end
        // Missing default case
    endcase
end

When we integrate models like this into Silicode, our primary architectural hurdle is preventing the agent from concluding its task simply because compilation succeeded. The agent loop must enforce zero-warning policies under strict flags (-Wall -Wextra in Verilator) and run formal property verification (FPV) on bounded state transitions before presenting the code to a human.

The Real Token Budget: How the Economics Shake Out

NVIDIA's reported average of 6,629 tokens per iteration provides a concrete baseline for analyzing engineering costs. Let us look at what a real-world multi-pass refinement cycle actually costs in compute and human oversight.

Suppose you are implementing an AXI4-Lite slave wrapper for an internal accelerator register file, consisting of 600 lines of synthesizable SystemVerilog with 12 distinct control registers, read/write strobes, and auto-clearing status bits.

Phase	Iterations	Context Ingested	Output Generated	Total Tokens
1. Interface & Register Decoding	2	8,000	2,200	10,200
2. Handshake State Logic	3	18,500	3,100	21,600
3. Lint & Unused Bit Fixes	2	14,000	1,800	15,800
4. Testbench Assertion Failures	4	36,000	4,200	40,200
5. Timing/Logic Optimization	1	12,000	1,200	13,200
Total Workflow	12	88,500	12,500	101,000

At 101,000 tokens for a fully resolved, verified sub-block, the inference cost is negligible (pennies on modern cloud hardware, or a few seconds of local H100 execution). The true cost lies entirely in verification sign-off.

If an RTL engineer spends 45 minutes setting up the harness, reviewing the 12 iterations of agent history, checking the formal bounds, and inspecting the generated code for unclocked reset glitches, they have saved about two to three hours of manual boilerplate writing. That is a real, measurable productivity gain.

If the agent introduces an unconstrained multi-cycle path that slips through to place-and-route, debugging that failure in post-layout gate-level simulation will consume two full days of engineering time. The economic return of agentic RTL hinges entirely on whether the validation guardrails catch structural hardware flaws before the module leaves the leaf-level workspace.

Formal Property Verification as the Missing Link

To move agentic RTL from interesting benchmark demonstrations to production tape-out flows, the industry must decouple the LLM from naive unit testbenches. Relying on an agent to write its own testbench creates a confirmation bias loop: the model generates a buggy implementation, writes a matching buggy verification suite, confirms that all tests pass, and reports complete success.

The only mathematically sound method to close this loop is automated Formal Property Verification (FPV).

Instead of asking Nemotron to emit directed test vectors, the supervisory harness must force the model to write SystemVerilog Assertions (SVA) derived strictly from the natural language interface contract. A model checker (like SymbiYosys, Cadence JasperGold, or Siemens Formal) then runs bounded model checking against the generated RTL.

// Formal properties that the agent must satisfy
property p_never_simultaneous_grant;
    @(posedge clk) disable iff (!rst_n)
    $onehot0({grant_ch0, grant_ch1, grant_ch2, grant_ch3});
endproperty
assert_mutual_exclusion: assert property (p_never_simultaneous_grant);

property p_credit_underflow_check;
    @(posedge clk) disable iff (!rst_n)
    (credit_count == 0 && !credit_return) |-> !tx_ready;
endproperty
assert_no_underflow: assert property (p_credit_underflow_check);

When the formal engine finds a violation, it generates a counterexample trace. That trace file, translated into a step-by-step state assignment log, becomes the prompt input for the next agent iteration. This strips away ambiguous simulator printouts and gives the Mamba-MoE architecture a deterministic target: eliminate the specific state transition that triggered the assertion failure.

Where Nemotron 3 Ultra Fits in the Design Stack

Nemotron 3 Ultra demonstrates that specialized hybrid architectures can dramatically improve inference economics for long-context hardware design problems. Achieving a 97.1% score on CVDP proves that when paired with an iterative compiler loop, these models can solve complicated structural syntax problems, resolve basic lint errors, and implement standard digital interfaces.

Treating that 97.1% metric as a sign that the model can autonomously design ASICs is a mistake.

Hardware design is defined by edge cases, physical constraints, electrical boundaries, and asynchronous interactions. An LLM running in a loop against an open-source simulator will happily emit Verilog that satisfies functional tests while violating basic clock domain rules, inferring unintended latches, and blowing up critical path margins.

For design teams evaluating Nemotron 3 Ultra or similar agentic systems, the practical implementation path is clear:

Confine the model to bounded leaf modules: protocol converters, bus shims, register banks, and standard datapath blocks.
Bind the generation loop to rigorous static analysis tools. Do not let the agent accept code that only passes functional simulation. Force it to clear zero-warning lint checks and CDC analysis.
Make SystemVerilog Assertions the primary communication channel between the verification environment and the generation model.
Treat LLM output as untrusted third-party IP. Every module requires human review of its sensitivity lists, clock routing, and reset behavior before integration into top-level SoC fabrics.

The token budget is finally cheap enough to run exhaustive agent loops. The remaining task is building the rigorous structural harness needed to keep the generated silicon from failing in the fab.