AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation
- Year
- 2025
- Authors
- Vaishnavi Pulavarthi, Deeksha Nandal, Soham Dan, Debjit Pal
- DOI
- 10.48550/arXiv.2406.18627
Related
Persistent Notes
In-text annotations
“1 Introduction:" Page 1
“However, there is no in-depth study nor a dataset to evaluate how well different state-of-the-art (SOTA) LLMs perform on generating the correct set of assertions without designer developed prompts." Page 2
“To address this gap, we propose AssertionBench: the first comprehensive benchmark to quantify the efficacy of SOTA LLMs for the assertion generation task." Page 2
“2 The AssertionBench Benchmark” Page 2
“AssertionBench1 contains ICL example designs and and test designs from OpenCores” Page 2
“3 Experimental Setup” Page 3
“We have not reported any other metric, e.g., assertion coverage (Athavale et al., 2014), as that is meaningful only when an assertion is valid and one wants to quantify the quality of the assertion or would like to induce a ranking on assertions (Pal et al., 2020). In current work, we did not target to quantify the quality of the assertions neither did we want to induce a rank on them. Rather we focused on the ability of the SOTA LLMs on generating correct assertions." Page 4
“4 Experimental Results” Page 4
“or assertion generation task (c.f., Figure 3). Our analysis shows that none of the LLM models can generate valid assertions an average of no more than 44% accuracy whereas up to 63% generated assertions produce CEX and on average up to 33% of generated assertions are syntactically wrong." Page 5
“5 Conclusion and Future Work” Page 5
“6 Limitations” Page 6
%% Import Date: 2026-07-10T11:52:06.098-04:00 %%