OpenAI’s GPT 4o has been recognized as the top AI model for writing Solidity smart contract code by IQ. BrainDAO has introduced SolidityBench, a benchmarking tool to evaluate LLMs in Solidity code generation. This tool includes innovative benchmarks like NaïveJudge and HumanEval for Solidity to assess and rank the proficiency of AI models in generating smart contract code.
NaïveJudge tasks LLMs with implementing smart contracts based on detailed specifications from audited OpenZeppelin contracts. The generated code is evaluated based on criteria such as functionality, adherence to Solidity best practices, security standards, and optimization efficiency. Advanced LLMs like OpenAI’s GPT-4 and Claude 3.5 Sonnet are used as impartial code reviewers to assess the code comprehensively.
Among the AI models evaluated, OpenAI’s GPT-4o emerged as the top performer with an overall score of 80.05. Other models like OpenAI’s o1-preview and o1-mini, as well as models from Anthropic and XAI, also demonstrated competitive performance. Nvidia’s Llama-3.1-Nemotron-70B scored the lowest in the top 10.
HumanEval for Solidity, adapted from OpenAI’s HumanEval benchmark, includes 25 tasks of varying difficulty to measure the model’s success rate on initial attempts and over multiple tries. This provides insights into both precision and problem-solving capabilities of AI models in Solidity development.
SolidityBench aims to advance AI-assisted smart contract development by refining AI models, promoting best practices, and setting new standards for smart contract development in the blockchain ecosystem. Developers, researchers, and AI enthusiasts are encouraged to explore and contribute to SolidityBench to drive continuous improvement in AI models and decentralized applications.
Overall, the initiative by IQ Code and BrainDAO addresses the increasing demand for secure and efficient smart contracts in the industry. By leveraging AI models like OpenAI’s GPT-4o, developers can enhance their capabilities in Solidity code generation and contribute to the evolution of blockchain applications.
To learn more about SolidityBench and benchmarking Solidity generation models, visit the leaderboard on Hugging Face. This tool provides valuable insights into the capabilities and limitations of AI models in smart contract development, paving the way for more sophisticated and reliable applications in the blockchain space.
