Descripción:

Nowadays, many software companies are required to meet high quality standards in the development of their applications. Rather than replacing traditional SAST tools based on fixed rules, approaches such as GemCA leverage a Large Language Model (LLM) to perform semantic reasoning over code in relation to the ISO/IEC 5055:2021 standard, providing a complementary decision-support mechanism for early-stage quality assessment. This paper presents GemCA and reports an empirical analysis evaluating its accuracy on a dataset of known code weaknesses. The goal is to examine the framework’s ability to apply the criteria defined by the ISO/IEC 5055:2021 standard. Results show that GemCA achieves an average accuracy of 81% across 15 repetitions, with significantly higher performance in C# than PHP. However, accuracy varies substantially across weakness categories (p = 0.0000), indicating sensitivity to CWE type and programming language. These findings highlight both the potential and the current limitations of LLM-based analysis for ISO/IEC 5055:2021 compliance.

Tipo de publicación: Conference Paper

Publicado en: Anais do XXIX Congresso Ibero-Americano em Engenharia de Software (CIbSE 2026)