ON-PREMISE OPEN-WEIGHT LARGE LANGUAGE MODELS FOR INTERPRETABLE WEB VULNERABILITY DETECTION: A BENCHMARK AGAINST CLOUD MODELS AND RULE-BASED STATIC ANALYSIS

Аннотация

Organisations handling regulated or proprietary source code are frequently prohibited from transmitting it to third-party inference providers, which excludes commercial cloud large language models from their security-analysis pipelines regardless of the accuracy those models achieve. Whether self-hosted open-weight models can substitute for them on ordinary developer hardware has not been established. This study benchmarks five quantised open-weight models, from 1.5 to 8 billion parameters and executed entirely on a consumer laptop graphics processor with eight gigabytes of video memory, on 120 annotated web-application code snippets spanning five weakness categories. Each was evaluated under zero-shot, few-shot and chain-of-thought prompting and compared against a rule-based static analyser and previously published cloud-model results on the identical corpus. The strongest local configuration reached an F1-score of 0.900, exceeding the previous-generation cloud flagship and trailing the current one by 4.7 points, while generating a remediation suggestion for every correct detection. The analysis further shows that F1 alone conceals degenerate behaviour: several configurations, including one cloud model, approach the score of a classifier labelling every input as vulnerable, so reporting specificity alongside F1 is proposed as a minimum requirement. Privacy-preserving on-premise vulnerability analysis is thus attainable without specialised infrastructure.