LADBench: A Benchmark for Logical Anomaly Detection in Images

Overview

Large Vision Language Models (VLMs) excel at visual question answering and semantic grounding, but their capacity for autonomous logical reasoning remains underexplored. Existing anomaly benchmarks emphasize visual errors or direct prompting rather than the physical and social common sense needed for open-world deployment. To address this, we introduce LAD-bench, a benchmark of more than 1,000 curated synthetic images with logical anomalies across four domains: Residential, Urban, Collaborative, and Nature. We further propose a Tiered Prompting Protocol based on progressive disclosure, which measures how much explicit assistance a model needs to localize and reason about a logical fault. Evaluating leading foundation models reveals substantial weaknesses: even the best achieves only 70.11% overall accuracy, showing that implicit logical fault detection remains unsolved. Crucially, models often fail to identify anomalies even after receiving explicit hints in deeper tiers. By surfacing these limitations in sequential multimodal reasoning, LAD-Bench offers a rigorous framework for advancing the safety, reliability, and cognitive alignment of autonomous visual systems.

Dataset

The dataset for LAD-Bench is fully synthetic and contains over 1,000 images generated using OpenAI’s gpt-image-1 model, selected for its image generation quality, speed, and prompt fidelity. Using synthetic images also reduces the risk of overlap with existing training data, since the samples are newly created. This also makes adding to the dataset simpler. The images are organized into both a super category and a sub category. The super categories are Nature, Collaborative, Urban, and Residential, and are designed to consider settings of human inhabitation or frequent human interaction. The sub categories are more specific, and help in organizing images into specific settings or processes associated with humans in the super categories. A small dataset of 100 negatives, where no anomaly is present, is also provided.

The images are spread accross a number of real world categories. Anomalies can range from subtle to obvious. Examples are shown below.

Approach

The benchmark uses a Tiered Prompting Protocol, where there are three levels of progressive disclosure and assistance. Level 1 provides no assistance or companion prompt with the image, and holds the highest weightage in results. Level 2 provides explicit disclosure about the existence of an anomaly, and rewards 66.67% towards their overall accuracy. Level 3 provides a hint, which uses the sub category to direct the model’s attention, and awards 33.33% towards their overall accuracy. If the model fails after all this, it gets 0% for that image.

To automate grading, we use OpenAI’s gpt-5-nano as our grading module. It is given ground truth label, the response to be graded, and some other instructions to explain the task at hand. The grading and prompting protocol is slightly different between the pictures with anomalies and without. For those with anomalies, an image is only tested at a level if it failed the previous level. If it succeeds at one level, it will not be passed for the remaining levels. For those without, an image is tested at all three levels. There is a high rate of success at level 1, since there’s neither direction nor anomaly. For level 2, we tell the model that there is an anomaly in this image, which also misdirects them. At level 3, we ask the models to look carefully and tell for certain if there is any anomaly in this image. We graded all responses for this dataset manually, and did not include it in our paper’s published results.

Results

Accuracy comparison across models on the evaluation dataset. Results are reported by category (Residential, Urban, Collaborative, Nature) and reasoning levels (Level 1-3), along with overall weighted and tuned accuracies. Models are compared based on weighted accuracy. Performance generally improves at Level 2 with added hints but declines at Level 3. Toned accuracy reflects adjustments from human-verified false positives and false negatives.

Table: Accuracy comparison across models on the evaluation dataset

Acknowledgments

This work is supported by NSF Award DGE #2235102 (CyberCorps Scholarship for Service: Cybersecurity Research and Education for Service in Government - CREST), the NVIDIA Academic Grant Program.

BibTeX

@misc{kondapalli2026ladbenchbenchmarklogicalfault,
    title={LADBench: A Benchmark for Logical Fault Detection in Images}, 
    author={Sahasra Kondapalli and Lara Radovanovic and Aadi Palnitkar and Mingyang Mao and Xiaomin Lin},
    year={2026},
    eprint={2606.17433},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2606.17433}, 
}