SALUS IQ: Technical White Paper & Benchmark Report

2. Related Work

2.1 Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) systems enhance large language models (LLMs) with external knowledge retrieval [1, 2]. Recent advances include:

• Dense retrieval: Dense Passage Retrieval (DPR) [3] and ColBERT [4]
• Hybrid approaches: Techniques that combine dense and sparse signals [5, 6]
• Reranking: Cross-encoders and other learned rankers [7]

SALUS IQ builds on these foundations with safety-specific adaptations for construction contexts.

2.2 Domain-Specific AI Systems

Specialized AI systems have demonstrated success in high-stakes domains:

• Medical: Med-PaLM 2 achieved expert-level medical question answering performance [8]
• Legal: The Reasoning-Focused Legal Retrieval Benchmark introduced domain-specific retrieval challenges [9]
• Scientific: Galactica demonstrated strong performance on scientific reasoning tasks [10]

However, construction safety presents unique challenges not directly addressed by these systems.

2.3 Safety AI Benchmarks

Research in safety AI for construction is limited compared to other fields. Prior efforts include:

• Accident cause classification using Word2Vec and deep learning [11]
• Incident report analysis with machine learning [12]
• Personal protective equipment (PPE) detection in images [13]

To our knowledge, SALUS-SafetyQA is the first comprehensive QA benchmark designed specifically for construction safety.

5. Evaluation Methodology

5.1 Benchmark Harness

Evaluation is performed with a dedicated script that:

1. Loads questions from JSON
2. Calls the SALUS IQ benchmark endpoint or model provider endpoint
3. Records selected answer, correctness, reasoning, retrieval stats
4. Computes metrics: accuracy, average confidence, retrieval usage

5.2 Comparative Baselines

We evaluate:

• SALUS IQ (domain expert LLM system)
• GPT‑5 (Api) (OpenAI)
• Claude 4.1 Opus (Anthropic)
• Claude 4.5 Sonnet (Anthropic)
• Gemini 2.5 Pro (Google DeepMind)

All baselines receive identical multiple-choice prompts with zero-shot evaluation.

5.3 Metrics

• Accuracy: Percentage of correct answers
• Confidence calibration: Mean confidence split by correctness
• Retrieval statistics: Average docs retrieved, percentage using context
• Statistical significance: Bootstrap confidence intervals and McNemar's test (α=0.05)

6. Results

6.1 Overall Performance

Table 1: Overall benchmark results on SALUS-SafetyQA (n=1,023 questions). SALUS IQ shows 13.78 pp improvement over GPT-5, 13.10 pp over Claude 4.1 Opus, 18.96 pp over Claude 4.5 Sonnet, and 15.05 pp over Gemini 2.5 Pro. All comparisons are statistically significant (p < 0.001, McNemar's test).

6.2 Performance Visualizations

6.3 Statistical Significance

All pairwise comparisons between SALUS IQ and baseline models show highly significant differences (McNemar's test, p < 0.001):

Table 2: McNemar's test results showing pairwise comparisons. On questions where models disagreed, SALUS IQ answered correctly in 84.9%–87.9% of cases (mean: 85.6%), demonstrating consistent superiority across all comparisons.

8. Transparency & Data Availability

8.1 What We Publish

• Full benchmark dataset (1,023 questions) with question text, options, correct answers, and metadata
• Complete evaluation harness (evaluate_benchmark.py) and configuration files
• Statistical analysis code with bootstrap confidence intervals and McNemar's tests

8.2 What We Do Not Publish

• Source document content or page snippets (proprietary corpus)
• Document titles and internal database identifiers
• Raw document files or training data

Note: Questions are designed to be answerable with publicly available safety documentation (SDS, OSHA regulations, equipment manuals) to enable independent evaluation.

8.3 Data Availability

The SALUS-SafetyQA benchmark is released under CC-BY-4.0 license:

• Dataset & Code: github.com/Salus-Technologies/SALUS-SafetyQA
• License: Creative Commons Attribution 4.0 International

8.4 Limitations

• English-heavy corpus: Incomplete coverage for non-US regulations
• Domain shift risk: New standards and equipment revisions may not be represented
• Multiple-choice format: May overestimate performance relative to open-ended extraction
• Temporal constraints: Safety regulations and standards change over time

9. Conclusion

We present SALUS IQ, a domain-specialized construction safety AI system, and introduce the SALUS-SafetyQA Benchmark containing 1,023 expert-validated questions. Results show that tailored RAG pipelines significantly outperform general-purpose LLMs in high-stakes safety domains, with 13-19 percentage point improvements and 100% document grounding.

The benchmark reveals systematic failure modes in frontier LLMs: hallucination of plausible but incorrect specifications, poor performance on equipment manuals, and overconfidence calibration. These findings demonstrate the critical need for domain-specific systems in safety-critical applications.

Future work includes expanding jurisdictional coverage, incorporating multimodal inputs (drawings, diagrams), and developing open-ended evaluation protocols.

References

1. Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.
2. Guu, K., et al. REALM: Retrieval-Augmented Language Model Pre-Training. ICML, 2020.
3. Karpukhin, V., et al. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP, 2020.
4. Khattab, O., & Zaharia, M. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR, 2020.
5. Ma, X., et al. Query Rewriting for Retrieval-Augmented Large Language Models. arXiv:2305.14283, 2023.
6. Ma, X., et al. Fine-Tuning LLaMA for Multi-Stage Text Retrieval. arXiv:2310.08319, 2023.
7. Nogueira, R., & Cho, K. Passage Re-ranking with BERT. arXiv:1901.04085, 2019.
8. Singhal, K., et al. Towards Expert-Level Medical Question Answering with Large Language Models. arXiv:2305.09617, 2023.
9. Zheng, C., et al. A Reasoning-Focused Legal Retrieval Benchmark. arXiv:2505.03970, 2025.
10. Taylor, R., et al. Galactica: A Large Language Model for Science. arXiv:2211.09085, 2022.
11. Zhang, F., et al. A Hybrid Structured Deep Neural Network with Word2Vec for Construction Accident Causes Classification. Complex & Intelligent Systems, 2019.
12. Tixier, A., et al. Application of Machine Learning to Construction Injury Prediction. Automation in Construction, 2016.
13. Fang, Q., et al. Detecting Non-Hardhat Use by a Deep Learning Method from Far-Field Surveillance Videos. Automation in Construction, 2018.