CALDERA AI

Splunk's open-source security scanner for analyzing AI agent codebases. Detects vulnerable patterns like unsanitized tool outputs, missing guardrails, and over-privileged tool permissions. Generates visual dependency graphs and security reports.

IBM's comprehensive toolkit for detecting and mitigating bias in AI datasets and models. Contains 70+ fairness metrics and 11 bias mitigation algorithms. Increasingly applied to evaluate and correct bias in LLM-powered decision systems.

Open-source LLM prompt injection and security scanner with multiple detection strategies. Combines vector similarity, YARA rules, and canary token techniques for multi-layered defense. Deployable as a local REST API alongside your inference pipeline.

Crowdsourced dataset and game for studying prompt injection attacks and defenses. Players submit attack and defense prompts that get evaluated against each other. Has generated one of the largest public datasets of adversarial prompt interactions.

Microsoft's open-source data protection and de-identification SDK for detecting and anonymizing PII in text. Supports 40+ entity types and 15 languages. Used as a preprocessing layer to scrub sensitive data before sending to LLMs.

The definitive security checklist for LLM applications, covering prompt injection, insecure output handling, supply chain vulnerabilities, and more. Published by OWASP and maintained by a global community of AI security practitioners. The starting point for any LLM security audit.