AI Agent Security, Guardrails & Red Teaming

Microsoft's managed content moderation API for detecting hate speech, violence, self-harm, and sexual content in LLM inputs and outputs. Provides severity scores across multiple harm categories. Deployed at scale in Microsoft's own AI products.

Open-source LLM prompt injection and security scanner with multiple detection strategies. Combines vector similarity, YARA rules, and canary token techniques for multi-layered defense. Deployable as a local REST API alongside your inference pipeline.

Microsoft's open-source data protection and de-identification SDK for detecting and anonymizing PII in text. Supports 40+ entity types and 15 languages. Used as a preprocessing layer to scrub sensitive data before sending to LLMs.

IBM's comprehensive toolkit for detecting and mitigating bias in AI datasets and models. Contains 70+ fairness metrics and 11 bias mitigation algorithms. Increasingly applied to evaluate and correct bias in LLM-powered decision systems.

Enterprise AI security platform providing real-time governance for LLM applications. Monitors all AI interactions for data leakage, prompt injection, and policy violations. Provides a control plane for enforcing AI usage policies across the organization.

Pangea's API service for inspecting LLM prompts and responses for malicious content, PII, and prompt injections. Integrates as middleware in any LLM pipeline with a single API call. Includes a library of pre-built detectors across 15 risk categories.

Open-source tool for evaluating LLM performance, hallucination, and safety across different models and prompt strategies. Generates comparative HTML reports for sharing with stakeholders. Designed for teams doing pre-deployment LLM quality assurance.

Cleanlab's Trustworthy Language Model — an API wrapper that adds calibrated confidence scores to any LLM response. Flags unreliable answers before they reach users. Used in RAG pipelines to detect hallucinations and uncertain responses automatically.

NVIDIA toolkit for adding programmable guardrails to LLM-based conversational agents.

LM Studio local inference with built-in model safety configuration, letting developers test guardrail prompts offline before production deployment. Supports Modelfile-based safety tuning.

Login security API that detects fraudulent and suspicious logins using ML and notifies users in real time. Free for 1,000 logins/month — useful for agent-facing auth flows.

Open-source framework for adding structural validation and semantic safety checks to LLM outputs. 50+ built-in validators for PII detection, toxicity filtering, JSON schema validation, and factual consistency. Programmable guard specs define pass/fail/fix behavior.

Meta's open-source safety classifier fine-tuned to categorize harmful content in both LLM inputs and outputs. Covers 11 hazard categories including violence, hate speech, and self-harm — can be deployed as a pre/post-processing filter in any agent pipeline.

NVIDIA's open-source toolkit for adding programmable guardrails to conversational AI. Define topical restrictions, fact-checking rails, and output safety checks in Colang — a purpose-built language for dialogue management. Integrates with LangChain and most LLM providers.

Security toolkit with prompt injection detection, PII anonymization, toxicity filtering, and output validation. Input/output scanner architecture runs independently or inline. Docker-ready REST API deployment. Scanner library spans 20+ security and safety checks.

Self-hardening prompt injection detector that uses a second LLM to evaluate whether inputs contain injection attempts. Learns from successful attacks to improve detection over time. Provides Canary word injection to detect exfiltration attempts.

Meta's open-source LLM-based safety classifier for human-AI conversations. Fine-tuned Llama model for detecting unsafe content in both inputs and outputs. Covers 14 hazard categories. Can be self-hosted or used via API for privacy-preserving content moderation.

Security scanning platform for finding and remediating vulnerabilities in LLM prompts and agents before deployment. Performs static analysis of system prompts, identifies injection risks and data exfiltration vectors, and provides remediation guidance tailored to the specific agent architecture.

Microsoft Azure's content moderation API detecting hate speech, violence, sexual content, and self-harm in text and images. Prompt Shield blocks prompt injection and jailbreak attacks. Groundedness detection identifies hallucinations. Enterprise-grade with compliance certifications.

Real-time AI security middleware protecting against prompt injection, data leakage, and harmful content. API-based with sub-50ms latency. Continuous learning from emerging threats. Enterprise features include audit logs, dashboards, and policy management.

AI control platform with guardrails, monitoring, and policy enforcement for production LLM applications. Real-time intervention to block harmful outputs, enforce brand voice, and detect sensitive data exposure. Supports any LLM with a drop-in proxy deployment.

Microsoft's unified library for evaluating LLM robustness against adversarial prompts. Tests models against typos, text styles, sentence structures, and semantic perturbations. Provides a standardized way to measure how much prompt variation degrades model performance.

Crowdsourced dataset and game for studying prompt injection attacks and defenses. Players submit attack and defense prompts that get evaluated against each other. Has generated one of the largest public datasets of adversarial prompt interactions.

Splunk's open-source security scanner for analyzing AI agent codebases. Detects vulnerable patterns like unsanitized tool outputs, missing guardrails, and over-privileged tool permissions. Generates visual dependency graphs and security reports.

The definitive security checklist for LLM applications, covering prompt injection, insecure output handling, supply chain vulnerabilities, and more. Published by OWASP and maintained by a global community of AI security practitioners. The starting point for any LLM security audit.

MIT's database of documented AI risk incidents and failure modes. Catalogues real-world cases of AI system failures for researchers and practitioners. Essential reference for threat modeling and risk assessment of deployed AI agent systems.

Carnegie Mellon paper demonstrating universal, transferable adversarial suffixes that cause aligned LLMs to generate harmful content. Shows that safety training can be bypassed with automated suffix search. Among the most important safety research papers of 2023.

NVIDIA's open-source LLM vulnerability scanner. Probes models and agents for jailbreaks, prompt injections, hallucinations, data leakage, and harmful content generation. 100+ attack probes across multiple categories. Reports detailed findings with severity ratings.

Microsoft Azure's Python Risk Identification Toolkit for adversarially testing generative AI systems. Automates red-teaming with multi-turn conversations, converter plugins for obfuscation, and scoring metrics. Supports cross-model and multi-modal attack scenarios.

MITRE adversarial emulation framework extended for AI agent attack simulations.

Standardized evaluation framework for automated red-teaming of LLMs. Tests 33 attack methods against multiple target models to measure resistance to jailbreaks and harmful content elicitation. Enables reproducible comparison of safety measures across different guard approaches.

Automated prompt injection testing tool for LLM applications. Sends crafted adversarial prompts to discover injection vulnerabilities in chatbots and agent pipelines. Generates HTML reports of successful attack vectors. Useful for pre-deployment security assessments.

Python library for LLM prompt and response security scanning. Detects prompt injection attempts, jailbreaks, and suspicious patterns using multiple detection methods including vector similarity, YARA rules, and keyword scanning. Lightweight and embeddable in any agent pipeline.